The First Major Refactor of the Glow Audit Standard

From the very beginning, Glow has been designed with the assumption that cheaters would successfully get fraudulent solar farms onto the protocol. This is because Glow measures carbon impact via electricity generation, and electricity does not have inherent cryptographic guarantees like what can be found in other blockchain protocols. Instead, Glow needs to rely on trusted auditors who physically visit solar farms and confirm their specifications.

Auditors can be deceived, they can be bribed, and they can make mistakes. An attestation by an auditor is only as good as the circumstances surrounding that attestation, and cheaters have many opportunities to manipulate the circumstances surrounding a solar audit.

That's why Glow was designed as a fact finding engine rather than a static certification standard. Glow assumes that cheaters are going to be clever and creative, and that keeping the protocol pure will require proactively evolving the audit standard as cheaters get more sophisticated.

In addition to verifying specific facts during the audit, Glow auditors collect as much information as possible to increase the chances that suspicious or unusual patterns can be detected in the future. Furthermore, all of this information is made available to the general public, so that anyone is able to inspect the audits and locate inconsistencies, mistakes, and other problems.  Finally, Glow audits are designed to be easily cross referenced against public data such as satellite images, weather reports, and city permits to improve the discoverability of any mistakes or dishonesty.

In fact, discoverability is one of the key principles guiding the Glow audit standards. We know that individual auditors and individual audits are susceptible to cheating. But as long as Glow gives the truth many opportunities to be discovered and reported, the truth will come forward eventually. If an audit report is found to have material problems, the corresponding solar farm can be removed and its historic carbon credits can be cancelled.

The Fundamentals of Cheating

Cheating is universal. From board games at home, to olympic sports, to the plant *Pentadiplandra Brazzena* which tricks apes into thinking it contains sugar, cheating them out of a nutritious meal, cheating is found in every walk of life.

Cheating happens when people believe that cheating is easy, that getting caught is unlikely, that consequences are minimal, that the rewards are valuable, or some combination of these.

Preventing cheaters therefore requires having systems in place that make cheating difficult, that are effective at catching people retroactively, that deliver meaningful consequences, and that minimize the rewards of cheating successfully.

Deterring cheaters also requires making sure that would-be cheaters are aware of the systems that are in place and convincing them that cheating isn't worthwhile. The best anti-cheating system is one that nobody bothers to challenge, because everybody already believes that cheating won't be worthwhile.

Finally, cheating and catching cheaters is a game that is always evolving. As cheat detection mechanisms get more sophisticated, cheaters will also get more sophisticated. A healthy anti-cheat system has to be constantly growing and changing to keep up with would-be cheaters.

Glow was built with these principles in mind, and has taken steps around each layer of deterrance to prevent and punish cheating.

The First Discovery of Cheating on the Glow Protocol

On February 12th, 2024, the audit standard architect for the Glow protocol, Fatima, discovered that 7 out of 11 solar farms had forged documents to lie about their installation date. Glow has a strict requirement that solar farms need to be brand new and installed for the purpose of participating in the Glow protocol.

In an effort to dodge this requirement, several already existing solar farms submitted documentation with forged installation dates, tricking auditors and successfully getting accepted into the Glow protocol. While Fatima was reviewing the audits prior to their publication, she noticed some suspicious patterns and chose to investigate further.

The ultimate proof was discovered by checking historical satellite images using Google Earth Pro. Several solar farms were visible from space and had clearly been installed multiple years before the creation of the Glow protocol. A smoking gun had been found for a small number of farms, leaving the question: exactly how many farms had committed fraud?

Fatima brought her discoveries to the GCAs, the Veto Council, and trusted advisors within the Glow ecosystem to develop a plan. After some discussion, it was determined that the best and most ethical path forward was to keep the discovery confidential until the full set of facts were available. It was additionally determined that all new applications from solar farms should be put on hold until the Glow audit standard could be refactored to make this form of cheating impossible.

From February 14th to March 29th, no new audits were published and no new solar farms were added to Glow, because everything was being reviewed to verify its integrity.

When the investigation finished, the solar farms with IDs 1, 5, 6, 7, 8, 11 and 13 were all determined to have committed fraud with respect to their installation date. All of the solar farms were confirmed to be real solar farms that were reducing overall grid emissions, but the additionality was not present because the solar farms were not purpose built for the Glow protocol.

These solar farms have been kicked off of the Glow protocol, and will stop receiving on-chain rewards. In aggregate, these solar farms paid $111,756.15 in protocol fees, and the full amount is going to be forfeit.

That means that the solar farms with IDs 2, 3, 4, 12, 14, and 15 are currently believed to be legitimate. Their full audits will be made available on https://glow.org/ shortly. The IDs of 9, 10, 16, 17 are all unused, and for various reasons were never assigned to a solar farm.

Responsible Disclosure

Responsible disclosure is a term that is typically used with software vulnerabilities. The idea is that if a good person finds a bug that could be used for evil, they keep that bug confidential to avoid telling bad guys how to exploit vulnerable users. Instead, they tell trusted members of the software development team, wait for a patch to be rolled out, and then they disclose their discovery after users are no longer in danger.

Responsible disclosure establishes that there is a limit to keeping a critical bug confidential. If a good guy can find a vulnerability, a bad guy can also find that same vulnerability. The bug is putting users in danger, and if the development team is not acting quickly enough to fix the bug, users should be alerted so that they know to stop using the software.

A typical timeline for responsible disclosure is 90 days, though that timeline is often adjusted based on the severity of the vulnerability, the obscurity of the vulnerability, and the difficulty of rolling out a patch. If a patch is not rolled out in time, all users are alerted to the vulnerability, which gives hackers a tool to exploit users but also allows users to protect themselves by disabling the vulnerable software.

Similar principles are at play with fraud in the Glow ecosystem. When fraud is first suspected, the most important task is to figure out the full story behind the fraud. For example, in the current incident, several farms were initially thought to be falsified that were later vindicated as legitimate farms.  Disclosing the discovery too early may have led to witch hunts and the incorrect punishment of genuine solar farms.

In the world of cheating, another element is at play. Frequently, exploits being used by cheaters do not get punished for some period of time, because the watchdog does not want to alert the cheaters that they have been caught, giving cheaters more opportunity to expose themselves as dishonest and therefore allowing a purge to be more effective.

Within Glow, I believe that a reasonable timeline for responsible disclosure will generally be 60 days. If someone finds evidence of fraud on the Glow protocol, they should bring that evidence forward to the GCAs or other trusted leaders within the Glow ecosystem and otherwise respect the short term confideniality of their findings.

The GCAs and leadership should then work to get to the bottom of the issue quickly disclose the findings to the community. If the leadership takes more than 60 days to handle the problem, then the whistleblower may ethically disclose their findings to the entire community.

What Broke Down

Several things went wrong to enable the first fraud incident in the Glow protocol. In my opinion, the most important breakdown is that the cheaters did not understand that the Glow audit standard could evolve over time to detect and retroactively correct for cheating. The cheaters felt safe once their forged documents had been submitted, because the auditor had signed off on the documents and the cheaters began receiving rewards. This encouraged more cheating.

A second thing that went wrong is that the Glow audit protocol did not properly anticipate that pre-existing solar farms would be willing to forge documents in order to get onto the Glow protocol. The Glow protocol incorrectly trusted the legal documentation, and did not take steps to independently verify that the solar was brand new to the property.

Finally, a clear conflict of interest existed between the initial installers, their friends, and other early members of the Glow ecosystem. Frankly, there was a lot of trust between the early participants of the Glow protocol that cheating would not be attempted, because such a thing would be damaging to the long term reputation of Glow. This incident demonstrates vibrantly that it's important not to skip steps even when strong trust relationships exist, and that it's also important to be keenly aware of conflicts of interest.

This will not be the last case of successful cheating within the Glow ecosystem. As Glow gets better at detecting cheating, cheaters will get better at cheating. What's important is that Glow continues to evolve, and that it evolves as fast or faster than the cheaters.

Evolving the Glow Audit Standard

As discussed previously, the important elements in deterring cheating are making cheating difficult, making detection easy, ensuring consequences are meaningful, minimizing the total rewards that can be claimed by cheating, and importantly making sure that would-be cheaters are aware of all the dangers associated with cheating on the Glow protocol.

This blog post is a step towards the latter. By exploring the mechanisms of the Glow audit process, we can inform people how cheaters get caught and discourage people from attempting to cheat. The information also helps the community improve its own ability to independently verify solar farms, and identify potential opportunities for cheating.

Before we talk about how the refactored audit standard fights cheating, we should establish what counts as cheating. The Glow protocol really only cares about three facts: solar farms must be brand new, solar farms must contribute the full value of their electricity revenue to the carbon credits rewards pool, and solar farms need to be displacing unclean energy from the power grid to create climate impact.

Everything that the Glow audit standard does is designed around gaining confidence in those three facts. And as the standard evolves, it's trying to find cheaper and more effective ways to prove that those three requirements are met by every solar farm that is certified by Glow.

Making Cheating More Difficult

The refactored Glow audit standard now requires solar farms to take photographs both before the solar panels are installed and after the solar farm has been completed. These photos need to be sourced by the auditor rather than supplied by the homeowner or installer. Typically, the auditor will pay an independent contractor such as a real estate agent or a drone photography agent to take the pre-installation photos. The post-installation photos are taken directly by the Glow auditor, who has to visit the solar farm themselves and install monitoring equipment.

These photos serve to both demonstrate that the solar is brand new solar, and also demonstrate that the solar is real solar that is connected to either the electrical grid, or is supplying energy to a building that was previously connected to the electrical grid.

The protocol fee that homeowners pay is based on the value of their electricity. This is confirmed by having the homeowner provide a copy of their power bill, and then independently verifying the electricity rates with the utility company. In many cases, the utility company has a publicly accessible website that openly publishes their power rates, making the utility bills easy to verify.

Finally, the auditor installs a custom, cryptographically authenticated electricity monitoring device that measures the amount of useful electricity produced and consumed by the solar panels with a tolerance of less than 4%.

Any attempt at cheating needs to satisfy all of these requirements, and any evidence produced by the cheater will be publicly available for at least 10 years. Producing fake variations of all of this evidence is of course not impossible, but it does require a comfortable degree of sophistication and intentionality.

Making it Easier to Detect Cheating

If history has taught us anything, it's that cheaters are capable of being very, very clever. Any non-cryptographic system designed to make cheating difficult can be circumvented by someone who has enough intelligence and motivation. Therefore, being able to detect cheaters retroactively is critical to any system that is trying to reduce cheating.

One of my favorite examples of cheating comes from the video game world. There is a community of speedrunners that compete to finish video games as fast as possible. Cheating is highly prevalent in the speedrunning community, because its relatively easy to cheat and the consequences of getting caught aren't very impactful.

This particular example involves a video game called A Link To the Past. The world record was obtained and held by a cheater who had made a relatively immaculate video that was devoid all common signs of cheating, and successfully fooled initial investigation.

In the game, there is a boss fight where a bunch of fireballs spawn in a random order. At least, the spawn pattern was understood to be random when the world record video was created. It was discovered later that the spawn pattern could be predicted by looking at previous events in the game.

Because nobody had known about the determinism of the spawn pattern when the video was created, the cheater had no way to ensure that their video used the correct spawn pattern. When the pattern was discovered, the cheater was caught and the record was thrown away. The cheater was caught by applying new knowledge to old evidence.

Applying new knowledge to old evidence is one of the most powerful ways to catch meticulous cheaters. Not only is it commonly used in speedrunning, but it's also used to catch criminals. One famous historical example is the development of DNA forensics, which was able to close out a number of cases that had previously been stalled.

This is the main reason that Glow collects a significant amount of evidence when auditing a solar farm. Even when a particular photograph or requirement doesn't directly contribute to validating a solar farm today, it may provide clues into the legitimacy of the solar farm in the future, after the audit standard has evolved further. A few extra photos may end up going a long way towards catching cheaters retroactively.

Enforcing Consequences

Joining the Glow protocol is expensive. Solar farms are required to pay a fee to Glow that is proportional to the expected future value of the next 10 years worth of electricity production. For most solar farms on Glow, this fee is close to the entire cost of building the solar farm in the first place: tens of thousands of dollars for smaller systems, and hundreds of thousands of dollars for larger systems.

If cheating is discovered, the corresponding farm is removed from Glow. Rewards stop without any refund to the protocol fee, and a giant hole is left in the balance sheet of the cheater.

Even more, the protocol fee is earned back over a period of 10 years. Even if a solar farm gets away with cheating for multiple years, they will still be left with a large financial loss after the cheating is discovered.

Minimizing Rewards

The Glow protocol currently has $4.5 million in USDC available as rewards for solar farms. If Glow succeeds in its mission, the rewards pool will grow by several orders of magnitude. That's big enough that many intelligent people will spend a large amount of time looking for ways to cheat the system and earn these rewards.

That said, Glow can still take a few steps to minimize the total number of rewards that can be extracted by a cheater. Larger solar farms receive more in-depth audits, and the Glow monitoriing equipment is cryptographically unable to report power production numbers that exceed the physical production capacity of the solar farm.

Each week, auditors manually review the set of rewards that are being distributed and have the ability to exclude any anomalous values, setting that farm aside for a deeper investigation.

Finally, the rewards pool is distributed over a period of 520 weeks, which minimizes the total amount of rewards that a cheater can extract in a given period of time.

Randomized Deep Investigations

One trick that Glow auditors can use to verify the effectiveness of audits is to perform randomized deep investigations on Glow solar farms. These investigations are considerably more expensive than a typical audit and can't be used everywhere, but doing a deep investigation can significantly improve the odds of catching a cheater.

By randomly checking 10-20 farms, Glow can get a reliable statistical estimation of how prevalent cheating is on the Glow protocol as a whole. A single farm out of 20 being bad suggests that the protocol as a whole is under 15% prevalence of cheating. 2-3 bad farms suggests that the cheating rate is potentially quite high, and that the audit standard needs to be revised.

A deep investigation can leverage numerous techniques to gain confidence in a solar farm. Historic satellite data covering many years can be requested from professional satellite companies. The utility company can be interviewed. The city that issued the construction permit can be interviewed. You can also interview employees at the installer, friends and family of the solar farm owner, the financing entities, neighbors, and so on. If you are willing to spend a lot of effort confirming that a solar farm is legitimate, there are a lot of places to dig up clues.

The results of these deep investigations can be used to inform future revisions of the Glow audit standard. The primary audit standard can be revised to remove steps and requirements that don't generally inhibit cheating, and new steps can be added if cheaters are using similar exploits to get past the auditors. These investigations aren't just useful for estimating fraud prevalance on Glow, they are also useful for improving Glow's overall ability to catch cheaters.

Secret Detection Techniques and Parallel Construction

A controversial tactic for detecting cheaters is to privately develop secret techniques for detecting cheaters. When cheating is discovered, a deeper investigation is launched to find evidence of the cheating that won't reveal any of the secret techniques.

When the cheater is finally confronted, the secret discovery techniques are kept fully hidden, and instead only the extra evidence is used to convict the cheater. This process of finding extra evidence after already knowing that someone is cheating is called parallel construction.

Parallel construction is controversial because it is frequently used by law enforcement to hide the fact that they are using illegal spying to catch criminals. The controversy comes not from the catching criminals part, but from the part where law enforcement is violating the rights of every day citizens in the name of catching cheaters. By using parallel construction, law enforcement does not need to reveal that they are using illegal spying.

Parallel construction is also used in more legal settings. For example, chess.com uses parallel construction to catch chess cheaters. They are quite open about their use of proprietary cheat detection technology, and it's not controversial because none of their proprietary technology is breaking any laws or violating the rights of users.

Glow doesn't currently use any parallel construction strategies. The main reason is that Glow wants every member of the Glow community to be able to independently verify that solar farms are real. A technique that Glow auditors keep confidential is a technique that can't be used by the community to spot mistakes that the auditors made.

I believe that Glow is better off avoiding the use of secret tactics and parallel construction, however if Glow is struggling to stop cheaters in the future, it is potentially a useful tool that auditors can fall back on.

Glow Cheating Goals

Any non-cryptographic system has to strike a balance between efficiency and integrity. Every dollar that is spent inhibiting and detecting cheaters is a dollar that is not spent putting up new solar panels. Therefore, spending large amounts of resources ensuring that there is absolutely no cheating on Glow is probably a suboptimal use of climate funding.

Instead, Glow aims to keep the number of cheated carbon credits under 15% of the total carbon credit issuance. This number can be verified using randomized deep investigations, and as long as all data points to Glow being more than 85% accurate, the amount of money spent dealing with cheaters can be kept to a minimum.

At launch, it was determined that all carbon credit production should be discounted by 35% to adjust for unknown quantities of error. Undetected cheating was one of the categories of error that was considered when establishing this number. Other factors included things like over-estimating the climate impact of pushing clean energy onto the grid, and incorrectly assessing the additionality of certain farms.

This 35% discount is part of Glow's commitment to not overstating its total climate impact. The truth is that reducing carbon emissions involves a large amount of unknown variables, and there are good chances that Glow's best efforts today overlook things that will later be discovered to be obviously important.

Stacking Up to Other Climate Efforts

Establishing the legitimacy and effectiveness of a climate project is difficult. Some of the largest carbon efforts in the world are estimated by critics to be 90% ineffective.

This is why Glow is narrowly focused on just a single climate strategy - photovoltaic solar that displaces unclean grid energy. This focus allows Glow to establish a deeper expertise and build a much stronger cheat detection system.

Other carbon registries spread their attention over hundreds of methodologies, which dilutes expertise, forces solutions to be broader (and therefore less effective), and ultimately creates lots of opportunities for cheaters to take advantage of well intentioned climate donations.

In my mind, the first cheating incident of the Glow protocol simply underscores how important it is to have a narrow focus. Glow went through great lengths to create an effective and robust audit standard using seasoned experts from the audit industry and still allowed cheaters onto the protocol. The audacity of forging legal documents is surprising, and underscores the necessity of putting a large amount of effort into an audit standard. By being so restrictive around the climate efforts that Glow is willing to certify, Glow can ensure that its audit standard can receive enough attention to be robust to cheating.

The climate industry currently has a very poor yet well deserved reputation among the general public. Historically, it has not been effective at spending money efficiently. I believe that Glow's approach will be able to stand apart as more reliable, and that the speed at which the current round of cheaters were caught is strong evidence in favor of Glow's potential. Time will tell.  Until then, I'm excited to be involved in the Glow project and looking forward to what I anticipate is a bright future.

‍