Memorization in reaction-conditioned generative enzyme design: folding confidence, database identity, and docking all fail to detect it

The environment is filled with synthetic chemicals that nature did not evolve to break down. From flame retardants in furniture to pesticides in crops, these substances persist because living organisms lack the biological tools to dismantle them. Scientists have long hoped to engineer custom enzymes—nature's microscopic catalysts—to digest these pollutants and clean up the mess. In recent years, powerful computer programs have emerged that can write the genetic blueprints for new proteins, promising to design these enzymes from scratch based on the specific chemistry of the pollutant they need to destroy. The idea is to feed the computer a description of a toxic chemical and have it output a protein sequence capable of neutralizing it.

A new study examines whether these computer programs are actually designing new tools or simply copying old ones. Researchers at Terram Technologies built a system called XenoFold to generate enzymes for breaking down xenobiotics, the scientific term for synthetic pollutants. They trained the system on known enzyme reactions and then asked it to create new candidates for various chemicals, including some that have no known natural degrader. The team subjected these computer-generated enzymes to a battery of standard computer tests designed to verify their quality, checking if they could fold into stable shapes and bind to their target chemicals. The results revealed a startling failure: the system was not designing new enzymes at all. Instead, it was memorizing the training data and reproducing it almost exactly, a flaw that every standard computer test used to validate the work completely missed.

The researchers began by training their model on fifteen different classes of enzymes, pairing specific chemical reactions with the protein sequences that perform them. They then asked the system to generate new enzymes for four specific scenarios: two reactions where natural enzymes are well known, and two reactions involving pollutants like perfluorooctanoic acid and a specific pesticide metabolite that have no documented natural enzyme capable of degrading them. To ensure the generated proteins were plausible, the team ran them through three widely accepted computer validation methods. First, they predicted how well the protein would fold into a 3D shape, a measure of structural stability. Second, they compared the new sequences against a massive public database of all known proteins to see if they were novel. Third, they simulated how tightly the new enzymes would bind to their target chemicals, a step intended to prove they could actually do the job.

On the surface, the results looked successful. The computer predicted that the generated enzymes would fold into stable, well-defined shapes with high confidence. When the researchers compared these new sequences to the public database of known proteins, the scores suggested the designs were unique and distinct from anything previously recorded. The docking simulations also produced scores that seemed to indicate the enzymes could bind to their targets. Based on these three standard checks, the study initially appeared to show a working system for designing new enzymes. However, the researchers realized that these standard checks were not telling the whole story. They performed a deeper, more direct investigation by comparing the generated sequences not to the general public database, but to the specific list of training examples the computer had memorized during its learning phase.

This direct comparison exposed the truth. When the researchers checked the generated enzymes against the training data, they found that forty percent of the sequences were exact, character-for-character copies of the examples the computer had seen before. Another forty-seven percent were so similar to the training examples that they differed by only a tiny fraction of their structure. This pattern held true even when the computer was asked to design enzymes for pollutants that had no known natural degrader and were completely absent from the training data. The system produced the same high rates of copying regardless of whether the target chemical was familiar or entirely new. The computer was not reasoning from the chemistry of the pollutant to create a new solution; it was simply retrieving and repeating patterns it had stored in its memory.

The study further investigated whether the computer was actually using the chemical information provided to it. In one test, the researchers fed the trained system a chemical reaction that was completely unrelated to the enzyme class it had been trained on. If the system were truly sensitive to the chemical input, changing the reaction should have changed the output. Instead, the generated enzymes remained almost identical to the ones produced for the correct reaction, and their predicted stability scores did not change in any meaningful way. This confirmed that the chemical input was having little to no effect on the final design. The system was essentially ignoring the specific instructions about the pollutant and defaulting to the general patterns it had learned during training.

The researchers also scrutinized the computer docking tests that had initially seemed promising. They found that these tests were unable to distinguish between the computer-generated enzymes and the natural enzymes they were supposed to mimic. More critically, when the researchers tried to use the docking scores to estimate how fast the enzymes would work, the results were wildly inaccurate. For two known natural enzymes, the computer's estimates for their efficiency were off by factors of up to fifteen thousand times compared to real-world measurements. In one case, the computer ranked two enzymes in the exact opposite order of their actual performance. Because the docking scores failed so dramatically to match reality, the researchers removed them from their final analysis, concluding that these standard computer simulations could not be trusted to validate the quality of the generated designs.

The core finding of this work is a cautionary tale about how we verify computer-generated biology. The study demonstrates that a system can produce sequences that look perfect according to standard computer checks—folding well, appearing unique in public databases, and scoring well in binding simulations—while actually being nothing more than a copy of the training data. The three most common validation tools used in the field failed to detect this memorization. Only by directly checking the output against the specific data the computer learned from could the researchers uncover the flaw. The author concludes that without these specific, inexpensive checks, claims of new enzyme design may be based on an illusion of novelty. The system they tested does not currently produce enzymes ready for real-world use, and the study suggests that the field needs to rethink how it validates computer designs to ensure they are truly new and not just recycled memories.