CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

In the race to build smarter machines, researchers rely on standard tests to measure how well an artificial intelligence can solve problems. These tests are like school exams: a set of questions with known answers that a model must solve to earn a score. The higher the score, the more capable the machine is considered to be. However, a shadow has fallen over these results. Because these tests are published online, the questions and answers are often available on the internet, where the machines learn. There is a growing fear that high scores do not always prove a machine is smart; they might simply prove it has memorized the test questions during its training. If a machine has seen the exact questions before, its score is inflated, and the ranking of different models becomes a contest of memory rather than reasoning.

This creates a difficult problem for anyone trying to judge these machines fairly. Most of the powerful models are "black boxes," meaning their internal training data is secret. An auditor cannot simply look inside the machine to see if it has memorized a specific question. They can only see the final score. For years, the best way to check for memorization was to rewrite the questions. If a machine truly understands the math or science behind a problem, it should solve a reworded version just as well as the original. If it fails the reworded version, it likely just memorized the original. But this method has a hidden flaw: it assumes that if a machine gets the reworded question right, it learned the concept. If the machine gets the reworded question right simply because it memorized the answer key, the test fails to catch the memorization.

A team of researchers set out to build a better way to audit these scores without needing to see inside the black box. They developed a method called CleanScore, which treats every test question as a parent with three children: the original public question, and two fresh, privately written versions that keep the same numbers and facts but use different words. The researchers asked five different open-source models to solve 200 questions from two major benchmarks, one focused on math and the other on science. For each question, they compared how well the models did on the original public version versus the two fresh versions. If the models consistently scored higher on the public version, it would suggest they had memorized the test.

The audit found no evidence of this kind of memorization. Across all five models and both benchmarks, the scores on the public questions were not significantly higher than the scores on the fresh, rewritten questions. The researchers calculated that any advantage gained from seeing the public wording was likely less than five percentage points. This result held up even after the team fixed a technical error in their initial setup where some models were cut off before finishing their answers. The study also included a "negative control," a set of questions the models had never seen and could not have memorized, to ensure that the rewriting process itself wasn't making the questions harder or easier. The results showed that the rewriting process was fair, and the lack of a score gap was a genuine finding, not an artifact of the test design.

However, the most striking part of the study came from a series of experiments designed to see what the audit would miss. The researchers deliberately trained four different models on the exact wording of 100 of the test questions. They then ran the same audit to see if it would catch the memorization. On the math benchmark, the models that had memorized the questions improved their accuracy by 26 points on the memorized wording. But when tested on the fresh, rewritten versions of those same questions, they also improved by 20 points. The audit, which only looks at the difference between the original and the rewrite, saw a gap of only about six points. In other words, the audit detected only a small fraction of the memorization. The vast majority of the advantage—the part that came from understanding the problem rather than just memorizing the words—was invisible to the test.

This blindness was even more extreme on the science benchmark. In one experiment, the researchers trained a model on 100 science questions and then tested it on the rewritten versions. The model improved its accuracy by 49 points on the memorized questions. Because the rewritten versions kept the multiple-choice answers exactly the same, the model also scored perfectly on the rewritten questions, simply by recognizing the correct answer string it had memorized. The audit calculated a gap of negative two points, effectively seeing no memorization at all. The model had learned the answers, but the test design, which kept the answer options fixed, allowed that knowledge to hide in plain sight.

The researchers concluded that while their method successfully measured whether a model was relying on the specific surface wording of a question, it could not measure the total amount of contamination. A model could be heavily trained on a dataset and still appear clean if the test questions were rewritten in a way that preserved the core answer. The study showed that for the five models they audited, there was no strong evidence that they had memorized the specific wording of the test questions. But it also proved that a "clean" score does not guarantee a model hasn't seen the material before; it only guarantees that the model didn't rely on the exact phrasing. The audit successfully bounded the surface-level inflation, but it could not rule out that the models had learned the material in a way that survived the rewriting process.

The study also highlighted the limits of how small an audit can be. The researchers found that to reliably detect a significant advantage, an audit needs to test at least 200 questions. With fewer questions, the statistical uncertainty is so large that the test cannot distinguish between a real advantage and random noise. They also showed that while it is possible to rank models correctly with fewer questions, determining the exact order of performance requires a much larger sample size. The work serves as a rigorous, registered experiment that sets a new standard for how to talk about uncertainty in AI testing. It does not declare the models innocent of all memorization, but it provides a clear, honest statement about what the data can and cannot show when only the final scores are available.