There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items
This paper demonstrates that modern LLM leaderboards are fundamentally unreliable because evaluation harness configurations (such as prompt wording and scoring methods) are the primary drivers of performance variance, often determining model rankings more than the models' actual capabilities.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, researchers rely on standardized tests to measure how well computer programs understand language. These tests, often called benchmarks, present a series of multiple-choice questions to a model and record how many answers it gets right. The resulting score is treated as a definitive fact about the machine's intelligence, allowing scientists to rank different models against one another and declare a winner. For years, the community has operated under the assumption that while a model's performance might fluctuate slightly due to random chance, the overall ranking reflects a stable truth about the technology's capabilities. The question of whether a model is truly smarter than another has been answered by looking at a single number: the percentage of correct answers.
However, a new investigation suggests that this single number tells a much more complicated story. The study reveals that the score a model receives is not just a measure of its intelligence, but also a product of the specific rules used to grade it. These rules, known as the evaluation harness, include choices that seem trivial on the surface: the order in which answer options are listed, the exact wording of the instructions given to the machine, and the method used to decide if an answer is correct. While these choices are often made arbitrarily or based on convenience, the researcher found that changing them can completely rewrite the leaderboard, turning a model ranked last into the top performer, or vice versa. The findings imply that the current way we rank artificial intelligence is far less stable than previously believed, and that the "best" model depends entirely on how the test is administered.
To uncover this hidden instability, a researcher at the University of California, San Diego, designed a massive experiment that treated the testing rules themselves as a variable. Instead of running a single test and accepting the result, they created a grid of twenty-six different ways to administer the same exam. They took twelve different language models from four major families and asked them to answer the same 3,679 multiple-choice questions. For every single question, they ran the models through every possible combination of the testing rules. This included shuffling the order of the answer choices, changing how the questions were phrased, and switching between two different methods for scoring the answers: one where the model generates a text response, and another where the computer calculates the mathematical likelihood of each option being correct.
The results were startling. When the researcher looked at the scores for a single model, they did not find a single point of accuracy. Instead, they found a wide band of possible scores. For one of the top-performing models, the score ranged from a low of 31 percent to a high of 89 percent, depending entirely on which of the twenty-six rule sets was used. This means that the same machine, with the same internal code and the same knowledge, could be credited with getting nearly all the answers right or failing to get a third of them, simply because the test administrator changed the order of the letters or the style of the prompt. On average, 85 percent of the answers that a model was credited with getting right under the standard rules could be marked as wrong under a different, equally valid set of rules.
The most critical discovery was that these fluctuations were not random noise spread evenly across the test. The instability was concentrated precisely on the questions that separated the models from one another. When the researcher looked at the questions that two neighboring models both answered correctly or incorrectly regardless of the rules, the models were essentially tied. The differences that created the rankings existed only on the questions where the models were uncertain. In fact, for the pairs of models that were ranked next to each other, nearly all of the gap in their scores—about 96 percent on average—was carried by these unstable questions. If you removed the questions that flipped back and forth, the rankings vanished, and the models became indistinguishable.
This fragility meant that the identity of the "winner" was not a fixed property of the models but a choice made by the evaluator. Under one set of rules, a specific model might take first place, while under a different set of rules, a completely different model would win. In this study, four different models were able to reach the number one spot depending on which configuration was used. One model that finished eleventh out of twelve under the standard rules jumped to first place under a different scoring method. The researcher noted that the choice of how to score the answer—whether to read the text the model wrote or to calculate probabilities—was the single most powerful factor, causing more shifts in the rankings than the order of the options or the wording of the prompt.
The study also examined a popular trend in the field: compressing these large tests down to a smaller set of the most "informative" questions to save time. The logic was that a few hundred well-chosen questions could represent the whole test. The researcher found that this practice actually makes the problem worse. The questions that are selected to be the most informative are exactly the ones that are most sensitive to the testing rules. By keeping only these difficult, high-stakes questions, compressed benchmarks become even more volatile than the full tests they replace. A smaller test does not provide a clearer picture; it provides a more fragile one, where the ranking is even more likely to change based on minor adjustments to the setup.
Ultimately, the paper argues that the current practice of reporting a single number as the truth about a model's ability is misleading. The score is not a property of the model alone; it is a property of the model interacting with a specific set of rules. The researcher suggests that future reports should stop treating the leaderboard as a fixed hierarchy and start presenting the range of possible scores. They propose that when comparing models, scientists should focus on the questions where the models are stable and agree, rather than the ones where the rules dictate the outcome. Until the community agrees on a single, unchangeable way to run these tests, the rankings will remain a reflection of the evaluator's choices rather than a definitive measure of artificial intelligence.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.