Benchmark Illusion: Disagreement among LLMs and Its Scientific Consequences

1. Problem Statement

The paper addresses a critical flaw in the current scientific adoption of Large Language Models (LLMs): the "Benchmark Illusion."

• The Assumption: Researchers currently operate under the premise that if two LLMs achieve comparable accuracy scores on standard benchmarks (e.g., MMLU-Pro, GPQA), they are functionally equivalent, interchangeable, and possess similar underlying reasoning capabilities.
• The Reality: This assumption masks deep epistemic divergence. Models with similar aggregate accuracy often make different errors on specific items. When these models are used for scientific data annotation (e.g., coding text, extracting variables), their distinct error profiles propagate into research results, potentially biasing statistical estimates, altering effect sizes, or even reversing the sign of a conclusion.
• The Gap: Current evaluation regimes focus on aggregate correctness, failing to account for the structure of errors and the lack of agreement between high-performing models.

2. Methodology

The authors employ a multi-stage approach combining benchmark analysis, theoretical simulation, and empirical re-analysis of published studies.

A. Benchmark Disagreement Analysis

• Datasets: Two major reasoning benchmarks were used:
  • MMLU-Pro: 12,032 multiple-choice questions across 14 disciplines.
  • GPQA: 448 expert-level questions in biology, physics, and chemistry.
• Metric: Pairwise disagreement rates were calculated between models with comparable accuracy scores.
• Control: Prompts and decoding strategies were held fixed to isolate differences in model representation and reasoning.

B. Theoretical Framework (Measurement Error)

• The authors model the problem using a linear regression framework where the outcome $Y$ is a latent variable annotated by an LLM ($\hat{Y}$).
• Bias Condition: Estimates are unbiased only if prediction errors ($e = \hat{Y} - Y$) are uncorrelated with independent variables ($X$).
• Hypothesis: If an LLM's error profile is systematically correlated with treatment groups or covariates (systematic bias), the resulting regression coefficients ($\hat{\beta}$) will be biased, even if the model's overall accuracy is high.

C. Simulation Study

• Design: A synthetic dataset ($N=10,000$) mimicking a clinical trial (antibiotic safety) was generated.
• Annotators: Three simulated annotators were compared:
  1. Low Performance (Random): 85% accuracy with random errors.
  2. High Performance (Systematic Bias A): 93.6% accuracy, but systematically under-calling adverse events in the treatment group.
  3. High Performance (Systematic Bias B): 93.8% accuracy, but systematically under-calling adverse events in the control group.
• Goal: To demonstrate how systematic errors in high-accuracy models can induce larger and opposing biases compared to lower-accuracy random errors.

D. Empirical Re-analysis

Two published studies were re-analyzed by replacing human annotations with outputs from eight different high-performing LLMs (both open-source and commercial):

1. Education (Kim et al., 2021): A randomized trial on literacy intervention. LLMs re-scored student essays based on a human rubric.
2. Political Science (Rozenas and Stukal, 2019): A study on "selective attribution" in Russian state media. LLMs re-classified news fragments regarding whether officials were credited/blamed.

3. Key Results

A. Benchmark Disagreement

• High Divergence: Even among models with comparable accuracy, pairwise disagreement is substantial:
  • MMLU-Pro: 16% – 66% disagreement.
  • GPQA: 17% – 65% disagreement.
• Frontier Models: Among top-tier models (accuracy > 60%), disagreement remains high (16% – 38%).
• Implication: Models "get 7 out of 10 right" but systematically get different 7 questions right, indicating distinct internal representations.

B. Simulation Findings

• Random vs. Systematic Error: The low-accuracy annotator (random error) produced an estimate of 0.62 (attenuation bias).
• Systematic Bias Impact: The two high-accuracy annotators (93%+ accuracy) produced drastically different results:
  • Annotator 2 (bias against treatment group): Estimated effect = 0.37 (Severe underestimation).
  • Annotator 3 (bias against control group): Estimated effect = 1.43 (Severe overestimation).
• Conclusion: The structure of errors is more consequential than the rate of errors. High aggregate accuracy does not protect against severe inferential bias.

C. Empirical Re-analysis Outcomes

1. Education Case (Kim et al.):

   • Original human estimate: 0.44.
   • LLM estimates ranged from 0.19 to 0.35.
   • Result: An 84% variation in the estimated treatment effect magnitude.
   • Correlation Paradox: The model with the highest correlation to human scores (0.73) did not produce the most accurate effect estimate, proving correlation is a poor proxy for inferential validity.

2. Political Science Case (Rozenas and Stukal):

   • Original finding: Officials are credited for good news (Positive effect ~0.24).
   • LLM results: Some models replicated the finding, but two models (Gemma-3-27B, Qwen-2.5-72B) produced estimates with the opposite sign, suggesting officials are blamed for bad news.
   • Disagreement: Pairwise disagreement among annotators ranged from 29% to 66%.

4. Key Contributions

1. Identification of the "Benchmark Illusion": The paper formally defines and demonstrates that benchmark convergence does not imply model interchangeability or shared epistemic profiles.
2. Quantification of Scientific Risk: It provides empirical evidence that model choice acts as a "researcher degree of freedom" capable of altering quantitative conclusions by >80% or reversing signs, comparable to changing statistical methods.
3. Shift in Evaluation Paradigm: The authors argue for a new evaluation agenda that prioritizes stability, agreement, and error structure over aggregate accuracy. They propose "Scientific Alignment" as a necessary complement to "Human Alignment."
4. Methodological Framework: They introduce a framework for understanding how systematic annotation errors translate into biased regression estimates, offering a theoretical basis for why "good" models fail in scientific contexts.

5. Significance and Implications

• Reproducibility Crisis: The findings suggest that the growing reliance on LLMs for scientific annotation threatens reproducibility. Results may depend more on the specific model API or version used than on the underlying data.
• Research Practice: Researchers must treat model selection as a critical design choice. The paper recommends:
  • Conducting sensitivity analyses using multiple distinct high-performing models.
  • Moving away from "plug-and-play" assumptions.
  • Adopting hybrid workflows (LLMs for scale + human calibration for bias correction).
• Model Development: The AI community must develop benchmarks that measure inferential robustness and error calibration relative to specific scientific variables, rather than just general knowledge retrieval.
• Epistemic Diversity: While disagreement is currently a threat, the authors suggest it could be harnessed as a resource to quantify uncertainty and stress-test conclusions, provided the disagreement is understood and managed.

In summary, the paper warns that equal accuracy does not equal equal validity. For science to safely integrate LLMs, the field must move beyond headline benchmark scores to understand and mitigate the systematic, model-specific biases that threaten the integrity of scientific inference.