Measuring all the noises of LLM Evals

1. Problem Statement

Evaluating Large Language Models (LLMs) requires distinguishing between genuine performance improvements (signal) and statistical fluctuations (noise). Current evaluation practices often suffer from low statistical power due to:

• Confusion of Noise Types: Failing to distinguish between noise arising from the model's stochastic generation (prediction noise) and noise arising from the specific set of questions sampled (data noise).
• Inefficient Analysis: Relying on unpaired comparisons or single-prediction metrics, which ignore the correlation between models on the same questions.
• Misleading Training Curves: Using training curves that plot performance over steps on a fixed dataset, which often ignores data noise and leads to false positives when prediction noise is low.

The paper argues that without properly quantifying and separating these noise components, empirical decisions regarding model improvements are often unreliable or require prohibitively large sample sizes.

2. Methodology

A. Decomposition of Noise

The authors define three distinct types of noise based on the Law of Total Variance:

1. Prediction Noise ($Var_\epsilon[A]$): The variance caused by a model generating different answers for the same question due to stochasticity (e.g., temperature, random seeds). This is reducible via averaging or majority voting.
2. Data Noise ($Var_x[E_\epsilon[A]]$): The variance caused by sampling a finite set of questions from a larger population. This reflects the difficulty variation of the specific questions chosen. This is irreducible without changing the question set.
3. Total Noise: The sum of prediction and data noise, representing the actual variance observed in a standard evaluation.

The relationship is formalized as:
$$Var_{x,\epsilon}[A] = Var_x[E_\epsilon[A]] + E_x[Var_\epsilon[A]]$$
Where $x$ represents the question and $\epsilon$ represents the random seed/prediction process.

B. The All-Pairs Paired Method

To maximize statistical power, the paper proposes an All-Pairs Paired Method:

• Paired Analysis: Instead of comparing models independently, the method analyzes the difference in performance ($A(x) - B(x)$) on the same question $x$. This leverages the high correlation between similar models on specific questions to drastically reduce data noise.
• All-Pairs: The method computes these paired variances for every pair of models in a leaderboard or experiment set, rather than just comparing against a single baseline.
• Estimation: The authors provide unbiased estimators for these variance components, including corrections for small sample sizes ($K$ predictions per question) to avoid bias in the data noise estimation.

C. Theoretical Modeling (Beta Distribution)

The paper hypothesizes that the distribution of question-level expected accuracies follows a Beta distribution $Beta(p, 1-p)$, where $p$ is the model's mean accuracy. This theoretical model predicts that the total paired variance for correctness evals approximates $p(1-p)$, allowing for the prediction of noise levels without extensive custom analysis.

3. Key Contributions

1. Formal Definition and Measurement: The paper provides the first clear, empirical measurement of prediction noise, data noise, and total noise across millions of question-level predictions on multiple evals (e.g., SWE-bench, MATH500, HumanEval).
2. The "All-Pairs Paired" Framework: A scalable methodology to measure noise components for all model pairs, enabling the creation of meaningful error bars for leaderboards.
3. Empirical Findings on Noise Characteristics:
   • Predictable Total Noise: Total noise levels are highly predictable and consistent across different model pairs for a given eval, closely matching the Beta theory prediction ($SE \approx \sqrt{p(1-p)/N}$).
   • Prediction Noise Dominance: In most standard settings (especially at higher temperatures), prediction noise exceeds data noise.
4. Statistical Power Gains: The paper demonstrates that by averaging multiple predictions (reducing prediction noise) and using paired analysis (reducing data noise), the statistical power increases significantly.
   • Example: On HumanEval, combining pairing and averaging reduced the minimum detectable difference from 12% (unpaired, single prediction) to 2-4%.
   • Example: On MATH500, prediction noise was found to be roughly 2x the data noise, meaning averaging predictions yields massive gains in sensitivity.

4. Results and Evidence

• Heatmaps: Visualizations (Figure 3) show that models with similar overall accuracy perform similarly on individual questions, validating the effectiveness of paired analysis.
• Training Curve Validity: The paper shows that standard training curves (plotting accuracy vs. training steps on a fixed set) are often invalid because they only show prediction noise. However, if prediction noise dominates (which is common), training curves become valid indicators of improvement when paired with bootstrapping.
• Noise Component Breakdown: Experiments on CRUXEval show that while total noise remains constant, the ratio of prediction to data noise shifts with temperature. At high temperatures (0.8), prediction noise dominates; at low temperatures (0.2), data noise becomes more prominent, yet the total noise remains predictable.
• Comparison to Existing Methods: The proposed method aligns with and generalizes established techniques like the Bootstrap and the Sign Test, proving they yield equivalent results when applied correctly to LLM evals.

5. Significance and Implications

• Lowering the Barrier for Rigor: By establishing that total noise is predictable and that prediction noise is often dominant, practitioners can now estimate statistical significance without running complex, custom statistical tests for every experiment.
• Optimizing Evaluation Costs: Since prediction noise is the larger component in many cases, the paper recommends averaging multiple predictions (or using majority voting) as a cost-effective way to increase statistical power, rather than simply increasing the number of questions (which is expensive and limited by data availability).
• Leaderboard Reliability: The "All-Pairs Paired" method allows for the generation of accurate error bars for leaderboards, addressing the issue where small differences between models are currently reported as significant when they may be noise.
• Meta-Analysis: The framework enables robust meta-analysis across different evals by normalizing results using z-scores derived from these specific noise measurements, allowing for the combination of results from evals of varying sizes and difficulties.

Conclusion

The paper fundamentally shifts the perspective on LLM evaluation from treating noise as a monolithic obstacle to a decomposable, measurable, and predictable phenomenon. By separating prediction noise from data noise and utilizing paired analysis, researchers can detect much smaller model improvements with higher confidence, leading to more rigorous and efficient empirical research in AI.