📊 statistics

Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation

This paper proposes a unified random-effects model that extends standard paired statistical methods to rigorously compare multiple language models on shared, clustered evaluations, demonstrating through simulations and real-world data that accurate pairwise rankings depend on properly accounting for both question-level correlations and multiple comparisons.

Original authors: Juan Francisco Mandujano Reyes

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Juan Francisco Mandujano Reyes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers constantly build new computer programs designed to understand and generate human language. To determine which program is the smartest, they put them through a series of tests, much like a standardized exam for students. These tests consist of thousands of questions covering topics from history and science to logic and math. For years, the standard way to compare these programs has been to look at their average scores. If one program gets a higher percentage of correct answers than another, it is declared the winner. However, this simple approach often misses a crucial detail: the questions themselves are not all the same. Some questions are inherently harder than others, and questions within the same subject, like a set of math problems, tend to be correlated. If two programs both struggle with the same difficult math questions, their scores are linked, not independent. Ignoring these connections can make a program look better or worse than it truly is, leading to confusing or incorrect rankings on the leaderboards that guide the field.

A researcher set out to fix this problem by developing a more rigorous way to compare multiple language models at once. Instead of treating every question as an isolated event, they proposed a statistical framework that acknowledges the structure of the test itself. They treated the different computer programs as the main subjects of interest, while viewing the questions and the groups of questions they belong to as random variations that affect everyone. By using a single, unified statistical model, they could account for the fact that the same set of questions was used for every program. This approach allowed them to separate the true skill of the model from the difficulty of the specific questions and the natural grouping of those questions into categories like reading comprehension or translation.

The researcher tested their method using real data from a major benchmark called MMLU-Pro, which contains over 12,000 questions across 14 different subject areas. They selected six popular language models of similar size and had them all answer the exact same 1,497 questions. When they analyzed the results using their new method, they found that the standard way of comparing models often led to false conclusions. In their simulation studies, they showed that if you simply compare models pair by pair without correcting for the fact that you are making many comparisons at once, you have a high chance of declaring a difference exists when it is actually just random noise. In their real-world test, they found that several claims about which model was better than another disappeared once they applied the proper statistical corrections. For instance, one model appeared to beat another by a small margin, but after accounting for the multiple comparisons and the structure of the questions, that difference was no longer statistically significant.

The study also highlighted the importance of how questions are grouped. When questions are clustered by topic, such as a block of math problems, the models' performance on those questions is correlated. The researcher found that ignoring this clustering can underestimate the uncertainty in the scores by a factor of up to nearly three times. By using their mixed-effects model, which treats these groups as random factors, they could accurately measure how much of the variation in scores came from the subject matter versus the actual ability of the model. This revealed that while subject clustering does play a role, the primary driver of the differences was the models themselves, but only when the statistical noise was properly controlled.

Ultimately, the paper provides a clear set of recommendations for how the field should move forward. The researcher argues that before declaring any ranking between multiple models, scientists should first run a broad test to see if there are any differences at all among the group. If that test shows a difference, they should then use specific, corrected methods to compare the models two by two, ensuring that the chance of a false alarm remains low. They also advise that when questions are grouped by topic, these groups should be included in the statistical model rather than trying to manually adjust the numbers. By following these steps, the community can ensure that the leaderboards they rely on reflect genuine improvements in artificial intelligence rather than artifacts of how the data was analyzed. The work does not claim to have solved every problem in evaluation, but it offers a solid, proven foundation for making fair and accurate comparisons in a field where the stakes are high and the margin for error is small.

Technical Summary: Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation

Problem Statement

Current evaluation practices in Large Language Model (LLM) research frequently involve comparing $K > 2$ models simultaneously on a shared set of questions, such as in leaderboards, ablation studies, or hyperparameter sweeps. While existing literature (e.g., Miller [7]) provides rigorous statistical frameworks for comparing exactly two models using paired t-tests and clustered standard errors, these tools do not scale naturally to multiple models.

The primary statistical challenge arises when researchers perform pairwise comparisons across all $K(K-1)/2$ model pairs without correction. This approach ignores the multiple-comparisons problem, leading to inflated false discovery rates. For instance, with $K=4$ models and a significance level of $\alpha=0.05$ , the probability of making at least one false positive claim exceeds 25%. Furthermore, standard unpaired tests fail to account for the correlation between models answering the same questions, and approaches that overlook the hierarchical structure of questions (e.g., clustering by subject or passage) can lead to underestimated standard errors.

Methodology

The paper proposes a unified random-effects model to handle scores from $K$ models on a shared evaluation set, treating models as fixed effects and questions (or question clusters) as random effects.

1. The General Model

For $K$ models answering $n$ questions, the score $s_{i,m}$ of model $m$ on question $i$ is decomposed as:
$s_{i,m} = \mu + \alpha_m + \delta_i + \gamma_{i,m}$
Where:

$\alpha_m$ is the fixed effect of model $m$ (the quantity of interest).
$\delta_i$ is a question-level random effect shared by all models.
$\gamma_{i,m}$ is the residual noise.

This formulation allows the total sum of squares to be partitioned into model, question, and residual components.

2. Two-Stage Testing Procedure

The author recommends a two-stage logic to control the family-wise error rate (FWER):

Omnibus Test: First, fit the model using a one-way repeated-measures ANOVA (or a Linear Mixed Model for clustered data) to test the null hypothesis $H_0: \alpha_1 = \dots = \alpha_K = 0$ . This single test controls the FWER.
Post-Hoc Analysis: Only if the omnibus test is significant, proceed to pairwise contrasts.
- For balanced, non-clustered designs, Tukey's test is used.
- For unbalanced or clustered designs, the paper advocates for Estimated Marginal Means (EMM) with explicit multiplicity adjustments (e.g., Holm or Bonferroni for exact guarantees, or Tukey-type as an approximation).

3. Extension to Clustered Data

To address correlated question groups (e.g., reading comprehension passages, subject categories), the model is extended to a Linear Mixed Model:
$s_{i,c,m} = \mu + \alpha_m + u_c + \delta_{i,c} + \epsilon_{i,c,m}$
Where $u_c$ represents the random effect of cluster $c$ and $\delta_{i,c}$ is the random effect of question $i$ nested within cluster $c$ . This approach estimates variance components ( $\tau^2$ for clusters, $\sigma^2_q$ for questions) directly, generalizing Miller's clustered standard error formulas to any $K$ .

Key Contributions

Unified Framework: The paper establishes that Miller's paired and clustered estimators for $K=2$ are special cases of a general random-effects model. This framework extends these results to any $K > 2$ and handles unbalanced, clustered designs without requiring new derivations for each scenario.
Recovery of Existing Estimators: The author demonstrates that fitting this model via ANOVA or Linear Mixed Models recovers the exact statistical properties of paired t-tests and clustered standard errors when $K=2$ .
Correction for Multiplicity and Clustering: The work highlights that ignoring multiple comparisons leads to unstable rankings and that ignoring question clustering underestimates standard errors (by factors of 1.4–2.7× in simulations).
Practical Implementation: The paper provides concrete R code (using lme4 and lmerTest) and a reproducible workflow for researchers to implement these corrections.

Empirical Results

The methodology was validated through simulation and a real-world application.

Simulation Study

Extending Miller's "Galleon/Dreadnought" example to four models ( $K=4$ ) with 400 shared questions:

False Discovery Rate: Uncorrected pairwise z-tests yielded a family-wise error rate of $\approx 0.26$ at $\alpha=0.05$ , whereas the omnibus repeated-measures ANOVA maintained the rate at $\approx 0.05$ .
Clustering Impact: Ignoring clustering when present underestimated standard errors by a factor of 1.4–2.7×.

Real-Data Application (MMLU-Pro)

The framework was applied to six ~7B parameter models evaluated on a stratified subsample of 1,497 MMLU-Pro questions across 14 subject clusters.

Impact of Multiplicity: While uncorrected paired tests identified 5 significant pairwise differences, only 3 survived Bonferroni or Holm correction. Specifically, claims that Mistral-7B-v0.1 outperformed Qwen1.5-7B-Chat and that Yi-6B-Chat beat Mistral-7B-Instruct-v0.1 were shown to be indistinguishable from noise once multiplicity was accounted for.
Impact of Blocking: The properly blocked repeated-measures ANOVA ( $F=4.45, p=4.8 \times 10^{-4}$ ) demonstrated significantly higher power compared to a one-way ANOVA ignoring question identity ( $F=2.90, p=0.013$ ).
Variance Components: The mixed-effects model estimated that subject-category clustering explained a modest but real portion of the residual variance ( $\hat{\tau}^2 \approx 0.011$ ), informing decisions on whether to include clustering corrections for specific benchmarks.

Significance and Recommendations

The paper argues that ranking claims in LLM evaluations are fragile and depend heavily on the statistical procedure used. The author concludes that a single random-effects model effectively generalizes two-model comparison techniques to multi-model settings, explicitly modeling correlation structures and controlling error rates.

Core Recommendations for Researchers:

Omnibus First: Always run an omnibus repeated-measures ANOVA (or Friedman test) before making pairwise claims.
Adjust for Multiplicity: Follow significant omnibus tests with multiplicity-adjusted post-hoc procedures (Holm/Bonferroni for exact guarantees in complex designs).
Model Clustering: Use mixed-effects models with random effects for clusters and questions rather than manually deriving clustered standard errors.
Transparency: Report both standard and correctly blocked test statistics to illustrate the impact of the evaluation structure.
Temperature Stability: Do not adjust sampling temperature to reduce variance; the model assumes estimation at the deployment temperature.

The paper emphasizes that these methods can fundamentally alter which model ranking claims a reader should believe, moving the field toward more rigorous and reproducible evaluation standards.