🤖 machine learning

Resolution Diagnostics for Paired LLM Evaluation

This paper diagnoses a critical statistical power deficit in current paired LLM leaderboards, revealing that many pairwise rankings are unresolved due to insufficient sample sizes and the widespread misuse of unpaired effect-size shortcuts that underestimate required sample sizes by approximately a factor of two.

Original authors: Anany Kotawala

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anany Kotawala

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a cooking competition. Two chefs, Chef A and Chef B, have just finished their dishes. You taste them and say, "Chef A's dish is 0.8% better than Chef B's." The crowd goes wild, the news headlines scream "Chef A Wins!", and people start buying Chef A's cookbooks.

But wait. Did you actually know Chef A was better, or did you just get lucky with the specific ingredients you happened to taste?

This paper is a "reality check" for the world of Large Language Models (LLMs). It argues that many of the current leaderboards (the "cooking competitions" for AI) are making claims that are too shaky to be true. They are declaring winners based on tiny differences that could easily be just random noise.

Here is the breakdown of the paper's findings using simple analogies:

1. The "Paired" Problem: It's Not a Race, It's a Duel

Most people think of testing two AI models like a race where they run on different tracks. But in reality, these models are tested on the exact same questions (the same track).

The Analogy: Imagine two runners running on the same track at the same time. If the wind blows, it blows on both. If the track is slippery, it's slippery for both. Because they face the exact same conditions, their results are "paired."
The Mistake: Many current tools calculate the "statistical power" (the confidence that the result is real) as if the runners were on different tracks. This is like ignoring the wind and the track conditions.
The Result: The paper found that by ignoring this "paired" nature, many tools are underestimating the sample size needed by a factor of two. It's like thinking you only need 50 taste-testers to prove a difference, when you actually need 100.

2. The "Resolution" Diagnostic: The Microscope Check

The authors created a new tool called a "Resolution Diagnostic." Think of this as a microscope for the leaderboard.

How it works: Before you declare a winner, you look through the microscope.
- If the gap between the models is huge (like 15%), the microscope shows a clear, sharp image. The winner is real.
- If the gap is tiny (like 0.5%), the microscope shows a blurry, fuzzy image. You can't tell if the blur is because one model is actually better, or just because of random noise.
The Finding: When they looked at two famous leaderboards (Open LLM Leaderboard and MMLU-Pro), they found that many of the "winners" were actually just blurry images.
- On the Open LLM Leaderboard, 11 out of 40 claimed pairings were too blurry to be sure.
- On the MMLU-Pro top 10, 4 out of 9 of the closest matches were unresolved.

3. The "Shortcut" Trap: The Broken Calculator

The paper discovered that many researchers are using a "shortcut" to do their math.

The Analogy: Imagine you have a calculator that is designed for single runners (unpaired). You try to use it for a duel (paired) by just pressing a "multiply by 0.7" button at the end.
The Problem: The paper proved mathematically that this shortcut is broken for close races. It consistently tells you that you need half the data you actually need.
The Evidence: They tested five popular statistical tools. Three of them (including a famous textbook formula and a popular software called G*Power) fell for this trap. They silently gave answers that were half the size they should have been, leading people to believe they had enough data when they didn't.

4. The "Group" Effect: Friends Sitting Together

Real-world tests aren't just random questions; they are often grouped by topic (e.g., math, history, science).

The Analogy: Imagine you are testing if a new study method works. If you test it on a group of friends who all sit together and copy each other's answers, their results are correlated. You can't treat them as 100 independent people; they act more like 10 people.
The Finding: When the authors accounted for these "groups" (clusters) in the MMLU-Pro test, the number of "unresolved" (blurry) pairs went up.
- Without checking for groups: 4 pairs were blurry.
- With checking for groups: 6 pairs were blurry.
- Some pairs that looked like clear winners suddenly became too close to call.

5. The "Live Stream" Problem: Stopping Too Early

Leaderboards update constantly. New models are added every day.

The Analogy: Imagine watching a live stream of a race. If you stop the race the moment you see one runner pull ahead, you might be wrong. They might have just gotten a lucky break, and the other runner might catch up later.
The Finding: The paper tested what happens if you treat the leaderboard as a continuous live stream rather than a single snapshot. This "anytime-valid" testing is stricter. It flagged one more pair as unresolved that the standard test missed.

The Bottom Line

The paper doesn't say the models are bad or that the rankings are fake. It says: "We are declaring winners too quickly."

Many of the tiny gaps we see on leaderboards (like 0.5% or 1%) are currently too small to be statistically certain. The "microscope" (the resolution diagnostic) shows that we are often looking at static noise and calling it a signal.

The authors' advice: Before you put a headline on a model being "better," you need to check if your test has enough "resolution" to see that difference clearly. If the answer is no, the gap is just a guess, not a fact.

Technical Summary: Resolution Diagnostics for Paired LLM Evaluation

Problem Statement
Modern Large Language Model (LLM) leaderboards frequently rank models based on percentage-point gaps derived from shared-prompt benchmarks. While a headline gap (e.g., 78.3% vs. 77.5%) drives product decisions and narratives, the statistical claim that one model is meaningfully better than another depends on whether the benchmark size and data structure provide sufficient resolution to distinguish the gap from sampling noise.

The paper identifies a critical gap in current evaluation practices: many displayed pairwise rankings on public leaderboards fail to meet conventional paired-test resolution targets (specifically $\alpha=0.05$ and power $1-\beta=0.8$ ). The authors argue that standard power calculators and evaluation protocols often ignore the paired nature of shared-prompt benchmarks, leading to underpowered comparisons. Specifically, the paper highlights that:

On the Open LLM Leaderboard (OLL) v1, 11 of 40 pairwise comparisons are unresolved.
On the MMLU-Pro top-10, 4 of 9 adjacent-rank pairs are unresolved at the actual benchmark size ( $N=12,032$ ).
Common "shortcuts" for calculating required sample sizes in paired settings (e.g., applying a $(1-\rho)$ correction to unpaired Cohen's $h$ ) systematically underestimate the required sample size by a factor of approximately two in close-comparison regimes.

Methodology
The authors frame paired LLM evaluation as a hypothesis-testing problem and derive a resolution-reporting framework by inverting level- $\alpha$ , power- $(1-\beta)$ tests.

Resolution Ratio ( $q$ ): The primary diagnostic is defined as $q = N / N^\star$ , where $N$ is the actual benchmark size and $N^\star$ is the required sample size to detect an observed gap $\hat{\delta}$ with target power. A gap is considered "resolved" if $q \ge 1$ .
Paired Variance: For binary accuracy, the required sample size $N^\star$ is derived using the McNemar-Connor formula, which accounts for the variance of the paired difference $\sigma_D^2 = p_A q_A + p_B q_B - 2\rho\sqrt{p_A q_A p_B q_B}$ . This contrasts with unpaired formulas that inflate the required $N$ by ignoring the covariance $\rho$ .
The Shortcut Lemma (Lemma 1): The paper provides a sharp small-effect expansion characterizing the failure of the widely used "unpaired Cohen- $h$ plus $(1-\rho)$ " shortcut. The authors prove that for small effect sizes $\delta$ , the ratio of the shortcut estimate ( $n_h$ ) to the correct paired estimate ( $N^\star$ ) converges to $1/2$ with an explicit second-order constant $C(p, \rho)$ . This explains why tools that apply $(1-\rho)$ to a per-arm unpaired output systematically underestimate $N^\star$ by roughly half in close-comparison regimes.
Extensions: The framework integrates:
- Multiplicity Correction: Adjusting for family-wise error rates (Bonferroni/Holm) or false discovery rates (BH) when evaluating multiple pairs.
- Cluster Sensitivity: Scaling $N^\star$ by a design effect ($DE$) to account for intra-cluster correlation (e.g., subject categories in MMLU-Pro).
- Anytime-Valid Testing: Replacing fixed- $N$ thresholds with time-uniform boundaries derived from e-processes to handle continuous leaderboard updates.

Key Contributions

Theoretical Characterization of a Misuse Pattern: The paper provides a new lemma (Lemma 1) with an explicit constant that quantifies the deviation of the "unpaired + $(1-\rho)$ " shortcut from the correct paired sample size. It demonstrates that this shortcut underestimates the required sample size by a factor of two in the close-comparison regime, a pattern silently inherited by three of five common power calculators (Cohen 1988, G*Power, R pwr).
Empirical Resolution Diagnostics: The authors apply their framework to real leaderboard data (OLL v1 and MMLU-Pro). They demonstrate that a significant portion of displayed adjacent-rank gaps are unresolved at standard power targets.
- On OLL v1, 11/40 pairs are unresolved.
- On MMLU-Pro, the count of unresolved adjacent pairs rises from 4/9 (under IID assumptions) to 6/9 when accounting for real subject-level clustering.
Reporting Protocol and Tooling: The paper introduces a packaged protocol (llm-power) that exposes resolution diagnostics as a one-line call, allowing benchmark designers to report the minimum detectable effect (MDE) and the resolution ratio $q$ alongside headline gaps.

Results

Calculator Misuse: In a worked example with parameters $(p_A, p_B, \rho) = (0.65, 0.60, 0.30)$ , three popular tools returned a required sample size of $\approx 515$ when the correct paired $N^\star$ was $1,028$. The tools applied the $(1-\rho)$ correction to a single-arm variance rather than the paired-difference variance.
Real-World Impact:
- OLL v1: 28% of pairwise comparisons (11/40) failed the resolution target. All pairs with gaps $\le 2\%$ were unresolved.
- MMLU-Pro: Under the assumption of independent items, 4/9 adjacent pairs were unresolved. When applying cluster correction based on the 14 subject categories, the unresolved count increased to 6/9. Two pairs (Rank 4 vs. 5 and Rank 5 vs. 6) flipped from "comfortably resolved" to requiring sample sizes $>3N$ .
- Anytime-Valid Testing: Applying time-uniform thresholds for continuous monitoring further tightened the verdicts, rendering 5/9 MMLU-Pro pairs unresolved (one additional pair compared to fixed- $N$ tests).
Calibration: The framework's $N^\star$ prescriptions were validated via bootstrap subsampling on real data, confirming that the derived sample sizes achieve the target power ( $\approx 0.80$ ) empirically.

Significance and Claims
The paper claims that the assertion of a model being "better" than another is a statistical claim about the gap, not the gap itself. The significance of this work lies in shifting the focus from merely reporting a gap and a p-value to reporting the resolution of the benchmark.

The authors argue that many current leaderboard claims are unsupported by the data's statistical power, particularly in the "close-comparison regime" where adjacent models are ranked. By exposing the factor-of-two underestimation in common tools and demonstrating that clustering and multiplicity further degrade resolution, the paper provides a diagnostic framework to prevent overconfident conclusions. The work does not claim that the models are equal, but rather that the current benchmark sizes are insufficient to distinguish the observed gaps from noise under rigorous statistical standards. The paper concludes that a benchmark can have high construct validity yet still be too small for the specific hypothesis test at hand, necessitating a new reporting standard for LLM evaluations.