Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation

1. Problem Statement

Current Large Language Model (LLM) evaluation relies heavily on aggregate accuracy metrics (e.g., "Model B is 2% better than Model A"). This approach masks the underlying dynamics of model updates:

• The "Net Residual" Problem: An aggregate gain can result from uniform small improvements, large improvements on a few items, or a mix of significant improvements and significant deteriorations that cancel each other out.
• Measurement Noise: Standard single-shot (greedy) evaluations treat items as binary (correct/incorrect) without accounting for stochastic response variability. A "flip" from correct to incorrect might be noise rather than a genuine regression, while a genuine shift in probability might not cross the binary threshold.
• Lack of Item-Level Rigor: While statistical methods exist for aggregate scores (e.g., confidence intervals), there is no principled framework to distinguish genuine item-level change from measurement error at the individual question level.

2. Methodology

The study adapts the Reliable Change Index (RCI), a standard tool in clinical psychology for determining if a patient's score change exceeds measurement error, to the context of LLM benchmarking.

• Experimental Design:

  • Dataset: 2,000 items from the MMLU-Pro benchmark (500 items each across Physics, Law, Psychology, and Economics).
  • Model Pairs:
    1. Llama 3 (8B) vs. Llama 3.1 (8B): A minor update.
    2. Qwen 2.5 (7B) vs. Qwen 3 (8B): A generational update.
  • Sampling Protocol: Each item was generated K=10 times per model at a temperature of T=0.7 with independent conversation contexts. This creates a distribution of accuracy ($p$) for each item rather than a single binary outcome.
  • Hardware: All models run at Q5_K_M quantization on AMD hardware.

• Statistical Framework:

  1. Reliability Estimation ($r_{xx}$): Calculated via split-half reliability (Spearman-Brown corrected) across the 10 samples to determine the stability of the model's response distribution.
  2. Standard Error of the Difference ($S_{diff}$): Computed as $\sqrt{SEM_{v1}^2 + SEM_{v2}^2}$, where $SEM = SD(p) \times \sqrt{1-r_{xx}}$.
  3. Reliable Change Index (RCI):
     $$RCI(i) = \frac{p(i, v2) - p(i, v1)}{S_{diff}}$$
     • Reliable Improvement: $RCI > 1.96$($p < .05$).
     • Reliable Deterioration: $RCI < -1.96$.
     • No Reliable Change: $|RCI| \le 1.96$.
  4. Exclusion Criteria: Items with floor/ceiling effects (0% or 100% accuracy in both models) were excluded as they cannot exhibit stochastic change.

3. Key Results

A. Aggregate vs. Item-Level Dynamics

• Full Benchmark: On the full 2,000-item set, the majority of items showed no reliable change (79% for Llama, 72% for Qwen). This is largely driven by floor/ceiling items (trivial or impossible questions).
• Analysable Subset: Among items with intermediate difficulty (where stochastic variation exists), the pattern shifted dramatically to bidirectional churn:
  • Llama: 34% improved, 28% deteriorated.
  • Qwen: 47% improved, 39% deteriorated.
  • Effect Size: The median absolute change ($|\Delta p|$) for reliably changed items was large: 0.50 for Llama and 0.90 for Qwen (representing 5–9 additional correct answers out of 10 samples).

B. Asymmetry by Difficulty (Regression to the Mean)

• Low-Accuracy Items: Predominantly improved (e.g., 100% of deteriorated items in Llama were high-accuracy; 100% of improved items were low-accuracy).
• High-Accuracy Items: Predominantly deteriorated.
• This suggests a "regression toward the middle" effect, where items near the performance floor can only go up, and items near the ceiling can only go down.

C. Domain-Specific Reversals

Aggregate gains masked specific domain regressions:

• Llama 3.1: Gained in Economics, Law, and Psychology but lost in Physics.
• Qwen 3: Gained in Economics, Physics, and Psychology but lost in Law.
• The specific domains that deteriorated differed between model families, indicating structured, family-specific training effects rather than random noise.

D. Failure of Greedy (Single-Shot) Evaluation

Comparing the RCI results against standard greedy ($T=0$) single-shot evaluation revealed significant flaws in the latter:

• False Negatives: Greedy evaluation missed 42% of items that RCI identified as reliably changed (probability shifts that didn't cross the binary threshold).
• False Positives: Greedy evaluation flagged 25% of items as changed when RCI classified them as unchanged (noise-driven flips).

4. Key Contributions

1. Methodological Innovation: Successfully adapted the Reliable Change Index (RCI) from clinical psychology to LLM evaluation, providing a statistical framework to distinguish signal from noise at the item level.
2. Decomposition of Aggregate Scores: Demonstrated that aggregate accuracy gains are often the net residual of opposing movements (large improvements canceling out large deteriorations), rather than uniform progress.
3. Quantification of Churn: Introduced the concept of "Churn Rate" (the proportion of items with reliable change in either direction) as a critical metric alongside aggregate accuracy.
4. Critique of Greedy Evaluation: Provided empirical evidence that single-shot binary evaluation is insufficient for detecting genuine model behavior changes, recommending $K \ge 3$ stochastic samples as a low-cost alternative.

5. Significance and Implications

• For Model Developers: Aggregate metrics are incomplete. A model update might improve performance on 40% of tasks while severely degrading it on another 40%, resulting in a misleading "net gain." Developers must monitor bidirectional churn to ensure updates do not introduce catastrophic regressions in specific domains.
• For Evaluators: The paper argues for a shift in reporting standards. Benchmarks should report Churn Rates and Domain-Specific Reversals in addition to aggregate accuracy.
• For the Field: This work bridges the gap between engineering (negative flips) and psychometrics (measurement theory), suggesting that LLM evaluation requires the same rigor regarding measurement error as human psychological testing.

Conclusion: The paper concludes that while aggregate accuracy gains are real, they often conceal significant instability. The "net gain" is frequently a small residual of large, opposing item-level movements. To truly understand model evolution, the community must move "beyond the mean" to analyze the structure of change within the model family.