Technical Summary: When LLMs Deteriorate Significantly

Problem Statement

The widespread adoption of Large Language Models (LLMs) has spurred extensive research into inference optimization techniques, ranging from theoretically lossless methods (e.g., efficient kernels, speculative decoding) to lossy methods (e.g., quantization, sparsity). A critical challenge in this domain is distinguishing genuine model degradation from benign statistical fluctuations or numerical errors.

Even at zero temperature, modern LLMs are not fully deterministic due to the non-associative nature of floating-point arithmetic (e.g., $(a+b)+c \neq a+(b+c)$) and variations in hardware or serving frameworks. These factors cause the same model to generate different responses across runs, leading to "flips" in binary accuracy metrics, even with theoretically lossless optimizations. Consequently, observed accuracy deviations in benchmarks with finite samples may be attributable to statistical fluctuations rather than actual model degradation. Existing evaluation practices often rely on arbitrary thresholds (e.g., a 2% accuracy drop or a 5% flip rate) or treat accuracy estimates as independent, failing to account for the correlation arising from evaluating both the baseline and the optimized model on exactly the same dataset. This leads to either false positives (flagging lossless optimizations) or false negatives (overlooking subtle degradations).

Methodology

The work proposes a statistically rigorous hypothesis testing framework based on the McNemar Test to detect model degradation with controlled false-positive rates.

1. Statistical Foundation

Instead of comparing aggregated accuracy metrics, the authors argue for contrasting model scores based on individual samples. They define a bivariate random variable representing the joint outcomes of a baseline model $M$ and an optimized model $\tilde{M}$ on a sample $x$. The outcomes are categorized into a contingency table:

• $a$: Both models fail.
• $b$: Baseline succeeds, optimized model fails (degradation).
• $c$: Baseline fails, optimized model succeeds (improvement).
• $d$: Both models succeed.

The authors define the degradation probability $q_{\downarrow}$ as the conditional probability that the baseline model succeeds, given that the two models disagree ($b$ vs. $c$). Under the null hypothesis $H_0$ (no degradation), $q_{\downarrow} = 0.5$. The alternative hypothesis $H_A$ is $q_{\downarrow} > 0.5$.

2. Exact One-Sided McNemar Test

The proposed test statistic is the empirical degradation probability $\hat{q}_{\downarrow} = b / (b+c)$. Since $b$ follows a Binomial distribution $Binomial(b+c, q_{\downarrow})$, the authors use an exact one-sided binomial test to calculate p-values. This approach:

• Focuses exclusively on samples where the models disagree ($b+c$), as agreeing samples provide no signal regarding relative performance.
• Avoids the overestimation of variance that occurs when accuracy estimates are treated as independent.
• Provides exact p-values rather than relying on asymptotic approximations for small sample sizes.

3. Aggregation Across Tasks

To handle evaluations across multiple benchmarks ($T$ tasks), the work introduces three aggregation strategies:

• Pooled Test: Aggregates all $b$ and $c$ counts across tasks into a single contingency table. This is robust against task subgroups and highly sensitive when degradation is homogeneous across tasks.
• Max Drop Test: Identifies the task with the most significant standardized degradation statistic ($z_{max}$). It uses a Monte Carlo simulation to derive the null distribution and is effective for detecting degradation in a single specific task.
• Fisher Method: Combines p-values from individual tasks using the statistic $\chi^2 = -2 \sum \ln(p_i)$. This offers a balance between the Pooled and Max Drop approaches.
• Combined Decision: The framework flags degradation if any of the three tests rejects the null hypothesis at significance level $\alpha$.

4. Dataset Pruning for Efficiency

The authors propose a method to reduce dataset sizes for faster testing. By running the baseline model at a finite temperature to identify examples that are "unstable" (likely to flip), they suggest removing stable examples (those that always succeed or always fail) from the evaluation set. This increases the flip probability $p_{\updownarrow}$ in the remaining dataset, thereby improving the signal-to-noise ratio without losing relevant degradation signals.

5. Generalization to Non-Binary Scores

For tasks with non-binary metrics (e.g., ROUGE), the work extends the framework using permutation tests. This involves calculating the mean score difference and comparing it against a null distribution generated by randomly flipping the signs of the differences, ensuring control of the Type I error rate for any sample size.

Key Results

The authors implemented their framework based on the open-source lm-eval harness and conducted extensive experiments with the Llama-3.1 8B, Llama-3.3 70B, and Mistral Small models.

• Detection of Subtle Degradations: The framework successfully detected empirical accuracy degradations as small as 0.3% (e.g., in the KV-FP8 variant of the 70B model) with high statistical confidence ($p < 0.05$), whereas baseline criteria (2% drop or 5% flip rate) failed to flag them.
• Validation of Lossless Optimizations: The tests correctly identified that theoretically lossless changes (e.g., reruns, different hardware, various inference engines) resulted in no statistically significant degradation, despite observed non-zero flip rates (1.3%–2.8%) and minor accuracy fluctuations.
• Analysis of False Positives: The study demonstrated that relying solely on flip probability (as suggested by prior work) can lead to false positives. For instance, a model with an 8.71% flip rate showed no accuracy degradation, while the proposed statistical test correctly cleared it.
• Impact of Quantization: The framework flagged significantly severe degradations in INT4 quantized models (w4a16) and a pruned/distilled variant, with p-values effectively zero.
• Efficiency: The dataset pruning approach reduced the MMLU-Pro sample size by nearly 50% while retaining the majority of "flipping" examples, significantly accelerating evaluation without sacrificing detection power.

Significance and Claims

The work claims to provide the first rigorous statistical framework specifically designed to detect LLM model degradation in the presence of numerical noise and uncertainty in finite samples.

• Statistical Rigor: It moves beyond heuristic thresholds (e.g., "2% drop") to a principle-guided hypothesis testing approach that controls the false-positive rate.
• Practical Utility: The authors provide a publicly available implementation integrated into lm-eval, enabling the community to apply these tests on standard benchmarks.
• Nuanced Insight: The work highlights that "flips" are a natural consequence of floating-point arithmetic in modern LLMs and do not inherently indicate degradation. Instead, the direction of the flip (degradation probability) is the critical metric.
• Sensitivity: The framework is sensitive enough to detect degradations that are statistically significant but practically small (e.g., 0.3%), enabling more informed decisions regarding model optimizations.

The authors conclude that proper statistical treatment of accuracy is essential for the reliable evaluation of optimized LLMs, particularly as the field moves toward mixed precision and highly compressed models where numerical errors are inevitable.