💬 NLP

The Flaw of Averages: Measuring Benchmark-Level Distributional Robustness

This paper introduces "Harmony," an entropy-based metric for measuring the distributional robustness of language model benchmarks, demonstrating that low-Harmony benchmarks often misrepresent broad model competence through uneven subdomain performance and recommending that Harmony be reported alongside aggregate accuracy to ensure more representative evaluations.

Original authors: Arda Uzunoglu, Tianjian Li, Daniel Khashabi

Published 2026-09-09

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Arda Uzunoglu, Tianjian Li, Daniel Khashabi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers rely on standardized tests, known as benchmarks, to measure how well language models think and learn. These tests are the yardsticks of the field, determining which models are considered the most capable and guiding the direction of future development. However, just as a student might ace a math exam but fail a history test, a language model can perform brilliantly on one type of question while struggling with another. When these varied results are rolled up into a single average score, the details can disappear. A high overall score might suggest a model is broadly competent, even if that competence is actually concentrated in just a few specific areas, masking significant weaknesses elsewhere. This phenomenon creates a misleading picture of progress, where the summary statistic hides the true distribution of a model's abilities.

A team of researchers at Johns Hopkins University has developed a new way to look past these averages to see the full picture. They introduced a diagnostic tool called HARMONY, which measures how evenly a model's performance is spread across the different topics within a single test. Instead of simply asking how many questions a model got right, this approach asks whether the model is getting them right across the board or if its success is clustered in a narrow set of subjects. By analyzing nineteen different benchmarks across five families of language models, the researchers found that many popular tests suffer from a lack of balance. In these cases, the aggregate scores often overstate a model's general intelligence because the test is dominated by topics where the model happens to be strong, while its failures in other areas are drowned out by the average.

The researchers demonstrated that this imbalance is not just a theoretical concern but has real consequences for how we evaluate artificial intelligence. They took several benchmarks and removed the questions that were overly similar to one another, effectively rebalancing the test to include a wider variety of topics. On tests that were already well-balanced, this change had little effect on the final scores. However, on tests that were unbalanced, the scores shifted dramatically. For example, a benchmark evaluating performance in a medically consequential domain exhibited substantial, often statistically significant, shifts in aggregate accuracy once the overrepresented items were pruned away, revealing that the model's earlier high scores were not as robust as they appeared. In contrast, a general knowledge test remained stable, suggesting its original score was a more faithful reflection of the model's true capabilities.

This work suggests that the current method of ranking models by a single number is often insufficient. The researchers argue that to truly understand a model's competence, we must look at how uniformly it performs across different domains. They found that larger models do not automatically become more balanced; in some families of models, increasing the size actually made the performance more concentrated in specific areas, while in others, it became more even. This indicates that simply making models bigger does not guarantee they will be more generally capable. The study concludes that whenever a benchmark score is reported, it should be accompanied by a measure of this distributional balance. Only by acknowledging where a model is strong and where it is weak can the scientific community avoid the trap of believing that an average score represents a broad, reliable intelligence.

Technical Summary: The Flaw of Averages: Measuring Benchmark-Level Distributional Robustness

Problem Statement

Benchmarks are central to measuring progress in language models (LMs), yet aggregate scores often obscure substantial variation across subdomains. This phenomenon, termed the "Flaw of Averages," can make models appear broadly competent despite having concentrated strengths and weaknesses in specific areas. While prior work has addressed benchmark robustness regarding redundancy, data contamination, and distribution shifts, less attention has been paid to whether an aggregate score faithfully represents performance across the internal subdomains a benchmark is intended to cover. The authors identify a specific failure mode where aggregate benchmark scores misrepresent broad competence by masking uneven performance distributions.

Methodology

Benchmark HARMONY

To quantify this issue, the authors introduce HARMONY, an entropy-based diagnostic metric that measures how uniformly a model's performance is distributed across a benchmark's subdomains.

Semantic Partitioning: The benchmark $B$ is partitioned into semantic subdomains (clusters) $G = \{A_1, \dots, A_k\}$ . Unlike fixed human-defined taxonomies, the authors induce these partitions using predictive similarity.
- Predictive Similarity: Defined as $S(x_i, x_j) = \exp\left(-\frac{\tau}{2} [D_{KL}(\bar{p}_f(x_i) \| \bar{p}_f(x_j)) + D_{KL}(\bar{p}_f(x_j) \| \bar{p}_f(x_i))]\right)$ , where $\bar{p}_f$ is the averaged predictive distribution of model $f$ . This measures how similarly a model behaves on two inputs based on their predictive distributions.
- Clustering: Spectral clustering is applied to the affinity matrix derived from predictive similarity to generate the subdomains. The number of clusters $k$ is selected via the silhouette score.
HARMONY Calculation:
- For each subdomain $A_i$ , performance $\Psi(f; A_i)$ (e.g., accuracy) is measured.
- A weighted mean performance $\mu$ is calculated.
- Performance masses $p_i$ are derived using a Gaussian kernel that penalizes deviations from the mean: $p_i \propto w_i \exp\left(-\left(\frac{\Psi(f; A_i) - \mu}{b}\right)^2\right)$ , where $w_i$ is the relative size of the subdomain.
- HARMONY is defined as the normalized Shannon entropy of these masses: $H(G_f) = -\frac{1}{\log k} \sum p_i \log(p_i + \epsilon)$ .
- Interpretation: High HARMONY (close to 1) indicates performance is uniformly distributed across subdomains, meaning the aggregate score is representative. Low HARMONY indicates performance is concentrated in a few subdomains, suggesting the aggregate score may be misleading.
Benchmark-Level Analysis:
- The authors compute HARMONY for 19 MCQA benchmarks across 36 models from five families (Llama 3, Qwen3, Gemma 3, Phi-3, OLMo 2).
- Benchmarks are plotted in a mean-variance plane defined by $\mu_H(B)$ (average HARMONY across models) and $\sigma^2_H(B)$ (variance of HARMONY across models). High mean and low variance indicate a benchmark is robustly representative.

Validation

The authors validate their approach using RedundantQA, a synthetic benchmark with a known ground-truth structure consisting of four domains (Biology, History, Economics, Popular Culture). Each item includes a reference question, two true-similar paraphrases, and two false-similar distractors. Results show a strong positive correlation between estimated HARMONY and ground-truth HARMONY, confirming the method's ability to distinguish between harmonious and non-harmonious distributions.

Key Results

Substantial Variation in Robustness: There is significant variation in benchmark-level distributional robustness. Benchmarks like BoolQ exhibit high HARMONY and remain stable even when rebalanced. In contrast, benchmarks like PubMedQA show low HARMONY; pruning overrepresented subdomains to increase HARMONY leads to substantial, often statistically significant, shifts in aggregate accuracy.
Rebalancing Effects: Low-HARMONY benchmarks are fragile. When pruned to remove overly similar items (increasing HARMONY), their aggregate scores shift significantly, suggesting their original scores were driven by a subset of subdomains. High-HARMONY benchmarks remain stable under similar pruning.
Scaling Trends are Family-Specific:
- Model Size: The relationship between parameter count and HARMONY varies by family. Qwen and Llama families show a negative correlation (larger models concentrate performance on fewer subdomains), while Gemma and OLMo show a positive correlation (larger models distribute performance more evenly).
- Pre-training Tokens: For OLMo models, increasing the pre-training token budget initially causes a dip in HARMONY followed by a steady rise, even when aggregate accuracy changes only slightly. This suggests additional training leads to more uniform performance distribution.
Mean-Variance Landscape: Benchmarks in the upper-left of the HARMONY mean-variance plane (high mean, low variance) provide more representative and stable evaluations. Benchmarks in the lower-right signal that conclusions may be fragile or model-dependent.

Significance and Claims

The paper claims that aggregate benchmark scores can systematically misrepresent broad competence when performance is unevenly distributed across subdomains. The authors argue that HARMONY should be reported alongside aggregate accuracy as a diagnostic of benchmark representativeness.

Diagnostic Utility: HARMONY is not a measure of overall benchmark quality (it does not detect contamination or annotation artifacts) but specifically diagnoses whether an aggregate score is a valid summary of performance across internal subdomains.
Complementary Role: The metric is intended to complement, not replace, subdomain-level analysis. While subdomain breakdowns reveal where strengths and weaknesses lie, HARMONY summarizes whether the aggregate score is broadly representative.
Recommendation: The authors recommend that when interpreting claims about broad model competence, researchers should consider the HARMONY of the underlying benchmarks to ensure the aggregate scores are not artifacts of uneven performance distribution.

Limitations

The authors acknowledge several limitations:

The study focuses primarily on multiple-choice benchmarks (MCQA) where accuracy is unambiguous, though they validate on open-ended tasks.
HARMONY depends on the induced subdomain partition, which may be sensitive to the similarity metric and clustering parameters.
High HARMONY does not guarantee a benchmark is free from contamination or narrow coverage; it only indicates uniformity of performance distribution.
The empirical findings are conditioned on the specific benchmarks and open model families studied.