Benchmarks Are Not That Out of Distribution: Word Overlap Predicts Performance

Technical Summary: Benchmarks Are Not That Out of Distribution: Word Overlap Predicts Performance

1. Problem Statement

A fundamental assumption in Large Language Model (LLM) research is that standard benchmarks (e.g., MMLU, ARC, HellaSwag) are "out-of-distribution" (OOD) relative to pre-training corpora. This assumption implies that high benchmark scores reflect a model's ability to generalize and reason rather than mere pattern matching. Consequently, researchers often struggle to identify which pre-training datasets truly improve "intelligence" versus those that simply increase statistical overlap with evaluation sets. The authors investigate whether benchmark performance is actually driven by the degree of statistical pattern overlap (word-level similarity) between the training and evaluation data.

2. Methodology

The researchers propose a framework to quantify the "distributional distance" between pre-training corpora and benchmarks using two primary word-level proxy metrics:

• Word-Level Unigram Cross-Entropy ($H(P_{eval}, P_{pre})$): This measures how well the word frequency distribution of the pre-training corpus predicts the word frequency distribution of the benchmark. It is a tokenizer-agnostic, information-theoretic measure of marginal distribution overlap.
• Word Frequency Statistics: Since unigram cross-entropy only considers normalized probabilities, the authors also track absolute word counts to account for the "strength" of the learning signal (frequency-based convergence).

Experimental Setup:

• Models: LLaMA architectures with scales ranging from 400M to 3.36B parameters.
• Datasets: Four pre-training corpora (FineWeb-Edu, DCLM, C4, and OpenWebText) across varying scales (8.5B to 60B tokens).
• Benchmarks: 10 representative zero-shot benchmarks (including ARC, MMLU, HellaSwag, SciQ, etc.).
• Controlled Variables: They performed controlled experiments by varying dataset subsets and model scales to isolate the effects of word overlap from model size and total token count.

3. Key Contributions

• Identification of a Robust Predictor: The paper demonstrates that word-level unigram cross-entropy is a highly reliable predictor of benchmark performance.
• Introduction of Tokenizer-Agnostic Metrics: By using word-level unigrams instead of subword tokens, the authors avoid "tokenization bias," where segmentation artifacts might be mistaken for distributional shifts.
• Decomposition of Overlap: They distinguish between coverage (whether the words exist in the training set) and entropy (how the probability mass is distributed among those words), showing that entropy is the primary driver of performance differences.
• Boundary Definition: The work identifies specific types of tasks (Grammar and Math) where word-level overlap fails to explain performance, thereby helping to define the boundary between "statistical pattern matching" and "true reasoning."

4. Key Results

• Inverse Correlation: A consistent negative correlation exists between unigram cross-entropy and benchmark scores. Lower cross-entropy (higher overlap) almost always leads to higher benchmark performance.
• Stability Across Scales: This relationship holds true across different model sizes (400M to 3B) and different training subset sizes (8.5B to 60B tokens).
• The "Frequency" Effect: While unigram cross-entropy explains the relative performance between different datasets, increasing the total number of tokens (word frequency) improves absolute performance by strengthening the learning signal for both frequent and long-tail vocabulary.
• Exceptions to the Rule:
  • Grammar (BLiMP): Performance is driven by higher-order contextual/syntactic representations rather than word frequency.
  • Math (MathQA): Performance is constrained by algorithmic reasoning and compositional rules rather than surface-level pattern compression.
  • Multilingualism: Word-level overlap does not consistently predict zero-shot performance in unseen languages, suggesting multilingual generalization involves more complex cross-lingual mapping.

5. Significance and Implications

• Re-evaluating Benchmarks: The study challenges the "OOD assumption." It suggests many standard benchmarks are actually "weakly OOD," meaning they may be measuring statistical affinity rather than abstract reasoning.
• Data Selection Strategy: The findings provide a principled, information-theoretic way to evaluate the quality of pre-training data subsets. Instead of heuristic filtering, researchers can use unigram cross-entropy to predict how a dataset will impact downstream tasks.
• Warning Against "Benchmark Hacking": The authors warn that one could theoretically "hack" benchmarks by synthetically increasing the frequency of benchmark-relevant words in pre-training data, which would inflate scores without necessarily improving reasoning.
• Future Research: The paper calls for the design of more "diagnostic" benchmarks—evaluations that are truly decoupled from the statistical distributions of web-scale pre-training data.