Technical Summary: Quantifying the Effect of Test Set Contamination on Generative Evaluations

Problem Statement

Test set contamination—the inadvertent inclusion of benchmark data in pretraining corpora—is a critical threat to the trustworthy evaluation of frontier AI systems. While the impact of contamination on discriminative evaluations (e.g., multiple-choice questions, classification) is well-documented, its effects on generative evaluations (where models must produce long-form solutions) remain underexplored. Generative tasks introduce unique complexities, including solution length, sampling temperature, and the requirement for coherent token chains, which may interact with memorization differently than short-choice tasks. This paper seeks to quantitatively assess these effects across the language model lifecycle to determine if contamination in generative settings leads to robust generalization or merely fragile memorization.

Methodology

The authors conducted a controlled, large-scale study focusing on the MATH benchmark (Hendrycks et al., 2021b), a dataset of competition math problems requiring generated solutions.

1. Pretraining Setup:

   • Models: Dozens of causal language models based on the Qwen 3 architecture were pretrained from scratch. Model sizes ranged from 34M to 344M parameters.
   • Contamination Protocol: Pretraining corpora were constructed by mixing a high-quality web crawl (FineWeb-Edu-Dedup) with varying numbers ($R$) of replicas of the MATH test set. The number of replicas ranged logarithmically from 0 (uncontaminated) to 3,162.
   • Compute Control: Training was performed at compute-optimal scales (20 tokens per parameter) and extended into "overtraining" regimes (up to 16x compute-optimal) using fresh, non-repeated data.
   • Fine-tuning: Some models underwent Supervised Fine-Tuning (SFT) on the MATH training set to observe interactions with pretraining contamination.

2. Evaluation Metrics:

   • Math Verify: The fraction of problems where the generated solution is mathematically equivalent to the gold answer. The authors identified and fixed a critical bug in the EleutherAI LM Evaluation Harness that previously underreported these scores (raising gold reference scores from ~70% to 100%).
   • Cross-Entropy Loss: The loss on the gold reference solutions, used to analyze scaling laws.
   • Robustness Checks: Models were tested on rephrased (same logic, different wording) and perturbed (same wording, different numbers) versions of the test set to distinguish memorization from generalization.

3. Inference Variables:

   • Experiments swept across sampling temperatures (0 to 1.5) and analyzed performance relative to solution token length.

Key Contributions and Results

1. Scaling Laws and the "Irreducible Error" Breach

The study confirms that performance (Math Verify scores) increases with both contamination level and model size. However, a fundamental breach in scaling laws was discovered:

• Single Replica Effect: Including even a single replica of the test set allows models to achieve a cross-entropy loss lower than the estimated irreducible error of training on the uncontaminated corpus.
• Economic Implication: Under standard scaling law assumptions, a single test set replica effectively simulates an "infinite" amount of pretraining compute relative to an uncontaminated corpus, allowing models to bypass the theoretical performance ceiling of the clean data.

2. Memorization vs. Generalization

Contamination-driven performance is not generalization.

• Verbatim Memorization: When tested on rephrased or perturbed problems, performance collapsed to baseline levels across all model sizes and contamination levels.
• Conclusion: The gains rely almost exclusively on the verbatim memorization of specific test sequences, not the acquisition of underlying mathematical reasoning.

3. Interaction with Further Training

The paper identifies a "dose-response" effect regarding subsequent training:

• Overtraining: Training on fresh data beyond the compute-optimal point dilutes contamination effects. For highly contaminated models, overtraining actually increases test set loss, as the fresh data reduces the proportion of contaminated tokens, weakening the memorized signal.
• Supervised Fine-Tuning (SFT): SFT on the training set has opposing effects depending on contamination levels:
  • Low Contamination: SFT improves test performance.
  • High Contamination: SFT degrades test performance. The authors conjecture that SFT forces the model to generalize, causing it to "forget" the memorized test samples. Since the memorized signal was the primary driver of performance in high-contamination models, this forgetting results in a net loss.

4. Inference-Time Regimes of Memorization

The authors characterize how sampling temperature and solution length modulate memorization, identifying three distinct regimes:

• Exponentially Fast Decoherence ($E > 0$): In the absence of meaningful contamination, the probability of generating a correct long sequence decays exponentially.
• Brittle Memorization ($E \approx 0, \alpha \le 1$): At intermediate contamination, the model can generate sequences, but the cumulative probability of success eventually vanishes as length increases.
• Deterministic Lock-In ($E \approx 0, \alpha > 1$): At high contamination, the model enters a regime where the certainty of the next token grows faster than the risk accumulates, allowing for the deterministic generation of long solutions.
• The "Truth Serum" Effect: High-temperature sampling ($\tau > 1$) acts as a "truth serum," decoupling contamination gains from robust generalization. It can instantly revert a model from "Deterministic Lock-In" back to "Brittle Memorization," causing performance to collapse to baseline levels.

5. Implementation Correction

The authors identified a critical bug in the EleutherAI LM Evaluation Harness (prior to v0.4.8) where the Math Verify scoring function stripped \boxed{} notation before parsing, leading to systematic underreporting of math performance. Fixing this raised gold reference scores from ~70% to 100%.

Significance and Claims

The paper claims to provide the first comprehensive quantification of contamination mechanics in the generative regime. Its primary significance lies in:

1. Redefining the Threat: While generative contamination inflates scores similarly to discriminative contamination, the underlying mechanism is distinct: it relies on fragile, verbatim memorization of long token chains rather than robust reasoning.
2. Economic Magnitude: The discovery that a single test set replica allows models to bypass the "irreducible error" barrier suggests that contamination can simulate infinite compute, fundamentally breaking the proxy of benchmarks for true capability.
3. Practical Levers: The study identifies sampling temperature and solution length as critical levers that differentiate memorization from generalization. High-temperature sampling and longer solution lengths expose the brittleness of contaminated models.
4. Counter-Intuitive Training Dynamics: The finding that SFT can hurt performance in highly contaminated models suggests that a drop in test accuracy after alignment training may be a positive signal of decontamination (forgetting the test set) rather than a negative "alignment tax."

The authors conclude that while contamination creates an "illusion of competence," it is a brittle illusion that can be dismantled through specific inference-time parameters and further training on fresh data.