Test of Time: Rethinking Temporal Signal of Benchmark Contamination

1. Problem Statement

The reliability of Large Language Model (LLM) evaluation is threatened by benchmark contamination, where models achieve high scores by memorizing evaluation data from their training corpora rather than demonstrating genuine reasoning.

• Current Detection Method: A widely accepted method for detecting contamination is temporal analysis. This involves comparing model performance on questions released before a model's training cutoff date (pre-cutoff) versus those released after (post-cutoff).
• The Assumption: If a model performs significantly worse on post-cutoff questions (a phenomenon termed post-cutoff performance decay), it is interpreted as evidence that the model memorized pre-cutoff data.
• The Conflict: Recent studies (e.g., RealMath) observed a lack of post-cutoff decay on LLM-generated questions derived from time-stamped sources (like arXiv), whereas traditional benchmarks (e.g., LiveCodeBench) show clear decay. This discrepancy suggests that the "temporal signal" of contamination may be fragile and dependent on how benchmark questions are constructed, rather than solely on the presence of contamination.

2. Methodology

The authors employ a multi-faceted approach combining large-scale empirical evaluation, comparative benchmark construction, and mechanistic interpretability analysis.

A. Large-Scale Temporal Analysis (LLM-Generated QA)

• Dataset: Retrieved 20,277 arXiv papers (Math and Physics) spanning 26 months (May 2023 – June 2025).
• Question Generation: Used o4-mini to generate multi-step reasoning questions from the papers. These were filtered for quality, resulting in 1,643 LLM-generated questions.
• Models Evaluated: 8 frontier models across 4 families (DeepSeek, OpenAI-o3/o4, Gemini, Llama) with varying knowledge cutoff dates.
• Goal: To verify if the lack of post-cutoff decay is a consistent pattern across different architectures and domains.

B. Comparative Benchmark Construction

To isolate the variable of "question construction," the authors created alternative benchmarks from the same source materials:

1. Cloze (Fill-in-the-blank) Questions: Generated by masking key semantic terms from the original paper abstracts (preserving the exact text structure).
2. LiveCodeBench Transformation: Applied LLM-based semantic transformations to existing coding problems while keeping the underlying solution code identical.
3. Wiki-based QA: Transformed FreshQA-style multiple-choice questions derived from Wikipedia events.

C. Mechanistic Interpretability (Influence Functions)

• Tool: Used Influence Functions (specifically Kronfluence) to trace which training documents most influence a model's prediction.
• Setup: Used OLMo2-7B-Instruct (an open-data model with a known training set) containing 40 provably contaminated arXiv papers.
• Task: For each paper, generated both a Cloze question and an LLM-generated question. The system then ranked the top 100 most influential training documents for each query to see if the original source paper was identified.

3. Key Contributions & Results

A. Absence of Decay in LLM-Generated Benchmarks

• Finding: Across 8 models and 2 domains (Math/Physics), no systematic post-cutoff performance decay was observed on LLM-generated questions. In fact, performance slightly increased post-cutoff (+2.19 percentage points on average, statistically significant).
• Implication: The "temporal signal" is not a universal indicator of contamination when questions are synthetically generated.

B. Presence of Decay in Cloze/Verbatim Benchmarks

• Finding: When the same source papers were converted into Cloze questions (verbatim extraction), a clear post-cutoff performance decay emerged across all models.
• Comparison:
  • LLM-generated QA: No decay (or slight improvement).
  • Cloze QA: Significant decay (e.g., -3.83% to -6.60% depending on the metric and model).
• Conclusion: The LLM generation process transforms the underlying material enough that models can no longer "recall" it via memorization, even if the source was in the training data.

C. Generalizability Across Domains

• LiveCodeBench: Original LiveCodeBench showed clear decay. After LLM transformation (rephrasing), the decay disappeared.
• Wiki-based QA: Original MCQs showed decay (up to -7.59% for GPT-4o-mini). Transformed variants showed significantly reduced or reversed decay.
• Significance: The sensitivity of the temporal signal is a general phenomenon, not limited to academic papers.

D. Mechanistic Evidence (Influence Functions)

• Cloze Questions: The model successfully identified the source paper as the Top-1 most influential document 77.5% of the time (100% within Top-3). This confirms the model is "recalling" the specific training instance.
• LLM-Generated Questions: The success rate dropped drastically to 17.5% (Top-1) and 25.0% (Top-3).
• Interpretation: The transformation process breaks the direct link between the query and the specific training token sequence, rendering influence functions (and by extension, simple memorization) ineffective.

4. Significance and Implications

• Re-evaluating Contamination Signals: The paper argues that post-cutoff performance decay is a highly sensitive and potentially unreliable signal for benchmark contamination. Its presence or absence depends heavily on the benchmark construction methodology (verbatim vs. transformed) rather than just the presence of data leakage.
• The "Transformation" Effect: Simple LLM-based rephrasing or synthesis can effectively "clean" a contaminated benchmark by breaking the memorization link, making the model appear to reason rather than recall.
• Recommendations for the Community:
  1. Methodological Transparency: Researchers must document exactly how benchmarks are constructed, as this dictates the validity of temporal analysis.
  2. Robust Detection: Temporal analysis should not be used in isolation. It must be validated against other methods (e.g., influence functions, paraphrasing variants).
  3. Beyond Scalar Scores: Future evaluation should account for the fluctuation in performance when problems are transformed, as this reveals the brittleness of reasoning vs. memorization.

Conclusion

The paper fundamentally challenges the assumption that "post-cutoff decay = contamination." Instead, it demonstrates that LLM-generated transformations can mask contamination signals by altering the input distribution enough to prevent direct memorization, even when the source material is present in the training data. This calls for more robust, multi-modal approaches to evaluating AI capabilities and detecting data leakage.