Technical Summary: LogitTrace

Problem Statement

Large Language Models (LLMs) are frequently evaluated on challenging benchmarks (e.g., AIME, Math500). A critical issue is benchmark contamination, where test data or its variants appear in the training corpus. This allows models to reproduce memorized solutions while appearing to reason, thereby inflating performance metrics and obscuring true generalization capabilities.

Existing detection methods primarily rely on:

1. Surface overlap: Lexical similarity or exact completion matches.
2. Completion behavior: Whether the model can reconstruct the target text from partial prompts.
3. Final-output statistics: Perplexity, output likelihood, or distributional agreement.

These approaches suffer from significant robustness issues. When benchmark items are rephrased, translated, or structurally perturbed, surface-level signals degrade rapidly, often failing to detect contamination that persists despite the lack of verbatim overlap.

Methodology: LogitTrace

The authors propose LogitTrace (Layerwise Logit Trajectories), a framework that detects contamination by analyzing the temporal dynamics of model decision-making across network layers, rather than relying solely on the final output.

Core Concept

The hypothesis is that memorization affects how an answer forms, not just what the answer is. Contaminated examples tend to exhibit earlier commitment to a specific solution across layers, whereas clean examples show a more gradual accumulation of evidence.

Technical Pipeline

1. Representation Extraction (Logit Lens):

   • For a given input $x$, the model's hidden states $h^{(\ell)}$ at each layer $\ell$ are projected through the unembedding matrix ($W_U$) to obtain vocabulary distributions $p^{(\ell)}$.
   • Since the focus is on numerical reasoning, the distribution is restricted to digit tokens ($0-9$).
   • For each layer, the authors compute:
     • Digit probabilities $p^{(\ell)}(d)$.
     • Digit entropy $H^{(\ell)}$.
     • Maximum digit probability $m^{(\ell)}$.
     • First-order temporal differences ($\Delta$) of these metrics.

2. Feature Construction:

   • These metrics form a multichannel sequence trajectory $X \in \mathbb{R}^{C \times T}$, where $C=24$ channels (12 raw features + 12 difference features) and $T$ is the number of layers.
   • This trajectory captures the evolution of the model's preference for specific digits as it processes the input.

3. Detection Model:

   • A lightweight 1D Convolutional Neural Network (CNN) is trained to classify these trajectories as "contaminated" or "clean."
   • The architecture includes three convolutional layers with ReLU activation, global average pooling, and a binary classifier head.
   • The detector is trained on an operational contamination dataset (e.g., AIME 1983–2024 as contaminated vs. AIME 2025 as clean) and remains fixed during evaluation.

Key Contributions

1. LogitTrace Framework: Introduction of a detector that identifies contamination-associated decision dynamics from layerwise logit trajectories, moving beyond surface-level cues.
2. Early Commitment Pattern: Empirical demonstration that contaminated examples consistently show earlier commitment to answers across layers compared to clean examples, a signal that remains robust under rephrasing, translation, and perturbation.
3. Controlled Causal Evidence: Validation via LoRA injection experiments. By fine-tuning models on target samples with varying LoRA ranks, the authors show that repeated exposure induces the same trajectory patterns observed in naturally contaminated examples, and the detector's confidence scores increase monotonically with the strength of the fine-tuning.

Experimental Results

The method was evaluated on four open-source LLMs (Qwen-2.5-7B, Qwen-2.5-Math-7B, Llama-3.1-8B, Qwen-3-8B) across four text distributions: Original, Rephrased, Translated, and Perturbed.

• Robustness: While completion-based baselines (e.g., exact match, TS-Guessing) saw contamination detection rates drop below 20% on rephrased or translated inputs, LogitTrace maintained high detection rates (e.g., 63.1% on rephrased Qwen-2.5-7B vs. 16.7% for baselines).
• False Positives: LogitTrace significantly reduced false positives on clean samples compared to distribution-based methods (e.g., perplexity or output distribution), which often mislabeled clean rephrased inputs as contaminated.
• Ablation Studies:
  • Layer Depth: Signals emerge early; using the first 1/3 of layers yielded F1 scores of ~68%, while full depth achieved ~95%.
  • Architecture: The 1D-CNN outperformed MLP and Transformer baselines, particularly in maintaining a low False Positive Rate (5.56% vs. 16.67% for Transformer).
• External Generalization: When applied to the MMLU mathematics subset (zero-shot), LogitTrace identified ~95% of samples as "seen-like." This group exhibited significantly higher model performance (Top-1 accuracy +0.33) compared to the "not-seen" group, suggesting the detector captures genuine familiarity.

Significance and Claims

The paper claims that LogitTrace provides evidence beyond surface overlap and final-output confidence. By analyzing the internal dynamics of the model, it offers a more robust lens for studying memorization-like behavior.

• Causal Link: The LoRA injection experiments provide controlled causal evidence that the trajectory patterns are a direct result of model exposure to training data, rather than surface-level artifacts.
• Practical Utility: The framework offers a tool for reassessing benchmark contamination, particularly in scenarios where standard detection methods fail due to paraphrasing or translation.
• Limitations: The authors note that LogitTrace relies on operational labels (as true training corpora are unavailable) and requires access to intermediate layer representations, limiting its application to closed-source models that only expose final outputs. Additionally, the current implementation focuses on numerical reasoning; extension to open-ended tasks would require new feature definitions.