Technical Summary: The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation

Problem Statement

Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, yet the objective evaluation of these abilities is severely compromised by data contamination. This occurs when benchmark data is included in a model's training corpus, artificially inflating performance metrics. While traditional detection methods rely on surface-level verbatim overlap, they fail against evasive (indirect) data contamination. In this stealthy scenario, malicious publishers or inadvertent training processes (e.g., via synthetic data or knowledge distillation) paraphrase benchmark questions. These alterations destroy exact token-level matches, rendering existing detectors based on n-gram overlap, embedding similarity, or likelihood-based metrics (like DPCC) ineffective. Furthermore, the opacity of pre-training corpora prevents direct inspection, leaving a critical blind spot in evaluating model capabilities.

Core Insight: Reasoning Masks Memorization

The authors uncover a fundamental phenomenon: intermediate reasoning steps actively mask underlying memorization. When an LLM is allowed to generate a full Chain-of-Thought (CoT), its reasoning ability acts as a confounder, making it difficult to distinguish between genuine problem-solving and memorized shortcuts. However, if a model has memorized a dataset (even a paraphrased version), it establishes a latent shortcut mapping from the semantics of a question ($x_i$) directly to the answer ($y_i$). This shortcut allows the model to generate correct answers without rigorous reasoning, but this behavior is often hidden when the model is permitted to "think" through intermediate steps.

Methodology: Zero-CoT Probe (ZCP)

To expose this latent memorization, the authors propose Zero-CoT Probe (ZCP), a novel black-box detection framework. The method operates without access to the model's training data or internal parameters.

1. Neutralizing Reasoning via CoT Truncation

ZCP forces the model to bypass the reasoning process entirely. By truncating the CoT and enforcing a Zero-CoT generation setting, the method compels the model to output the final answer directly.

• Implementation: For open-weight models, the prompt is pre-filled with a prefix (e.g., "The final answer is: \boxed{"). For closed-source API models, strict instructions are appended to the user query to forbid any content other than the boxed answer.
• Effect: This intervention neutralizes the "reasoning" factor. If a model relies on a memorized shortcut, it will succeed in Zero-CoT. If it relies on genuine reasoning, its performance should degrade significantly without the intermediate steps.

2. Isolating Memorization via Isomorphic Perturbation

High Zero-CoT performance could stem from either memorization or an intrinsic "superpower" (the ability to solve complex problems without explicit reasoning). To decouple these, ZCP introduces a cleaned reference dataset ($\tilde{D}_{eval}$) constructed via isomorphic perturbation.

• Process: The original benchmark questions are systematically altered by changing numerical values and textual context while strictly preserving the logical structure, mathematical depth, and order of magnitude.
• Logic: A model with genuine reasoning capabilities should perform similarly on the original and the perturbed reference data. However, a contaminated model, which relies on specific question-answer mappings, will fail on the reference data because the memorized shortcuts are broken by the new variables, leading to a statistically significant performance gap.

3. Quantifying Contamination Confidence

Moving beyond binary "clean vs. contaminated" classifications, ZCP introduces Contamination Confidence ($C_{cont}$), a metric that quantifies the severity of contamination.

• Statistical Framework: The method calculates the performance gap between the original benchmark and the reference dataset. It employs non-parametric bootstrap tests (for continuous metrics) or McNemar's test (for discrete metrics) to determine statistical significance.
• Bayesian Calibration: The resulting p-values are calibrated into Bayesian posterior probabilities using the Bayes Factor. Assuming a neutral prior ($\pi = 0.5$), the final score $C_{cont}$ ranges from 0.5 (no evidence of contamination) to 1.0 (definitive memorization).

Key Contributions

1. Discovery of the Masking Effect: The paper identifies that reasoning chains can actively obscure data contamination, acting as a severe confounder in detection.
2. Zero-CoT Probe (ZCP): A robust black-box method that combines CoT truncation with isomorphically perturbed reference data to detect both direct and evasive contamination.
3. Contamination Confidence: A new statistical metric that provides a continuous measure of contamination severity, addressing the limitations of binary detection.
4. Systematic Evaluation: The authors demonstrate the broad existence of data contamination across prominent closed-source and open-source models.

Experimental Results

The authors evaluated ZCP on existing models and specially fine-tuned models designed to simulate evasive contamination.

• Existing Models: On models like Qwen2.5-Math and DeepSeek-Math, ZCP detected contamination on standard benchmarks (GSM8K, MATH) with high confidence ($C_{cont} \approx 1.0$). Crucially, the method remained effective even when the models were tested on paraphrased versions of the benchmarks, where traditional detectors fail.
• Fine-Tuned Models: In controlled experiments where models were fine-tuned on paraphrased datasets (Omni-MATH, Multi-domain data), ZCP successfully identified the contamination ($C_{cont} \to 1.0$) while correctly identifying held-out uncontaminated data as clean ($C_{cont} \approx 0.5$).
• Real-World Auditing: The method was applied to state-of-the-art models (Qwen series, GPT-4o, GPT-5.1). Results indicated severe contamination in Qwen-2.5-Math and GPT-4o, while GPT-5.1 showed baseline scores, suggesting effective decontamination efforts.
• Ablation Studies: Experiments confirmed that without CoT truncation (Full-CoT setting), the performance gap between contaminated and clean data vanishes, and detection confidence drops to baseline levels, validating the necessity of the Zero-CoT approach.

Significance and Limitations

The paper argues that ZCP establishes a principled paradigm for detecting data contamination that transcends surface-level text matching. By forcing models to bypass reasoning, it exposes the latent shortcut mappings that define memorization. This is significant for the community as it offers a reliable tool to audit models against evasive strategies that currently game leaderboards.

The authors acknowledge a limitation: while ZCP can enforce Zero-CoT in open-weight models via direct token manipulation, applying this to closed-source APIs relies on prompt engineering. As models become increasingly optimized for step-by-step reasoning, such prompts may become less effective, representing a future challenge for auditing closed-source systems.