Technical Summary: Search-Time Contamination in Deep Research Agents

Problem Statement

Public benchmarks have traditionally served as the standard for evaluating the reasoning capabilities of Large Language Models (LLMs). However, the emergence of Deep Research Agents—autonomous systems that actively navigate the web, decompose queries, and synthesize evidence—introduces a critical vulnerability known as Search-Time Contamination (STC).

Unlike static models, deep research agents perform inference by retrieving information from the open web. Since many benchmark instances (questions, contexts, and ground-truth answers) are curated from web corpora or hosted on public platforms (e.g., HuggingFace, GitHub, educational forums), agents may inadvertently retrieve these artifacts during the search process. This allows the agent to bypass the intended reasoning chain and simply extract the answer, leading to performance inflation that does not reflect genuine reasoning ability. The paper argues that existing evaluation protocols fail to distinguish between true reasoning improvements and artifacts of retrieval-based answer matching.

Methodology

1. STC Taxonomy

The authors define STC as a failure mode where retrieval shortcuts the reasoning process. They classify contamination into three severity levels based on the information exposed during the agent's trajectory:

• Benchmark Metadata Leakage (BML): The agent retrieves URLs exposing benchmark-specific metadata (e.g., dataset names, question IDs) rather than task-relevant knowledge. This indicates the search process is steered toward benchmark artifacts.
• Question-Context Leakage (QCL): The agent retrieves content containing the exact phrasing or specific context of the test question but not the ground-truth answer. This narrows the search space and provides external priors, potentially shortcutting independent reasoning.
• Explicit Answer Leakage (EAL): The agent retrieves content containing both the evaluation query and its corresponding ground-truth label. In this scenario, the reasoning chain collapses into a trivial key-value extraction task.

2. Detection Algorithms

To quantify STC, the paper proposes a multi-stage detection framework:

• BML Detection: Utilizes regular-expression matching against search results to identify URLs from common data-hosting platforms (e.g., HuggingFace, GitHub) or exam-preparation sites (e.g., Quizlet, Chegg) that match known dataset patterns.
• QCL Detection: Measures lexical overlap (longest common substring) between the query and retrieved content. High overlap indicates the question context was exposed, excluding cases where the answer is also present (handled by EAL detection).
• EAL Detection: Employs an LLM-as-a-Judge approach (using DeepSeek V4 Pro) to analyze retrieved content and determine if it provides direct evidence for the answer. This method was validated against human annotations, achieving high precision (94.85% on MedQA) and recall (83.3% on Medbullets-5op).

3. Evaluation Framework

The study evaluates agents using two complementary perspectives:

• Question-Level Evaluation: Compares agent performance across subsets split by STC types (e.g., questions with EAL vs. those without) to measure performance inflation.
• Turn-Level Evaluation: Analyzes the dynamic impact of contamination on the agent's reasoning trajectory. This includes:
  • Prediction Shift: Tracking how answers evolve when contamination is detected.
  • Survival Analysis: Using time-varying Cox proportional hazards models to measure how STC events affect the likelihood and speed of reaching a correct prediction.
  • Cascading Effects: Investigating whether earlier leakage types (BML/QCL) catalyze subsequent, more severe leakage (EAL).

Key Results

1. Prevalence and Severity

The study evaluated six medical benchmarks (MedQA, MedMCQA, MMLU, MedXpertQA, HLE, Medbullets-5op) across multiple agents (Tongyi, Step, Gemini, Valyu).

• Widespread Contamination: STC is not an isolated anomaly but a common risk. Older benchmarks like MedQA and MedMCQA show high contamination rates due to extensive online redistribution.
• Performance Inflation: STC can inflate performance by up to 4%. For instance, on HLE biological and chemical subsets, agents achieved significantly higher accuracy when EAL occurred.
• Dataset Specificity: While older datasets are more vulnerable, even recently released datasets (e.g., MedXpertQA, HLE-149) exhibit leakage signals, primarily due to exposure via LLM publications and online discussions.

2. Impact on Agent Performance

• EAL Dominance: Explicit Answer Leakage has the strongest positive association with correct predictions. Hazard ratios (HR) for EAL ranged from 2.20 to 8.92, indicating a significantly increased likelihood of immediate correct answers upon retrieval.
• BML and QCL Ambiguity: Metadata Leakage (BML) alone does not consistently improve performance; in some cases, it correlates with lower accuracy (HR < 1). Question-Context Leakage (QCL) shows mixed effects, sometimes introducing noisy priors rather than direct shortcuts.
• Cascading Dynamics: Kaplan-Meier curves reveal that BML often escalates into EAL. Encountering the exact question context (QCL) significantly increases the hazard of subsequently retrieving the explicit answer (HR 2.50–6.74).

3. Agent Variability

• Gemini Deep Research: Exhibited a 60% leakage rate on 100 sampled MedQA questions, visiting pages containing both questions and answers.
• Valyu Deep Research: Showed 0% leakage on MedQA but 78% leakage on PubMedQA, suggesting that restricting search to specific domains (e.g., PubMed) does not eliminate STC if the benchmark itself is derived from that domain.

Significance and Claims

The paper claims that the exceptional performance observed in modern deep research agents is partially an artifact of web-based answer leakage, rather than a pure reflection of improved reasoning capabilities.

Key contributions include:

1. Systematic Quantification: Providing the first systematic measurement of STC-induced performance inflation, demonstrating that current evaluations may overestimate true reasoning ability.
2. Granular Taxonomy: Moving beyond coarse repository matching to define and detect three distinct contamination types, highlighting that not all "leakage" leads to performance inflation (e.g., BML vs. EAL).
3. Methodological Recommendations: Advocating for contamination-aware evaluation practices, specifically:
   • Isolated Sandboxes: Preventing agents from retrieving benchmark artifacts during inference.
   • Transparent Search Trajectories: Requiring agents to expose search queries, visited URLs, and intermediate evidence for auditing.
   • Controlled Benchmark Access: Implementing gated access or dynamically generated instances to prevent public indexing of test sets.

The authors conclude that without these measures, the evaluation of agentic systems risks devolving into a test of retrieval efficiency rather than genuine reasoning, undermining the validity of benchmark-driven progress in the field.