Technical Summary: DRNOISE – Benchmarking Deep Research Agents in Misleading Evidence Environments

Problem Statement

Deep research agents are increasingly deployed to perform complex knowledge work on the open web, where they must synthesize conclusions from a mix of authoritative records, redundant summaries, and conflicting claims. A critical vulnerability in these environments is the presence of "answer-like" documents—such as executive summaries, cached fact sheets, or secondary news articles—that present a fluent, direct answer without being authoritative.

Current evaluation benchmarks (e.g., GAIA, BrowseComp) primarily assess an agent's ability to retrieve information or navigate complex search spaces. However, they offer limited insight into whether agents maintain sound evidential standards when a plausible, false document is deliberately seeded into a searchable environment. Specifically, it remains unclear if an agent will treat such a document as a sufficient basis for a conclusion or if it will continue to reconcile the claim against underlying record-level evidence. This paper posits that a key failure mode is not the inability to retrieve truth, but the premature cessation of verification in favor of an attractive, direct shortcut.

Methodology: The DRNOISE Benchmark

The authors introduce DRNOISE, a 100-task benchmark designed to isolate and measure "answer recovery under misleading evidence."

Task Formulation

Each task instance is defined as a tuple $(q, a^\star, \tilde{a}, E_A, E_B, F)$:

• $q$: The research question.
• $a^\star$: The unique gold answer.
• $E_A, E_B$: Two independent, corroborating chains of indirect records. No single document states $a^\star$ directly; the answer must be reconstructed by intersecting constraints across these lower-level records (e.g., joining vendor IDs with certification states and shipment logs).
• $F$: A "noisy" document. In the noisy condition, a single, ordinary-looking document is added that directly states a conflicting answer $\tilde{a}$.

Experimental Design

The benchmark employs a paired clean–noisy intervention:

1. Clean Condition: The agent operates on a corpus containing only the indirect evidence chains ($E_A, E_B$).
2. Noisy Condition: The corpus is identical to the clean version, except for the addition of the single false summary $F$.

This design ensures that any performance drop is not due to a lack of retrieval capability (since the truth-supporting records are present in both conditions) but rather a change in the agent's evidence-use policy. The benchmark spans 10 task families (e.g., entity selection, date selection, boolean verification, multi-hop join, total cost aggregation) to ensure the phenomenon is not limited to a specific reasoning pattern.

Evaluation Protocol

The authors evaluated five contemporary language models (Gemini 3.5 Flash, MiMo v2.5, GPT-5.4, DeepSeek V4 Flash, Qwen3-32B) using a fixed BrowseComp-Plus search-agent harness with a ReAct-style interface.

• Metrics:
  • Clean Accuracy: Performance without the misleading document.
  • Noisy Accuracy: Performance with the misleading document.
  • Conditional Deference: The fraction of tasks the model solved correctly in the clean condition that it subsequently answered with the injected false value $\tilde{a}$ in the noisy condition. This metric isolates the specific failure of deferring to the misleading shortcut.

Key Results

1. Sharp Accuracy Degradation

Across agents with strong clean-task performance, the introduction of a single misleading document caused massive accuracy drops (66–88 percentage points).

• DeepSeek V4 Flash and MiMo v2.5 dropped from ~82–89% clean accuracy to ~1% noisy accuracy.
• Gemini 3.5 Flash dropped from 96.3% to 26.0%.
• GPT-5.4 dropped from 81.7% to 15.7%.

2. Systematic Verification Inertia

Trace analysis revealed that the dominant failure mode is premature stopping.

• Retrieval Exposure: In the noisy condition, agents retrieved the truthful indirect records in 100% of cases (co-retrieved with the false summary).
• Stopping Behavior: Despite retrieving truth-supporting evidence, agents often stopped searching before completing the evidence chain. For GPT-5.4, the mean number of search calls dropped from 7.1 (clean) to 4.5 (noisy), and the rate of retrieving a complete evidence route fell from 45% to 16%.
• Deferral: Agents frequently treated the direct, answer-like document as a sufficient conclusion, deferring to it even when the underlying records refuted it.

3. Generalization Across Task Families

The degradation was observed across both symbolic (e.g., entity selection, boolean verification) and numerical (e.g., total cost aggregation) task families, indicating the issue is a general stopping/verification problem rather than an arithmetic or specific reasoning failure.

4. Analysis of Failure Mechanisms

• Verification Prompts: Generic instructions to "verify against documents" reduced the gap but did not close it (e.g., GPT-5.4 noisy accuracy improved from 12% to 28%).
• Source Credibility: Changing the false document's source from a low-credibility forum post to a high-credibility audit report did not significantly reduce deference (54% vs. 84% conditional deference), suggesting the effect saturates beyond a low credibility threshold.
• Retriever Type: The effect persisted and even intensified under lexical (BM25) retrieval compared to dense retrieval, ruling out retrieval artifacts as the primary cause.
• Direct Truth vs. Direct False: When a direct-true document was added to the noisy condition (making the truth as direct as the lie), accuracy recovered significantly (12% to 63%), though a residual deference remained. This suggests the asymmetry between derived truth and stated falsehood is a major factor, but not the sole cause.
• Oracle Full-Context: When the complete evidence was provided directly in the context (bypassing retrieval), conditional deference dropped from 84% to 8%. This confirms that the primary failure is premature stopping during the search process, not an inability to reconcile visible conflicts.

Contributions and Significance

The paper makes three primary contributions:

1. DRNOISE Benchmark: A controlled 100-task benchmark that isolates the specific failure mode of deferring to misleading direct claims in the presence of sufficient indirect evidence.
2. Empirical Evidence of Capability-Deployment Gap: The results demonstrate that high-performing agents possess the capability to solve these tasks (evidenced by clean performance) but fail to deploy this capability under noise. The failure is a policy shift toward "verification inertia" rather than a lack of reasoning ability.
3. Mechanism Identification: Through trace analysis and ablation studies, the authors identify premature stopping as the dominant mechanism. Agents retrieve truthful records but stop before reconciling them with the misleading document, treating the latter as a sufficient endpoint.

Conclusion

The paper concludes that reliable deep research requires more than retrieval and citation; it necessitates the active reconciliation of direct claims with record-level evidence. Current agents exhibit a systematic tendency to treat answer-like assertions as sufficient evidence, even when contradictory lower-level records are available. The authors argue that future evaluations must move beyond testing information acquisition to testing whether agents can resist the "shortcut" of a plausible, direct answer in favor of reconstructing conclusions from corroborating evidence.