Technical Summary: Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback

Problem Statement

Existing benchmarks for Deep Research Agents (DRAs) primarily assess single-shot outputs, failing to address a critical practical question: Can DRAs improve their reports when guided by feedback? In real-world scenarios, users rarely accept a first draft; they iteratively revise reports using feedback. Current evaluation methods overlook this iterative capability. Furthermore, while some research explores self-reflection (agents critiquing their own work) or criterion-level feedback (targeting specific content gaps), these approaches often fail to address deeper issues in the agent's research strategy, such as inappropriate source selection, narrow scope framing, or missed subtopics. There is a lack of systematic evaluation on how DRAs adapt their research processes when provided with feedback specifically targeting these strategic gaps.

Methodology

Experimental Framework

The authors evaluate three models—GPT-4.1-mini, GPT-4.1, and DeepSeek-V4-Flash—within the LangChain Open Deep Research (LC-ODR) modular multi-agent framework. The evaluation uses 50 tasks from the DRACO benchmark, which features complex, open-ended research tasks paired with expert-designed rubrics. The rubrics assess four axes: Factual Accuracy (FA), Breadth and Depth of Analysis (BD), Presentation Quality (PQ), and Citation Quality (CQ).

The study compares two multi-turn feedback settings:

1. Self-Reflection: The agent revises its report without external diagnostic signals, relying solely on its own ability to identify flaws.
2. Process-Level Feedback: The agent receives guidance targeting gaps in its research strategy rather than just specific missing content.

Research Gap Inference (RGI)

To enable process-level feedback, the authors introduce Research Gap Inference (RGI). This method analyzes the pattern of satisfied and unsatisfied rubric criteria (specifically FA, BD, and CQ, excluding PQ) to infer underlying research-process gaps.

• Input: A structured summary of the previous report's evaluation, including both passing and failing criteria. Passing criteria serve as contrast signals to distinguish between "shallow coverage" and "missing sources."
• Mechanism: RGI clusters related passes and failures by topic to identify research-process gaps (e.g., relying on secondary summaries instead of primary legislative texts). It then generates a concise feedback message organized around 2–3 research themes, instructing the agent on where to deepen investigation and what types of evidence to prioritize, without explicitly stating the correct answers or specific citations.
• Agent Behavior: The agent performs a full rewrite of the report in every turn, incorporating the original query, the previous report, and the new feedback.

Metrics

Performance is measured using:

• Normalized Score: A weighted aggregation of rubric criteria.
• Pass Rate: The unweighted proportion of satisfied criteria.
• Incorporation Rate: The percentage of previously unsatisfied criteria satisfied in the new turn.
• Regression Rate: The percentage of previously satisfied criteria lost in the new turn.
• Net Criterion Gain: The difference between incorporated and regressed criteria.

Key Results

1. Self-Reflection Yields Negligible Gains

Under self-reflection, agents fail to reliably identify which research dimensions need improvement.

• Performance: Normalized scores showed negligible or negative changes (e.g., +2.42 for GPT-4.1-mini, +0.09 for GPT-4.1, -0.54 for DeepSeek-V4-Flash).
• Behavior: Agents increased research activity (more searches, more sources) but failed to direct this effort toward relevant gaps. Incorporation and regression rates were nearly equal (e.g., ~15% vs. ~13% for GPT-4.1), resulting in near-zero net improvement.

2. Process-Level Feedback Drives Substantial Initial Gains

A single round of process-level feedback (Turn 2) produced significant improvements across all models.

• Performance: Normalized scores increased by approximately 8–15 points (e.g., +15.35 for GPT-4.1-mini, +11.42 for GPT-4.1, +8.15 for DeepSeek-V4-Flash).
• Mechanism: The largest gains occurred in Breadth and Depth (BD) and Factual Accuracy (FA), indicating agents successfully adapted their search strategies to locate missing facts and cover broader dimensions when given strategic guidance.
• Incorporation: Incorporation rates rose to roughly 35–40%, significantly outperforming self-reflection.

3. Multi-Turn Improvement Does Not Compound Reliably

The gains from the first feedback round did not consistently compound in the second revision (Turn 3).

• Regression: When rewriting the full report to address remaining gaps, agents frequently regressed on previously satisfied criteria.
  • GPT-4.1: Normalized score dropped by 4.97 points; regression rate was 23.57%.
  • GPT-4.1-mini: Score improved marginally (+1.34); regression rate was 18.59%.
  • DeepSeek-V4-Flash: Sustained a moderate gain (+4.01) with a significantly lower regression rate (8.96%).
• Cause: The "full rewrite" paradigm forces agents to re-satisfy all criteria from scratch. For GPT models, the number of regressed criteria often exceeded the number of newly incorporated ones.
• Headroom Effect: Turn 3 improvements were conditional. Tasks with low Turn 2 scores (substantial "headroom") were more likely to improve, while tasks with moderate Turn 2 scores were more likely to degrade due to the high risk of regression during aggressive rewrites.

4. Architectural Limitations and Computational Costs

• Rewrite Behavior: Analysis of citation retention and n-gram overlap revealed that GPT models effectively restart research and writing from scratch at each turn (retaining <30% of citations and <7% of 5-grams). DeepSeek-V4-Flash retained more prior content (~54% citations, ~27% 5-grams), explaining its lower regression rate.
• Cost: The model that preserved content best (DeepSeek-V4-Flash) did so at a substantially higher computational cost, consuming roughly 3x the input tokens and 4x the web-search calls of GPT models at Turn 1, with costs widening by Turn 3.

Significance and Claims

The paper claims that reliable multi-turn improvement remains out of reach for current DRA architectures under the evaluated settings. The key findings highlight a fundamental limitation: the absence of a structural mechanism to retain prior coverage.

• Self-Diagnosis Failure: DRAs cannot consistently self-diagnose research gaps without external signals.
• Strategic Adaptation: DRAs can effectively adapt their research strategies when guided by process-level feedback, but this capability is fragile in iterative settings.
• Architectural Bottleneck: The current "plan-search-write" paradigm, which involves full rewrites, creates a trade-off between regression and computational cost. Models that implicitly preserve content regress less but require significantly more compute.
• Future Direction: The authors conclude that dependable multi-turn improvement will require architectures with explicit mechanisms to preserve prior coverage while addressing remaining gaps, rather than relying on implicit preservation or full rewrites.

The work establishes Research Gap Inference (RGI) as a method for generating process-level feedback and provides a rigorous multi-turn evaluation framework that reveals the limitations of current DRAs in iterative refinement scenarios.