Technical Summary: IDRBench – Benchmarking Interactive Deep Research Agents

1. Problem Statement

Current Deep Research agents, powered by Large Language Models (LLMs), excel at autonomous multi-step reasoning, web exploration, and long-form report generation. However, existing systems operate under a fully autonomous paradigm, assuming user intents are fully specified and static. In practice, research goals are often underspecified and evolve during exploration. When agents execute long-horizon tasks without feedback, they risk "intent drift," where the research trajectory diverges from the user's actual needs.

Furthermore, existing benchmarks fail to address this gap because they:

1. Assume fixed tasks: They overlook the interactive and evolving nature of real-world research.
2. Lack controlled interaction environments: Reliable evaluation requires consistent user feedback, but human-in-the-loop interaction is variable and difficult to scale.
3. Rely on static evaluation: They assess only final report quality, ignoring the alignment gains and communication costs introduced by interaction.

Consequently, there is no standardized method to evaluate whether an agent can effectively solicit clarification to align with user intent or to measure the trade-off between interaction benefits and overhead.

2. Methodology: IDRBench

To address these limitations, the authors introduce IDRBench (Interactive Deep Research Benchmark), a framework designed to systematically evaluate the interactive capabilities of deep research agents.

2.1 Interactive Deep Research Framework

IDRBench formulates deep research as an interactive process rather than a static generation task. The framework builds upon a modular pipeline (Planning, Research Loop, Generation) augmented with an Interaction Mechanism:

• Evaluator: Determines when interaction is necessary by balancing the benefit of resolving ambiguity against the cognitive load and latency of interruption. It generates a rationale based on topic ambiguity, task completeness, and remaining budget.
• Questioner: Formulates targeted inquiries (1–2 questions) regarding scope, direction, or emphasis when interaction is triggered, adapting its tone to the user's style.
• User Simulator: Provides standardized, intent-level feedback using a reference document as a proxy for latent user preferences. Crucially, it is constrained to avoid "reference leakage," answering in a natural first-person style with high-level guidance rather than exposing specific document content.

2.2 Data Construction

The benchmark is built upon DeepResearch Bench (100 high-quality Query-Reference pairs). To simulate real-world underspecification, the authors introduce an Ambiguity Injection process:

• Queries are compressed by 10%–90% using LLM-based summarization while preserving core intent.
• This process selectively omits preferences, constraints, or contextual details, forcing agents to actively resolve uncertainty through interaction rather than passively executing fully specified prompts.
• All compressed queries are manually verified to ensure safety and intent preservation.

2.3 Model Selection

The study evaluates seven representative LLMs under both autonomous and interactive settings:

• Proprietary: GPT-5.1, Gemini-2.5-Pro, Claude-Sonnet-4.5, Grok-4.1-Fast.
• Open-Weight: Qwen3-235B, Llama-4-Maverick, DeepSeek-V3.2.

2.4 Evaluation Suite

IDRBench measures performance along two complementary dimensions:

1. Interaction Benefit (Alignment Gains):
   • Report Similarity: Normalized cosine similarity between generated and reference reports.
   • Multi-Granularity F1-Score: Structural coverage at sentence, paragraph, and chunk levels.
   • LLM Aspect Coverage Score (LLM-ACS): Measures how well the report satisfies specific intent aspects derived from the query.
2. Interaction Cost (Overhead):
   • Interaction Turns: Frequency of pauses to solicit input (capped at 1 in planning, 3 in research, 1 in generation).
   • Interaction Tokens: Volume of information exchanged (question tokens and response tokens).

3. Key Results

Experiments across the seven models reveal several critical findings:

• Universal Improvement: Interaction consistently improves research quality and robustness across all models and metrics. For instance, DeepSeek-V3.2 surpassed GPT-5.1's autonomous score once interaction was enabled (78.64 vs. 75.59).
• Diminishing Returns: There is an inverse relationship between model capability and interaction gains. Weaker models (e.g., Llama-4-Maverick, +10.96 average score) benefit significantly more from clarification than stronger models (e.g., GPT-5.1, +3.38).
• Granularity Shift: Interaction helps weaker models establish global topic coverage and intent alignment (coarse-grained improvements), while stronger models benefit more from refining local details (fine-grained improvements).
• Cost-Efficiency Trade-offs:
  • Interaction generally increases API costs, though open-weight models incur negligible overhead.
  • DeepSeek-V3.2 achieved the best benefit-cost trade-off, delivering strong gains with minimal additional cost (+$0.039).
  • Qwen3-235B slightly reduced costs (-$0.006), suggesting interaction can streamline reasoning and search.
• Robustness: Interaction reduces extreme failures. For strong models, it improves worst-case behavior; for weaker models, it shifts the entire performance distribution upward.
• Interaction Strategies: Models exhibit distinct styles. Some (e.g., Claude-Sonnet-4.5) ask long, context-rich questions, while others (e.g., Llama-4-Maverick) favor concise interactions. Frequent interactors (e.g., Gemini-2.5-Pro) tend to ask shorter, incremental questions.

4. Significance and Claims

The paper claims that interactive capability is a distinct evaluation dimension separate from raw autonomous model strength. By introducing IDRBench, the authors establish:

1. A reusable benchmark for future user-aligned deep research agents that moves beyond static, fully-specified task evaluation.
2. A controlled simulation environment that enables reproducible evaluation of interaction benefits and costs without the variability of human-in-the-loop testing.
3. Evidence that interaction is not merely optional but a critical factor for robustness, particularly for models with lower autonomous reasoning capabilities.

The authors position IDRBench as a foundational step toward evolving deep research from autonomous execution to a collaborative, interactive paradigm where agents iteratively communicate progress and refine direction with users.