Technical Summary: DISCOBENCH for Clarification-Aware Deep Search

1. Problem Statement

Search agents powered by Large Language Models (LLMs) are increasingly deployed for complex, multi-step information-seeking tasks. However, existing evaluation benchmarks largely operate under the assumption that user queries are complete, explicit, and factually correct. This assumption diverges significantly from real-world scenarios where users frequently provide vague, underspecified, or factually incorrect queries due to memory limitations or cognitive load.

In "deep search" scenarios involving complex reasoning chains, initial ambiguity can propagate through subsequent steps, leading agents down incorrect trajectories and wasting computational resources. While the academic community recognizes the importance of query ambiguity, current benchmarks fail to adequately evaluate an agent's ability to proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through interaction. Existing datasets either focus on static ambiguity (e.g., AmbigQA) or lack the depth of open-domain web search and the amplification effects of ambiguity within long reasoning chains (e.g., InteractComp).

2. Methodology

2.1 DISCOBENCH Construction

The authors introduce DISCOBENCH (Deep Interactive Search with ClarificatiOn Benchmark), a dataset designed to model ambiguity as a dynamic phenomenon arising during multi-step search trajectories. The construction pipeline consists of two phases:

• Phase 1: Seed Data Preparation: The team collected seed topics from 11 diverse domains (e.g., Film/TV, Sports, Academic, Finance) using encyclopedic resources and search engine results. They constructed high-quality multi-hop reasoning chains (211 samples) where answers are objectively verifiable and require external retrieval.
• Phase 2: Ambiguous Data Construction: Deterministic reasoning chains were transformed into interactive tasks by injecting ambiguity. The authors identified "ambiguity points" where relaxing constraints leads to multiple plausible candidates. They replaced strong constraints with shared attributes among candidate entities, creating four specific ambiguity types:
  1. Entity: Multiple entities satisfy the same description.
  2. Version: Different temporal or version-specific states.
  3. Criteria: Different evaluation standards or ranking criteria.
  4. Factual Inaccuracy: Descriptions inconsistent with objective facts.

The dataset contains 211 samples with 463 ambiguity instances. For each ambiguity point, discriminative facts (clues) were generated to allow a user simulator to provide the necessary information to resolve the ambiguity.

2.2 Evaluation Framework

The evaluation simulates a multi-turn interaction between an agent and a user simulator (implemented via an LLM). The agent navigates a sequence of structured Checkpoints (CPs). At each checkpoint, the agent must decide between three actions:

1. SEARCH: Retrieve information.
2. ASK: Request clarification from the user.
3. ANSWER: Submit a final answer.

The framework evaluates agents from four perspectives:

• Task Utility: End-to-end accuracy and checkpoint pass rate.
• Ambiguity Detection: Detection accuracy and F1 score (identifying when a checkpoint is ambiguous).
• Interaction Strategy: Clarification effectiveness (accuracy of the question asked and whether it leads to advancing the checkpoint).
• Cost Efficiency: Token consumption and tool-use turns.

Two prompting settings were used: Neutral (no explicit instruction to look for ambiguity) and Guided (explicitly reminding the agent to be aware of potential ambiguity).

3. Key Contributions

1. DISCOBENCH Benchmark: A novel benchmark modeling ambiguity as a dynamic, propagating issue in multi-step search rather than a static property of a single query. It covers 11 domains and four ambiguity types.
2. Ambiguity-Aware Evaluation Framework: A unified protocol including a user simulator that progressively reveals discriminative clues, enabling the evaluation of detection, clarification quality, and interaction cost simultaneously.
3. Empirical Insights: Extensive experiments revealing that ambiguity detection and effective clarification are distinct capabilities. The authors identify a critical failure mode where agents repeatedly search instead of asking for clarification, often performing worse than direct guessing.

4. Experimental Results

The authors evaluated 11 frontier LLMs (including GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.7, and various Chinese models) on DISCOBENCH.

• Limited Performance: Under the Neutral setting, the best-performing model (Doubao-Seed-2.0-Pro) achieved only 43.1% end-to-end accuracy. Most models remained below 40%.
• Gap Between Detection and Clarification: While "Guided" prompting improved ambiguity detection (F1 increased from ~45% to ~65%), it did not proportionally improve end-to-end accuracy. This indicates that recognizing ambiguity is distinct from formulating effective clarification questions.
• The "Search-Heavy Guess" Failure Mode: A significant finding is that agents often fail by performing excessive retrieval (SearchHeavyGuess) rather than asking for help. This behavior yielded lower success rates (51.9%) than direct guessing (56.5%). The authors note that repeated retrieval often indicates the model is aware of multiple candidates but fails to escalate this uncertainty into a clarification request.
• Tool Usage vs. Performance: Increased tool use (more search calls) did not correlate with better performance. Models that searched more frequently did not necessarily achieve higher accuracy, suggesting that strategic allocation of retrieval actions is more critical than volume.
• Impact of Ambiguity: Removing ambiguity from the tasks significantly improved accuracy across all models (by 26.8% to 40.2%), confirming that ambiguity is a primary source of difficulty in deep search.

5. Significance and Claims

The paper claims that current LLM-based search agents possess strong retrieval and reasoning abilities but lack ambiguity awareness and proactive interaction strategies. The authors argue that deep interactive search requires mechanisms that explicitly bridge retrieval uncertainty with user interaction.

The significance of DISCOBENCH lies in its ability to expose a gap in current agent capabilities: the inability to distinguish between "searching for more information" and "asking for clarification." The authors conclude that future search agents must be developed not just with stronger reasoning, but with better capabilities to identify underspecified states and strategically request user input to recover correct reasoning trajectories.

The paper maintains a modest tone regarding its limitations, acknowledging that the user simulator is LLM-based rather than human, and that the benchmark currently focuses on objective question answering, leaving subjective preference ambiguity for future work.