💻 computer science

Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration

This paper introduces "Clarify-Then-Search," a benchmark built on real-world Baidu data that evaluates how LLM-generated clarification questions improve deep search performance by restoring missing query constraints, demonstrating that such interactions significantly boost end-to-end utility while revealing common failure modes like over-asking unanswerable location questions.

Original authors: Deqiang Huang, Jingbo Zhou, Xinjiang Lu, Tong Xu, Hua Wu, Enhong Chen

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Deqiang Huang, Jingbo Zhou, Xinjiang Lu, Tong Xu, Hua Wu, Enhong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of digital information, search engines have evolved from simple keyword matchers into complex assistants capable of deep investigation. These systems, often called deep search agents, do not just return a list of links; they actively browse the web, gather evidence from multiple sources, and synthesize a comprehensive answer to a user's question. However, these powerful tools have a critical weakness: they struggle when a human's request is vague. If a user asks for "the best universities," the system does not know if they mean the best for engineering, the best in Europe, or the best for a specific budget. Without these missing details, the search can drift off course, wasting time on irrelevant information and delivering a final answer that feels incomplete or wrong. The solution seems obvious: the computer should ask for clarification. But building a system that knows exactly what to ask, and how to use the answer to improve the search, has been difficult to test. Until now, researchers have lacked a reliable way to measure whether a clarifying question actually helps the search engine find better information, or if it is just polite chatter that leads nowhere.

A team of researchers from the University of Science and Technology of China and Baidu has created a new testing ground to solve this problem. They introduced a benchmark called "Clarify-Then-Search," a rigorous experiment designed to see if asking questions before searching truly leads to better results. The team started with real-world search queries from the Baidu search engine. For each test case, they first fused and normalized the raw data into a clear, specific intent query (the "fused_query"), such as "the best universities for a master's degree in AI in Europe." They then deliberately blurred this intent by removing key constraints like time, region, or scope to create the vague request (the "blurred_query") that the system would see. This created a paired dataset for every test case: the vague query the system must handle, and the clear intent that represented the user's true goal.

To ensure the test was fair and free of manipulation, the researchers set up a strict, two-phase interaction. First, a "Clarifier" model looked only at the vague query and generated a set of questions to ask the user. Next, a "User Answerer" model, which knew the true, clear intent, answered those questions. Crucially, the User Answerer was programmed to be honest: if the true intent did not contain the specific information requested, it had to say "unknown." It could not invent details to be helpful. Finally, a "Rewriter" model took the original vague query and the answers it received to create a new, improved search query. This rewritten query was then fed into a fixed deep-search system called WebDancer, which went out to find the actual information. The researchers then compared the final answer against a "golden reference" built from the clear intent query to see how much of the correct information was successfully restored.

The results revealed that while asking questions helps, it is not a magic bullet. When the system was allowed to ask just one question, every model tested performed better than if it had asked nothing at all. The best-performing model, GPT-5.2, managed to recover significantly more useful information than the baseline. However, the study uncovered a persistent flaw in how these systems think. Many models, particularly those that performed well in other areas, had a strong habit of asking about location or region, such as "Which city are you looking for?" In the strict conditions of this test, where the true intent often did not specify a location, these questions frequently resulted in an "unknown" answer. This meant the system wasted a turn asking a question it could not answer, leaving the rewritten query just as vague as the original. The researchers found that this "region-only" bias was a major reason why some interactions failed to improve the search, even when the questions sounded natural and relevant.

Increasing the number of allowed questions from one to two or three generally improved the results, but the benefits depended heavily on the model's strategy. While GPT-5.2 led with a single question, a different model, ERNIE-4.5-Turbo-128K, took the top spot when allowed three questions. This suggests that some systems are better at picking a single, high-value question, while others excel at a sustained conversation that gradually uncovers the missing details. The study also showed that simply having more turns does not guarantee success; if the extra questions continue to target information that is not present in the user's intent, the system continues to hit dead ends. The most effective systems were those that managed to avoid these low-yield questions and successfully elicited at least one piece of concrete, usable information.

This work provides a clear, reproducible way to measure the true value of clarification in deep search. It moves beyond simply checking if a question sounds polite or relevant, focusing instead on whether the question actually leads to a better final answer. The findings suggest that for deep search systems to become truly reliable, they must learn not just to ask, but to ask the right things. They need to recognize when a question is likely to yield an answer and when it is better to proceed with the search using the information they already have. By exposing these specific failure modes, the benchmark offers a roadmap for building search agents that are not only smarter but also more efficient, ensuring that every interaction brings the user closer to the truth they are seeking.

Technical Summary: Clarify-Then-Search

Problem Statement

Deep search systems, which interleave reasoning with tool use (e.g., web search and browsing), are increasingly deployed for complex information needs. However, these systems exhibit brittleness when faced with underspecified user queries. Real-world search logs frequently reveal omissions of critical constraints such as temporal scope, geographic region, entity disambiguation, numeric thresholds, or concept definitions. In deep search, such ambiguities have severe downstream consequences: they can mislead search planning, waste tool-call budgets on low-yield exploration, trigger off-target evidence visits, and propagate into incomplete or incorrect final answers.

While Large Language Models (LLMs) possess the capability to ask clarifying questions, existing evaluations of clarification in deep search remain limited. Prior benchmarks often focus on question-level properties (fluency, plausibility) rather than downstream utility, or they suffer from intent leakage, where the rewriting or retrieval stages indirectly observe the ground-truth intent, making it difficult to attribute performance gains to the quality of clarification.

Methodology: The Clarify-Then-Search Benchmark

The authors introduce Clarify-Then-Search, a benchmark designed to evaluate the ability of LLMs to ask clarifying questions that facilitate deep search under strict information constraints. The benchmark is built on 518 curated instances derived from real-world Baidu search queries.

1. Data Construction

Each instance consists of a pair:

$q^*$ (fused_query): The underlying intent query containing all necessary constraints.
$q$ (blurred_query): An underspecified version of $q^*$ where key constraints (time, location, scope, etc.) have been removed.

For every $q^*$ , the authors run a fixed deep-search backbone (WebDancer) once to archive evidence traces. From these traces, they extract a static golden reference consisting of weighted, evidence-grounded "nuggets" (atomic, judgeable statements with traceable source identifiers). This static gold is reused across all evaluations to ensure reproducibility.

2. Two-Phase Closed-Book Protocol

To prevent intent leakage and isolate the contribution of clarification, the evaluation follows a strict two-phase protocol:

Phase 1: Clarification & Answering. A Clarifier generates $k$ clarification questions ( $k \in \{1, 2, 3\}$ ) based only on the blurred query $q$ . A User Answerer (UA), acting as a constrained simulator, receives the hidden intent $q^*$ and the clarification question. The UA must answer only if the requested attribute is explicitly present in $q^*$ ; otherwise, it must output unknown. The UA is forbidden from fabricating information.
Phase 2: Rewriting & Search. A Rewriter receives the blurred query $q$ and the elicited Q&A pairs. Crucially, the Rewriter never sees $q^*$ . It must rewrite the query $\hat{q}$ using only the information explicitly elicited. If an answer was unknown, the Rewriter must preserve the ambiguity rather than forcing a concrete value.
Execution: WebDancer executes a deep search on the rewritten query $\hat{q}$ .

3. Evaluation Metric

Performance is measured by restore_score_100, a weighted nugget recall score (0–100).

A coverage judge (fixed to ERNIE-4.5-Turbo) compares the final search output against the static golden nuggets.
Labels are assigned as full (1.0), partial (0.5), or none (0.0).
The score is the weighted sum of these labels divided by the total weight of the golden nuggets.

Key Contributions

Dataset and Task Formalization: The paper formalizes deep-search clarification as an end-to-end benchmark from underspecified queries to nugget restoration. It releases a dataset of 518 instances with standardized evaluation outputs.
Leakage-Free Protocol: The introduction of a two-phase closed-book protocol prevents the Rewriter from observing the oracle intent, enabling controlled comparisons of Clarifiers.
Static Golden Nuggets: The use of a one-time, evidence-grounded static reference allows for reproducible comparisons across different models and clarification budgets ( $k$ ).
Diagnostic Signals: The benchmark makes intermediate interaction behaviors measurable, including User Answerer (UA) answerability, multi-turn "all-unknown" rates, and question-type bias, allowing for granular analysis of failure modes beyond final scores.
Reproducibility Artifacts: The authors release the dataset, pipeline scripts, prompts, and per-instance tool traces required to reproduce golden construction and evaluation.

Experimental Results

The study evaluated multiple LLMs (including GPT-5.2, Claude-Sonnet-4.5, Gemini-2.5-Pro, and open-weight models like ERNIE-4.5-Turbo-128K, Qwen3, and DeepSeek-V3.2) using the fixed WebDancer backend.

Baseline Improvement: All Clarifiers improved over the no-interaction baseline (running WebDancer directly on the blurred query) at $k=1$ .
Budget Effects: Larger clarification budgets ( $k=2, 3$ $k = 2, 3$ ) generally yielded further gains.
- At $k=1$ , GPT-5.2 achieved the highest mean score among all models. Among open-weight models, ERNIE-4.5-Turbo-128K led.
- At $k=3$ , ERNIE-4.5-Turbo-128K became the overall top-performing model, surpassing GPT-5.2, Claude-Sonnet-4.5, and Gemini-2.5-Pro. This suggests that larger budgets reward sustained question-selection quality across turns.
Diagnostics & Failure Modes:
- High Unknown Rates: A significant bottleneck is the "information gain" failure. At $k=1$ , the UA returned unknown in 60–70% of cases, meaning many single-turn clarifications failed to elicit intent-grounded constraints.
- Region-Only Bias: A consistent low-yield failure mode was identified where models frequently asked region/location questions. Under the closed-book protocol, these questions often elicited unknown (unknown rates of 0.75–0.87 for region-only questions), wasting clarification turns without improving the rewritten query.
- Multi-turn Benefits: Increasing $k$ significantly reduced the probability of an entire interaction yielding no grounded signal (e.g., GPT's "all-unknown" rate dropped from 0.618 at $k=1$ to 0.189 at $k=3$ ).

Significance and Claims

The paper claims that Clarify-Then-Search enables leakage-resistant and reproducible evaluation of clarify-then-search pipelines. Its primary significance lies in shifting the evaluation focus from question-level metrics to end-to-end utility (nugget restoration) under realistic information constraints.

The authors emphasize that while clarification consistently improves deep search over a non-interactive baseline, reliably improving deep search remains challenging due to systematic failure modes. Specifically, many systems over-ask questions (particularly about regions) that are unanswerable given the hidden intent, leading to low-yield interactions. The benchmark provides the necessary diagnostic tools to analyze why clarification succeeds or fails, highlighting the need for future systems to better decide when to ask, what to ask, and how to utilize clarification signals effectively.

The results are presented with modesty regarding absolute scores, noting that the benchmark is best interpreted as a controlled evaluation of clarification utility under a shared deep-search pipeline rather than a backend-agnostic ranking of all possible search systems.