Technical Summary: Delegation Intelligence in Deep Search

1. Problem Statement

Current evaluations of deep search agents rely heavily on end-to-end answer accuracy. This paradigm creates a "coupled" evaluation where retrieval quality, long-context comprehension, evidence verification, and tool-use decisions are entangled. Consequently, it is difficult to diagnose whether a model's failure stems from an inability to retrieve information, a failure to synthesize evidence, or a lack of judgment regarding when to search. A correct answer may result from parametric memory or lucky retrieval rather than effective search delegation, while a wrong answer may obscure a model's strong reasoning capabilities if the retrieval backend failed. The paper argues that the core intelligence of a deep search agent lies not merely in producing correct answers, but in making calibrated decisions about when and how to delegate information seeking to search tools.

2. Methodology

2.1. Conceptual Framework: Delegation Intelligence

The authors formalize Delegation Intelligence as a metacognitive capacity to autonomously gauge information needs, choose cognitive routes, and vet source reliability. They decompose this into two complementary dimensions:

1. Search Decision-Making: The ability to recognize evidence insufficiency, decide whether to search, and formulate effective search queries (including decomposing complex queries).
2. Information Synthesis & Verification: The ability to integrate evidence from multiple sources, assess source authority and timeliness, detect inconsistencies, and reject plausible distractors.

2.2. Controllable Synthesis Pipeline

To enable disentangled measurement, the authors developed a document-grounded reverse engineering pipeline (Figure 1) that constructs controlled evaluations rather than relying on static, uncontrolled datasets. The pipeline operates in three stages:

• Stage A: Reverse Engineering Synthesis: Starting from high-quality real-world documents (e.g., technical reports, official announcements) as "hindsight," the pipeline works backward to synthesize a natural user query, a golden answer, supporting evidence (golden_docs), and reasoning steps. This ensures the answer is grounded in verified evidence. The pipeline preserves topically related but irrelevant documents (irrelevant_docs) to simulate natural retrieval noise.
• Stage B: Evidence-Anchored Distractor Generation: A generator from a different model family synthesizes adversarial distractor_docs. These documents are topically aligned but invalid under strict evidence verification. Crucially, the distractors differ from the golden evidence only along dimensions explicitly constrained by the query (e.g., publication time, source authority, version number), creating targeted challenges for verification.
• Stage C: Rejection Filtering: A multi-view filtering process ensures item quality:
  • C1 (Answerability): Verifies the query is answerable given only the golden_docs.
  • C2 (Non-memorability): Ensures the query cannot be answered solely from parametric knowledge.
  • C3 (Irrelevance Audit): Confirms irrelevant_docs contain no supporting evidence.
  • C4 (Distractor Validity): Validates that distractors create a meaningful discrimination challenge based on query constraints.

2.3. DelegSearchBench and Evaluation Protocol

The pipeline yields DelegSearchBench, a benchmark of 429 items across six capability modes (sufficiency, multi-hop, authority, publish-time, intra-doc consistency, and plain). The evaluation employs two disentangled settings:

• Setting I (Full-Context, Search Disabled): The model receives the complete document pool (golden + distractor + irrelevant) without search tools. This isolates Information Synthesis & Verification. The study varies the position of golden_docs (first, middle, last) to test robustness to evidence placement.
• Setting II (Partial-Context, Search Enabled): The model receives a deliberately incomplete subset of golden_docs (missing key evidence) but has access to a search tool that retrieves from the item's specific document pool. This isolates Search Decision-Making, testing whether the model recognizes insufficiency and initiates search rather than answering prematurely.

3. Key Results

3.1. Mode-Dependent Weaknesses

Under Setting I, models exhibit significant performance stratification across different capability modes. No single model consistently dominates all modes, indicating that full-text reasoning is a composite of specialized competencies (e.g., source attribution vs. temporal reasoning) rather than a homogeneous capability.

3.2. Positional Sensitivity (Lost-in-the-Middle)

Even when all relevant evidence is provided, most models show non-negligible sensitivity to evidence position. Performance peaks when golden_docs are placed first and degrades substantially when evidence is buried in the middle or end of the context. This confirms that the "lost-in-the-middle" effect is an inherent limitation of context processing, independent of retrieval errors.

3.3. Unstable Reasoning Trajectories

There is a consistent gap between Pass@1 and Pass@3 across all modes and positions. This suggests that models often rely on stochastic exploration to find correct answers rather than following stable, convergent reasoning paths, raising concerns about agent reliability.

3.4. Delegation Failures in Partial Context

Under Setting II, two opposing failure modes emerge:

• Premature Answering: Some models (e.g., doubao-seed-2.0-pro) frequently answer directly despite missing evidence, failing to recognize insufficiency.
• Ineffective Searching: Other models (e.g., hunyuan-3-preview) search aggressively but fail to retrieve or integrate the necessary information, resulting in low hit rates and poor final accuracy despite high search usage.
• Calibrated Delegation: Models like claude-sonnet-4.6 demonstrate better calibration, combining appropriate search rates with effective post-retrieval synthesis.

4. Significance and Claims

The paper claims that end-to-end accuracy is insufficient for diagnosing deep search capabilities. By decoupling retrieval, reasoning, and tool-use decisions, the proposed framework reveals capability boundaries masked by standard benchmarks.

The central insight is that reliable deep search requires more than strong reasoning; it requires calibrated coordination among uncertainty awareness, evidence acquisition, and faithful synthesis. The authors argue that future agent benchmarks must assess not only task-level performance but also the robustness, reliability, and stability of the underlying reasoning processes. The controllable synthesis pipeline is presented as a principled method for exposing these specific capability boundaries, moving beyond the "black box" evaluation of integrated systems.