🤖 AI

Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents

This paper diagnoses long-horizon search agents by distinguishing between retrieval and utilization failures, revealing that answer accuracy correlates more strongly with the quality of retrieved evidence than with search effort, and suggesting that improved query formulation, evidence selection, and adaptive stopping criteria are key to building better deep research systems.

Original authors: Qi Liu, Jiaxin Mao, Fengbin Zhu, Tat-Seng Chua

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qi Liu, Jiaxin Mao, Fengbin Zhu, Tat-Seng Chua

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. In the world of artificial intelligence, there are special "deep search" agents designed to do exactly this. Instead of just guessing an answer from what they already know, these agents act like digital detectives: they ask questions, read the answers, ask more questions, and piece together clues until they solve the case. This process is called a "long-horizon search" because it can take many steps, or "turns," to get from the first question to the final solution. For a long time, people assumed that if an agent worked harder—asking more questions and reading more documents—it would get smarter and solve more mysteries. It seemed logical: more effort should equal better results. But just like a human detective who keeps digging in the wrong garden even after finding the treasure in the first one, these AI agents might be wasting their energy. The big question researchers wanted to answer was: Does working harder actually make these AI detectives better, or are they just spinning their wheels?

This paper dives deep into the "search trajectory" of six different AI agents to figure out exactly how they think, search, and sometimes fail. The researchers didn't just look at the final answer; they watched every single step the agents took, like reviewing a detective's notebook page by page. They used a special test called BrowseComp-Plus, where every question has a known "gold" answer and a specific set of documents that contain the truth. By comparing what the agents found against these known truths, they discovered something surprising: working harder doesn't make you smarter. In fact, the agents that asked the most questions and read the most text were often the ones getting the answers wrong.

The study found that the secret to a successful search isn't the amount of work, but the quality of the clues found. The researchers broke down failures into two types: "retrieval gaps" and "utilization gaps." A retrieval gap is like a detective who never finds the crucial piece of evidence because they asked the wrong questions. A utilization gap is when the detective finds the evidence but still gets the wrong answer because they misinterpreted it. The paper shows that for most agents, the problem is the retrieval gap—they simply aren't finding the right documents.

Here is the most vivid part of the discovery: evidence arrives early or not at all. The researchers found that if an agent is going to find the "gold" evidence (the key to the answer), it usually finds it in the first few steps of the search. If the agent hasn't found it by then, it rarely will. Yet, many agents keep searching for a long time after the clues have stopped coming. This creates a "wasted tail"—a long, boring stretch of the search where the agent is just reading the same old information or finding dead ends, burning up time and computer memory without making any progress.

The paper also looked at how the agents ask questions. Some agents are like a bull in a china shop, asking the same question over and over again or making tiny, useless changes to their queries. The smartest agents, however, are disciplined. They don't repeat themselves. They try different angles (pivoting) but stop immediately once they have enough evidence. They don't need to search 30 times to solve a puzzle; they might only need 10, provided those 10 searches hit the mark.

In short, this research suggests that the future of better AI detectives isn't about making them search longer or read more. It's about teaching them to ask better questions, stop searching when they have enough, and not waste time re-asking questions that have already failed. The agents that win aren't the ones who work the hardest; they are the ones who work the smartest, finding the needle in the haystack quickly and knowing exactly when to stop digging.

Technical Summary: Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents

1. Problem Statement

Deep search agents (e.g., OpenAI Deep Research, Tongyi DeepResearch) are designed to solve complex, multi-hop information-seeking tasks by iteratively issuing search queries, retrieving documents, and synthesizing evidence. However, current evaluation practices rely almost exclusively on final-answer accuracy metrics (e.g., on the BrowseComp benchmark), treating the search trajectory as an opaque byproduct. This "black box" approach creates three critical diagnostic challenges:

Misattribution of Effort: High search volume and long horizons are often mistaken for capability, obscuring whether agents are searching efficiently or merely persisting.
Misdiagnosis of Bottlenecks: A wrong answer is frequently assumed to stem from a lack of evidence (retrieval failure), when the error may actually occur after the correct evidence has been found (utilization/reasoning failure).
Indistinguishable Persistence: Long trajectories often contain a "wasted tail" of low-yield steps after evidence accumulation has ceased, making it difficult to distinguish between agents that need to search deeper versus those that need to search better.

The paper argues that without decomposing the trajectory into retrieval and reasoning components, it is impossible to prescribe effective improvements for deep research systems.

2. Methodology

The authors propose a diagnostic framework grounded in human-annotated, document-level relevance judgments to analyze search trajectories at a fine granularity.

Experimental Setup

Benchmarks: The primary analysis uses BrowseComp-Plus, a dataset of 830 questions paired with a fixed corpus and human-annotated relevance judgments (qrels). These qrels are split into a broader "evidence" set and a stricter "gold" set sufficient to derive the answer. Validation is performed on BrowseComp using an open-web search API.
Agents: Six agents are evaluated under a unified ReAct framework with identical tools (search and visit) and a fixed retriever (Qwen3-Embedding-8B). The agents span two tiers:
- Mid-scale: Tongyi DeepResearch, Qwen3.5-35B-A3B, gpt-oss-120b.
- Frontier-scale: Kimi K2.6, GLM 5.1, Deepseek V4 Pro.
Metrics: The study moves beyond final accuracy to measure:
- Retrieval Recall: The cumulative proportion of gold/evidence documents retrieved.
- Failure Modes: Decomposition of errors into Retrieval Gaps (evidence never found) and Utilization Gaps (evidence found but answer incorrect).
- Trajectory Dynamics: Analysis of episode utility (productive vs. redundant), query strategies (pivoting, batching, repetition), and context budget composition.

Diagnostic Framework

The analysis is structured around six research questions (RQs) covering aggregate signals, failure taxonomy, episode-level mechanisms, query strategies, and within-question comparisons.

3. Key Findings

RQ1: Search Effort vs. Accuracy

Effort does not predict accuracy: There is no positive correlation between search volume (number of queries) or context budget (tokens consumed) and final answer accuracy. In fact, the most accurate agents (e.g., GLM 5.1) often achieve their results with the lowest search volume and context usage.
Recall is the primary predictor: Answer accuracy is strongly correlated ( $r=0.99$ ) with cumulative retrieval recall. If an agent retrieves the gold evidence, it answers correctly with high probability; if it misses the gold evidence, it almost never answers correctly.

RQ2: Failure Mode Taxonomy

Errors are strictly categorized into two gaps with distinct remedies:

Retrieval Gap (RG): The agent fails to surface the necessary evidence. This is the dominant failure mode for five of the six agents (51.6%–64.1% of errors).
Utilization Gap (UG): The agent retrieves the gold evidence but fails to synthesize the correct answer. This is the dominant failure mode for Kimi K2.6 (52.0% of errors), making it the exception to the trend.

Subtypes: RGs are further split into directional (wrong topic) and last-hop (missed specific document). UGs are split into true extraction (reasoning failure) and boundary (partial evidence).
Implication: Improvements must be diagnosis-specific. Retrieval gaps require better query formulation; utilization gaps require better verification and reasoning.

RQ3: Trajectory Mechanisms

Early Evidence Concentration: Useful evidence is found early. By episode 5, many agents have already retrieved gold documents.
The Wasted Tail: Incorrect trajectories are 1.9–2.9 times longer than correct ones, but they do not find more evidence. Instead, they continue searching long after the point where gold evidence would have been found, generating a "wasted tail" of redundant snippets.
Context Dominance: Search snippets (not full visited documents) dominate the context window (66–85%), suggesting that context management should focus on snippet filtering rather than just document selection.

RQ4: Query Strategies

Discipline over Style: Strong agents are defined by query discipline, not a specific search style (e.g., broad pivoting vs. anchor-and-verify).
Redundancy is a Failure Signal: The strongest predictor of failure is the rate of redundant re-querying (near-identical consecutive queries). The weakest agent (gpt-oss-120b) repeats queries 5.0% of the time, while top performers do so <1.5% of the time.
Pivoting is Neutral: The rate of pivoting to new angles does not correlate with accuracy; what matters is whether the pivot lands on evidence.
Batching: Stronger agents often issue multiple queries per turn (parallel batching), which correlates with higher efficiency, though this is a style choice rather than a strict requirement.

RQ5 & RQ6: Generalizability

Within-Question Control: When controlling for question difficulty, retrieval recall absorbs 56–60% of the performance gap between agents. Agents that retrieve gold evidence on the same question succeed 73.8% of the time, while those that miss it succeed only 2.2% of the time.
Open-Web Validation: Findings hold on open-web search (BrowseComp-en), where accuracy tracks judged evidence coverage rather than search effort.

4. Key Contributions

Diagnostic Framework: A novel framework for analyzing search agents that grounds every step in human-annotated relevance, separating retrieval performance from reasoning performance.
Comprehensive Empirical Study: A controlled comparison of six diverse agents (open and frontier) revealing that search effort is weakly aligned with quality, while retrieval recall is the decisive factor.
Failure Mode Decomposition: The identification of "Retrieval Gaps" vs. "Utilization Gaps," demonstrating that agents with similar accuracy scores may require fundamentally different fixes (query reformulation vs. verification).
Behavioral Signatures: The characterization of "query discipline" (low redundancy, early evidence saturation) as the hallmark of strong agents, contrasting with the "wasted tail" behavior of weaker agents.

5. Significance and Claims

The paper claims that the next generation of deep research systems should not focus on deeper search (more steps, larger context) but on better-directed search.

For Weaker Agents: The primary opportunity lies in query formulation. Agents must learn to recognize when a hypothesis has failed and pivot effectively, avoiding redundant re-queries.
For Stronger Agents: Since they are already near the "oracle-reader" ceiling (retrieving most gold evidence), the remaining gains lie in verification, answer normalization, and stopping criteria based on evidence sufficiency rather than fixed budgets.
System Design: The authors advocate for evidence-driven stopping rules, snippet-stream management to prevent context overflow, and gating mechanisms against low-value page opens.

The work concludes that current evaluation metrics obscure the true bottlenecks in agent performance. By shifting focus from final accuracy to the trajectory-level mechanics of retrieval and utilization, developers can prescribe targeted interventions that address the specific failure modes of their systems.