💬 NLP

Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests

This paper analyzes 14.44 million real-world agentic search requests from DeepResearchGym to reveal key behavioral patterns, such as session length distributions, intent-dependent exploration strategies, and the strong lexical traceability of query reformulations to retrieved evidence, ultimately offering actionable signals for optimizing search agent design.

Original authors: Jingjie Ning, João Coelho, Yibo Kong, Yunfan Long, Bruno Martins, João Magalhães, Jamie Callan, Chenyan Xiong

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingjie Ning, João Coelho, Yibo Kong, Yunfan Long, Bruno Martins, João Magalhães, Jamie Callan, Chenyan Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, bustling library where, instead of human readers, thousands of robot librarians are working. These robots are powered by advanced AI. Their job isn't just to find one book; they are tasked with solving complex puzzles. To do this, they don't just ask for a book once. They ask, read the answer, ask a follow-up question based on what they just read, read again, and keep going until they think they have the full picture.

This paper is a massive observation study of these robot librarians. The researchers looked at over 14 million search requests (about 4 million "conversations" or sessions) from a public system called DeepResearchGym. They wanted to understand: What are these robots trying to do, and how do they change their questions as they go?

Here is a breakdown of their findings using simple analogies:

1. The "Shopping List" vs. The "Detective" (Intent)

The researchers found that the robots fall into three main personality types, or "intents," which change how they search:

The Fact-Seeker (Declarative): This robot just wants a specific fact, like "Who owns this snack company?"
- Behavior: It tends to get stuck in a loop. If it doesn't find the answer immediately, it keeps asking the same question or slightly tweaked versions of it (like a person knocking on a door and shouting the same question over and over).
The How-To Guide (Procedural): This robot wants to know how to do something, like "How do I fix a leaky faucet?"
- Behavior: It asks longer, more detailed questions and digs deeper into the "library shelves" (retrieves more documents) to get a complete set of instructions.
The Detective (Reasoning): This robot has a complex puzzle, like "Why is gas expensive?"
- Behavior: It is the most adventurous. It doesn't just stick to one angle; it jumps between different topics (facets) to connect the dots. It explores widely rather than just digging deeper in one spot.

2. The "Drill-Down" Habit

One of the biggest discoveries is that these robots have a strong habit of zooming in rather than zooming out.

Imagine you are looking at a map. Most of the time, these robots take a broad view and then immediately zoom in on a tiny street to look for details. They rarely step back to say, "Wait, maybe I'm looking at the wrong city entirely."

The "Drill-Down" Bias: About 36% to 48% of the time, they are just pivoting to a slightly different angle of the same topic.
The "Zoom Out" Rarity: Only about 9% of the time do they actually broaden their search to look at the bigger picture. When they do zoom out, they almost immediately zoom back in again. It's like taking a quick glance at the whole map before immediately focusing on a single street again.

3. The "Echo Chamber" Effect (Repetition)

The study found that 90% of these robot conversations are short (10 steps or fewer), and they happen very fast (often under a minute between questions).

However, for the Fact-Seekers, there is a problem. As the conversation goes on, they start repeating themselves more and more. By the 9th step, nearly half of their questions are almost identical to previous ones. It's like a person who forgot they already asked a question, or is stuck in a loop trying to get the same answer from a different angle without success.

4. The "Term Adoption" Test (Did they actually read?)

The researchers wanted to know: When the robot asks a new question, does it actually use the information it just found?

Since they couldn't see the robot's internal "brain" (memory), they invented a clever test called CTAR (Context-driven Term Adoption Rate).

The Analogy: Imagine a detective reading a clue that says "The suspect wore a red hat." If the detective's next question is "Who wore a red hat?", they are "adopting" the term.
The Result: The researchers found that 54% of the new words the robots used in their next questions came directly from the text they had just read.
The Catch: This doesn't mean the robot understood the text perfectly, just that the words were there. It's like a parrot repeating words it heard, rather than a human thinking deeply about them. But it proves that the robots are at least looking at the evidence they retrieved.

5. The "Fixed Budget" Problem

The robots were given a "budget" for how many documents to read per question (e.g., read 5 pages, or 10 pages).

The Finding: Most robots treated this budget like a hard-coded setting. They didn't change it based on the task. Whether they were looking for a simple fact or solving a complex mystery, they mostly stuck to the same number of pages. They didn't seem to realize that a complex mystery might need them to read more pages than a simple fact.

Summary

This paper is like a traffic report for AI search agents. It tells us that:

Most searches are short and fast.
Robots get stuck in loops when looking for simple facts.
They prefer to zoom in on details rather than step back and look at the big picture.
They do pick up words from what they read, but they often stick to a rigid "reading budget" regardless of how hard the task is.

The authors released this data so other researchers can build better "traffic controllers" to help these robots break out of loops, explore more, and use their reading budget more wisely.

1. Problem Statement

The Information Retrieval (IR) community lacks empirical understanding of how LLM-powered agentic search operates in real-world scenarios. While benchmarks exist, they do not reveal:

How agents evolve their queries across multi-step sessions.
How retrieved evidence influences subsequent query reformulations.
How search behaviors differ based on the underlying user intent (e.g., fact-seeking vs. reasoning).
Whether agents effectively utilize context or fall into repetitive loops.

Current logs lack implicit feedback signals (like clicks) found in human search, creating a "measurement gap" where it is difficult to determine if an agent is successfully using retrieved evidence or merely hallucinating/repeating.

2. Dataset and Infrastructure

The authors analyze data from DeepResearchGym (DRGym), an open-source, model-agnostic search API accessed by external agentic clients.

Scale: 14.44 million search requests collected over six months (June–December 2025).
Sessions: Preprocessed into 3.97 million sessions.
Diversity: The logs cover 25 countries (primarily China and the US) and exhibit high semantic diversity (low average cosine similarity between queries), indicating they are not dominated by a few benchmark tasks or repetitive prompts.
Privacy: IP addresses are anonymized; client-side prompts and memory are not logged, meaning the analysis relies strictly on API-level traces (queries, parameters, timestamps, and returned evidence).

3. Methodology

The authors propose a three-component measurement framework to analyze session structure and query evolution:

A. Sessionization

Unlike human logs that use fixed time gaps, agentic requests can arrive in fast parallel bursts. The authors developed a semantic-continuity criterion combined with a 10-minute hard temporal cutoff:

They trained a semantic continuity model (MLP with Qwen3 embeddings) to score query pairs.
Queries are assigned to the most semantically continuous active session if the score exceeds a threshold; otherwise, a new session starts.

B. Intent and Trajectory Labeling (LLM-as-a-Judge)

Using gpt-5-nano, the authors annotated the data at two levels:

Session-Level Intent: Classified into three categories based on web search goal modeling:
- Declarative: Fact retrieval (e.g., "Who owns X?").
- Procedural: Method execution (e.g., "How to fix Y?").
- Reasoning: Complex synthesis (e.g., "Why is Z happening?").
Trajectory-Level Reformulation: Classified adjacent query pairs ( $q_k \to q_{k+1}$ $q_{k} \to q_{k + 1}$ ) into four types:
- Specialization: Narrowing constraints.
- Generalization: Broadening constraints.
- Exploration: Pivoting to a related facet.
- Repetition: Near-duplicate reformulations.

C. Context-driven Term Adoption Rate (CTAR)

To measure evidence traceability without access to internal agent attention, the authors introduced CTAR.

Definition: The fraction of newly introduced terms in a query ( $q_{k+1}$ ) that appear lexically in the retrieved evidence from previous steps.
Contexts:
- Last-step: Terms traced to the immediately preceding retrieval ( $E_k$ ).
- Aggregated: Terms traced to the union of all evidence retrieved so far ( $\bigcup E_i$ ).
Significance: This provides a conservative, interpretable metric for whether an agent is incorporating retrieved information into its next query, distinct from semantic paraphrasing.

4. Key Results

Session Structure and Dynamics

Length: Over 90% of multi-turn sessions contain at most 10 steps.
Speed: 89% of inter-step intervals are under one minute, indicating rapid iterative loops compared to human search.
Retrieval Depth: Most agents treat the number of documents retrieved ( $K$ ) as a static parameter (mostly $K \in \{1, 5, 10\}$ ), with only 1.35% of sessions varying $K$ across steps.

Intent-Conditioned Behaviors

Declarative (88.6% of sessions): Exhibit the highest repetition (near-duplicate loops), which increases over time. They use shallow retrieval but require more iterations.
Procedural (3.96%): Use deeper retrieval ( $K \approx 37$ ) and show semantically stable progressions with fewer refinement steps.
Reasoning (7.41%): Sustain the broadest exploration (pivoting) and show the highest semantic drift between the first and last query.

Trajectory Dynamics

Drill-Down Bias: Agents overwhelmingly favor Specialization and Exploration. Generalization (broadening) is the least used move (<11%) and often acts as a brief "reset" before re-specializing.
Stability vs. Turnover: Repetition maintains high result overlap (~~78%), while Exploration causes major evidence turnover (~~7% overlap).
Loops: Declarative sessions frequently enter high-stability "retry loops" (repeating similar queries) in later steps, suggesting potential stalls.

Evidence Traceability (CTAR)

Overall Adoption: On average, 54.35% of newly introduced query terms are lexically traceable to the aggregated evidence context.
Recency Effect: The last-step context accounts for 48.54% of adoption. The difference (~5.8%) indicates that agents also utilize evidence from steps prior to the immediate previous one.
Variation by Move:
- Specialization: High CTAR (~78%), indicating new terms are directly drawn from recent evidence.
- Repetition: Low CTAR (~21%), suggesting agents are retrying without incorporating new lexical terms from evidence.

5. Key Contributions

Large-Scale Behavioral Characterization: The first analysis of agentic search at the scale of 14M+ requests, revealing that agents operate in short, fast loops with static retrieval budgets.
CTAR Metric: A novel, interpretable metric to quantify evidence-conditioned query evolution without requiring access to internal agent states or semantic similarity thresholds.
Intent-Adaptive Insights: Demonstrated that search strategies (repetition vs. exploration) are heavily dependent on the task intent, challenging the notion of a uniform agentic search process.
Open Data: Release of the anonymized DRGym logs and analysis protocols to the public via HuggingFace.

6. Significance and Implications

The findings provide actionable signals for designing better agentic IR systems:

Repetition-Aware Stopping: High repetition rates in Declarative tasks suggest a need for controllers to detect "stall loops" and trigger strategy switches (e.g., forcing exploration or human review).
Intent-Adaptive Budgeting: Since Procedural tasks benefit from deeper retrieval while Declarative tasks rely on iteration, systems should dynamically adjust retrieval depth ( $K$ ) based on detected intent rather than using fixed parameters.
Cross-Step Context Tracking: The measurable traceability to older evidence (beyond the last step) suggests that context management modules should explicitly surface historical evidence terms to aid query formulation.
Evaluation Design: CTAR offers a lightweight, audit-ready signal for evaluating whether agents are effectively grounding their queries in retrieved evidence, bridging the gap between API logs and internal reasoning.

In conclusion, the paper establishes that while agentic search is powerful, it currently exhibits rigid behaviors (fixed budgets, repetitive loops) that can be optimized through intent-aware control policies and better context tracking mechanisms.