💬 NLP

Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search

This paper demonstrates that static retrieval utility fails to predict causal utility in multi-step agentic search, revealing that approximately one-third of documents essential for an agent's success ("bridge documents") appear useless to static readers because their true value lies in providing discriminative entities that redirect future queries rather than directly answering the current question.

Original authors: Debayan Mukhopadhyay, Utshab Kumar Ghosh, Shubham Chatterjee

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Debayan Mukhopadhyay, Utshab Kumar Ghosh, Shubham Chatterjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, multi-layered mystery, like figuring out who stole the cookie jar by asking a series of questions. You have a super-smart detective (an AI agent) and a massive library of books (a search engine). In the old days, scientists thought the best way to help the detective was to hand them the book that contained the final answer right away. They measured a book's "usefulness" by asking: "If you read this book and the question, can you solve the puzzle?" If the book had the answer, it got a gold star. If it didn't, it was tossed aside.

But here's the twist: modern detectives don't just read one book and stop. They ask a question, read a bit, realize they need more info, ask a new question, and keep going. This is called "multi-step agentic search." The problem is, the old way of grading books doesn't work for this kind of detective. A book might look completely useless on its own—it doesn't have the final answer—but it might contain a single, tiny clue (like a name or a place) that allows the detective to ask the next question that actually solves the case. If you take that "useless" book away, the detective gets stuck and fails. This paper investigates exactly that gap: the difference between a document that looks good on paper and a document that actually helps a detective solve a mystery step-by-step.

The "Bridge" That Looks Like a Dead End

The researchers at the University of Calcutta and Missouri University of Science and Technology decided to test a big assumption: Does a document that looks helpful to a static reader also help a searching agent?

To find out, they set up a digital experiment using 1,000 tricky, multi-step questions (from a dataset called HotpotQA). They watched a smart AI agent try to solve them. The agent would read a document, think, ask a new question, and read again. But here is the clever part: for every single document the agent read, the researchers played a "what if" game. They deleted that specific document from the agent's view and let the agent try to finish the rest of the journey without it.

They called this Counterfactual Trajectory Exploration. It's like watching a movie, pausing it, erasing one scene, and seeing if the rest of the movie falls apart.

They measured two things for every document:

Static Utility (SRU): If you gave the document to a fresh reader with just the original question, would it help? (The "Gold Star" test).
Causal Utility (CTU): If you removed the document, did the agent fail, get a worse answer, or have to ask more questions? (The "Real Help" test).

The Shocking Discovery: They Are Unrelated

The results were surprising. The researchers analyzed over 23,322 documents the agent read. They found that the two tests were almost completely unrelated. The statistical link between them was -0.026, which is basically zero.

In plain English: A document that looks useless to a standard reader is often the most important thing for a searching agent.

They discovered a special category of documents they call "Bridge Documents."

The Scenario: Imagine the question is, "What political party was the 6th governor of Hawaii who passed bills for global warming?"
Step 1: The agent finds a document saying, "Linda Lingle was the 6th Governor of Hawaii." This document does not mention political parties or global warming. A standard reader would say, "This is useless! It doesn't answer the question."
Step 2: But the agent reads it, learns the name "Linda Lingle," and uses that name to ask its next question: "What is Linda Lingle's political party?"
Step 3: That second search finds the answer.

If the researchers had deleted that first "useless" document, the agent would never have known to ask about Linda Lingle. It would have failed. That first document was a Bridge: it didn't have the answer, but it built the path to the answer.

The study found that 35.72% of the documents the agent read were these "Bridge Documents." They were causally essential (the agent needed them to succeed) but looked completely useless to a static reader.

Why Do These Bridges Work?

The researchers didn't just stop at finding the bridges; they wanted to know why they worked. They hypothesized that these documents work because they hand the agent a specific "key"—a specific entity (like a name, place, or thing)—that the agent then uses in its next search.

They tested this by looking at Observable Entity Relevance (OER). This is a fancy way of asking: "Does this specific word or name appear in the right documents to help distinguish the good answers from the bad ones?"

They found that when a document contained a "discriminative" entity (one that helps separate good clues from bad ones), that entity showed up in the agent's next search query 4.02 times more often than entities found only in irrelevant documents.

6.1% of the time, important entities from good documents made it into the next question.
Only 1.5% of the time did irrelevant entities make the cut.

This proves that the agent isn't just guessing; it's mechanically picking up specific clues from "useless" documents and using them to pivot its search in the right direction.

What This Means for the Future

The paper concludes that the current way we train and test search engines is broken for this new style of AI. We are currently optimizing for documents that contain the answer right now, but for agents that think in steps, we actually need documents that contain the next clue.

The researchers are careful to note that they didn't solve the problem of how to find these bridges automatically yet. They just proved that the current tools are blind to them. They suggest that future systems should look for these "discriminative entities" (the keys that unlock the next step) rather than just looking for the final answer.

In short: Just because a document doesn't have the answer doesn't mean it's useless. Sometimes, it's the only thing holding the bridge together.

Technical Summary: Bridge Evidence in Multi-Step Agentic Search

Problem Definition

Current retrieval systems are trained and evaluated based on a static notion of usefulness: a document is considered valuable if, when paired with a question, it improves a reader model's answer. This assumption underpins standard metrics (nDCG, MAP, MRR) and the training objectives of learned rankers. However, this definition assumes a single-turn interaction where the document is read in isolation.

The paper argues that this assumption breaks down in multi-step agentic retrieval (e.g., ReAct-style agents). In these settings, an agent issues a sequence of queries, reasoning across turns. A document's value is often not in its direct contribution to the final answer, but in what it enables the agent to do next. A document might contain no answer string yet be "causally load-bearing" because it supplies a specific entity that redirects the agent's subsequent search. Conversely, a document containing the answer might be causally inert if the agent already possessed that information. The paper posits that static retrieval utility does not predict causal utility in agentic search, creating a gap where "bridge evidence" (documents that are useless statically but essential causally) is systematically overlooked.

Methodology

1. Counterfactual Trajectory Exploration (CTE)

To measure the causal contribution of a document without relying on static proxies, the authors introduce Counterfactual Trajectory Exploration.

Intervention: For every document an agent actually reads during a trajectory, the system performs an omission intervention: the document is removed from the evidence set, and the trajectory is replayed from that point forward with all other conditions (previous context, model parameters, hardware) held fixed.
Metric: This generates a Counterfactual Trajectory Utility (CTU) score, calculated as the sum of three normalized deltas between the original run and the counterfactual run:
1. $\Delta$ answer: Change in the F1 score of the final answer.
2. $\Delta$ next-query: Change in the nDCG@10 of the next query generated by the agent (measuring if the agent was steered toward better evidence).
3. $\Delta$ effort: Change in the number of turns required to reach a solution.
Thresholding: The authors derive a principled threshold for CTU. Since the raw deltas for a "no-op" document are zero, the normalized CTU for a document with no causal effect is exactly 1.400. Documents with CTU > 1.400 are considered causally helpful; those below are not.

2. Static RAG Utility (SRU)

To measure static utility, the authors compute Static RAG Utility (SRU). This involves feeding a document and the original question to a stateless reader model (without any trajectory history) and measuring the F1 improvement over a zero-shot baseline.

3. Experimental Setup

Dataset: A stratified sample of 1,000 questions from the HotpotQA development set.
Agent: A ReAct-style agent using Qwen2.5-7B-Instruct, searching a Wikipedia paragraph index (BM25 retrieval + Cross-Encoder reranking).
Scope: The study analyzes 23,322 document observations across multi-step trajectories.

4. Mechanism Analysis: Observable Entity Relevance (OER)

To explain why bridge documents work, the authors test the hypothesis that these documents supply discriminative entities. They utilize the Observable Entity Relevance (OER) metric, which measures whether an entity's presence in a document statistically separates relevant from non-relevant candidates. They track whether entities found in bridge documents propagate into the agent's next query.

Key Results

1. Independence of Static and Causal Utility

The core finding is that static utility (SRU) and causal utility (CTU) are statistically independent.

Correlation: The Spearman rank correlation ( $\rho$ ) between SRU and CTU is −0.026, indicating they measure unrelated quantities.
Quadrant Analysis: When plotting documents on a 2x2 grid (High/Low SRU vs. High/Low CTU):
- Cell C (Bridge Evidence): 35.72% of the documents the agent read were causally load-bearing (High CTU) but appeared useless to a static reader (Low SRU).
- Cell B (Redundant): 2.18% of documents were high SRU but low CTU (useful statically but redundant causally).
- Robustness: This pattern persists even when replacing the reader-based SRU axis with a proxy based on BM25 and Cross-Encoder scores. In this proxy quadrant, the bridge cell remains significant at 27.16%, and the correlation remains near zero ( $\rho = -0.016$ ).

2. The Mechanism: Entity Propagation

The study validates the mechanism behind bridge evidence:

Propagation Gap: Entities with high Observable Entity Relevance (OER) appear in the agent's next query 4.02 times more often (6.1% vs. 1.5%) than entities found only in non-relevant documents.
Conclusion: Bridge documents function by introducing discriminative entities that redirect the search trajectory. The signal that static rankers use to identify useful entities (OER) is the same signal that drives agentic behavior, even if the document itself scores poorly on static answer containment.

3. Limitations and Confounds

The authors explicitly address and withdraw certain secondary observations:

Turn Count Confound: An initial observation that bridge evidence increased with the number of turns was found to be an artifact. The grouping variable (counterfactual turn count) was mechanically tied to the $\Delta$ effort component of CTU, artificially inflating the bridge cell for longer trajectories.
Static Axis Skew: The SRU axis is highly compressed (only 3.3% of documents had positive SRU), meaning the 35.72% bridge figure largely restates the independence result rather than being a separate discovery.
Hypothesis H2 (Localization): The hypothesis that propagating entities are specifically localized to bridge documents (Cell C) rather than other high-CTU cells did not replicate at scale and is reported as inconclusive.

Significance and Claims

The paper claims that static relevance and causal usefulness are distinct quantities in agentic retrieval. Optimizing for static utility (what a reader needs) does not deliver causal utility (what an agent needs).

Bridge Evidence: The authors define "bridge documents" as those that enable future search steps without containing the answer themselves. They constitute a non-trivial portion (approx. one-third) of the evidence consumed by agents.
Implication for Training: The labels and metrics currently used to train and evaluate retrieval systems are misaligned with the needs of agentic search.
Future Direction: The paper suggests that Observable Entity Relevance (OER) could serve as a computable, non-counterfactual proxy for causal utility. Since OER predicts entity propagation (and thus agent redirection) by a factor of four, it offers a potential path to training rankers that serve agents rather than just static readers.

The authors conclude that the field must move beyond static evaluation to understand the "bridge" between current knowledge and the next necessary search step.