💬 NLP

Equal Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents

This paper argues that commercial search APIs should be evaluated not just by answer accuracy, but as "decision surfaces" whose distinct snippet and ranking characteristics significantly influence agent retrieval budgets and exploration strategies, even when final answer performance remains comparable across providers.

Original authors: Sriram Selvam, Anneswa Ghosh

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sriram Selvam, Anneswa Ghosh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you can't look at the crime scene yourself. Instead, you have to rely on a "search engine" to hand you a stack of index cards. Each card has a URL (the address), a title, and a tiny snippet of text. Based only on these cards, you decide: "Do I have enough info to solve this now?" or "Do I need to spend extra time and money to go read the full article at that address?"

This paper is about how the design of those index cards changes your detective work, even if you end up solving the same number of mysteries.

The Big Misunderstanding

For a long time, people thought that if two search engines gave you the same number of correct answers, they were basically the same tool. The paper argues this is like saying two different maps are identical just because they both get you to the same destination. It misses the journey.

The authors suggest that a search engine isn't just a list of links; it's a "decision surface." It's the specific set of clues (snippets, titles, and rankings) that forces your AI agent to make a choice: Stop and answer, or keep digging?

The Experiment: A Controlled Detective Game

To test this, the researchers set up a strict game. They froze the detective (an AI agent), the rules, and the tools. The only thing they changed was the search engine provider handing out the cards. They tested three: Brave, Tavily, and Firecrawl.

They asked 100 very hard questions (from a dataset called SEALQA-HARD). Here is the twist: All three search engines got the final answer right about the same number of times (25, 25, and 26 out of 100). If you only looked at the scoreboard, you'd think they were identical.

But when the researchers looked under the hood at how the detective worked, the story changed completely.

The Three Different "Card Decks"

Even though the final score was tied, the "economy" of evidence was totally different for each provider:

Brave (The "Gold-Rich" Deck):
Brave handed out cards where the answer was often hidden right in the tiny snippet text. The detective could see the gold answer just by reading the card. However, because the cards were so helpful, the detective sometimes got confused by too much info or missed the specific card that mattered.
- The Stat: Brave offered pre-fetch support (clues visible before reading the full page) on 30 of the questions.
- The Catch: Even though the clues were there, the agent only used them to solve the case immediately on 13 of the 101 "gold-supporting" rows (about 13%).
Tavily (The "Top-Heavy" Deck):
Tavily was different. It didn't give as many answers in the snippets, but when it did have the right answer, it put that card at Rank 1 (the very top).
- The Stat: For the questions where the answer was visible in the snippet, 17 out of 34 (50%) were at the very top spot.
- The Implication: If your detective has a rule like "Always read the first card," Tavily is a super-efficient partner.
Firecrawl (The "Explorer" Deck):
Firecrawl's cards were less likely to have the answer right on the front. This forced the detective to be more adventurous. Under the same rules, the agent using Firecrawl ended up clicking "fetch full page" more often.
- The Stat: The agent fetched pages for 81% of the questions with Firecrawl, compared to 65% with Brave.
- The Catch: This meant the agent spent more tokens (digital money) and time exploring, even though the final accuracy was the same.

The "Contradiction" Trap

The paper also found something scary: sometimes the cards lied. The researchers measured how often a snippet contradicted the true answer.

Brave: 0.92 contradictions for every 1 correct clue.
Firecrawl: 2.59 contradictions for every 1 correct clue.
This means with Firecrawl, the detective had to wade through more confusing or conflicting noise before finding the truth.

The Verdict

The paper concludes that choosing a search engine is a policy decision, not just a quality one.

If you pick Brave, you get a lot of info upfront, but you might need a strategy to filter it.
If you pick Tavily, you should trust the top result, because that's where the good stuff hides.
If you pick Firecrawl, you need to be ready to spend more resources digging deeper.

The authors suggest that we stop treating search APIs as simple "list makers" and start treating them as decision surfaces. The right choice depends on how your agent is programmed to act. A "one-size-fits-all" approach doesn't work because, as this study suggests, equal accuracy can hide very different, and sometimes very expensive, internal journeys.

So, the next time you see two AI agents giving the same answer, don't assume they took the same path. One might have found the treasure on the first map, while the other had to dig through a whole mountain of dirt to get there.

Technical Summary: Equal Accuracy, Unequal Evidence

Problem Statement
Traditional evaluation of search APIs for tool-using agents focuses primarily on final answer accuracy. However, in modern agent architectures utilizing progressive disclosure, agents first view ranked snippets, titles, and metadata before deciding whether to fetch full pages. The authors argue that treating search APIs merely as ranked-list retrievers is insufficient. Instead, a commercial search API functions as a decision surface: the specific evidence state (snippets, URLs, metadata) exposed to the agent before a fetch decision determines whether the agent answers immediately, searches again, or spends tokens fetching pages. The paper posits that two providers can yield similar aggregate answer accuracy while exposing fundamentally different evidence economies, leading to divergent costs, latencies, and failure modes.

Methodology
The study employs a controlled experimental protocol to isolate the impact of the search provider on agent behavior:

Fixed Agent & Tools: A single frozen GPT-5.4 agent is used with a fixed prompt, maximum iterations (10), and toolset (search_web and fetch_page). The fetch_page backend is shared (Jina Reader) across all conditions to ensure page content extraction is identical.
Variable: The only experimental variable is the search provider: Brave, Tavily, or Firecrawl.
Dataset: A deterministic, stratified sample of 100 questions from the SEALQA-HARD benchmark, selected for requiring exploration beyond simple snippet answers.
Oracle Protocol: A "per-URL oracle" using Kimi-K2.6 labels every content element visible to the agent (URL, title, snippet, and fetched page). The judge operates without outside knowledge, labeling whether evidence contains the gold answer, supports the model's answer, contradicts the gold answer, or is garbage.
Metrics:
- Semantic Match: An audited label for final answer correctness.
- Decision Partition: A four-way classification of provider-query pairs based on pre-fetch support and agent action: SMART (pre-fetch support exists and agent fetches it), MISSED (pre-fetch support exists but agent does not fetch), BLIND (no pre-fetch support, agent fetches), and NO-OP (no support, no fetch).
- Surface Contradiction-to-Gold Ratio ( $r_{c:g}$ ): The ratio of snippet-only URLs contradicting the gold answer to those containing it.

Key Contributions

Formalization of Decision Surfaces: The paper reframes commercial search APIs not as static retrievers but as decision surfaces that dictate the agent's grounding state and retrieval budget allocation.
Per-URL Oracle Protocol: Introduction of a rigorous labeling method that distinguishes between pre-fetch surface support (available in snippets/metadata) and post-fetch discovered support (found only after opening a page).
Decision Partitioning: Definition of a four-way partition (SMART, MISSED, BLIND, NO-OP) to analyze agent behavior relative to the evidence available at the decision surface.
Evidence Economy Analysis: Demonstration that equal answer accuracy masks significant differences in provider behavior, including pre-fetch support density, rank concentration, contradiction rates, and exploration patterns.

Results

Accuracy Parity: Under the semantic match metric, all three providers performed similarly (25/100 for Brave and Tavily, 26/100 for Firecrawl). Statistical analysis (paired-bootstrap intervals) confirms these differences are not meaningful.
Divergent Evidence Economies:
- Brave: Exposed the highest volume of pre-fetch support (30 queries) but had low rank-1 concentration (13% of gold-supporting rows were rank 1). This led to a "snippet-rich" surface where agents often missed available support (MISSED) rather than fetching.
- Tavily: Showed the highest rank-1 concentration (50% of gold-supporting pre-fetch rows were rank 1), suggesting top-result fetch policies could be highly efficient. However, it had lower total pre-fetch support (16 queries).
- Firecrawl: Associated with the highest rate of BLIND exploration (81% of queries involved fetching without pre-fetch support) and the highest contradiction-to-gold ratio (2.59), indicating a surface that encourages broader exploration but introduces more noise.
Complementarity: Only 10 questions were answered correctly by all three providers. Provider choice significantly altered which questions were solved, not just the aggregate count.
Retrieval Proxies: Deterministic proxies (e.g., gold URL hits, answer string presence) were insufficient; agents frequently failed (53–57 wrong answers) despite the answer text being visible in the trajectory, highlighting that the decision surface determines if evidence is actionable.

Significance and Claims
The paper claims that provider choice is a policy decision, not merely a recall decision. Because different providers create different decision surfaces (e.g., snippet-rich vs. rank-concentrated vs. volume-heavy), a fixed agent policy yields different utilities depending on the provider.

For Practitioners: Evaluation must move beyond final accuracy to include metrics like pre-fetch support, rank concentration, and contradiction ratios. A provider should be paired with a provider-aware fetch policy (e.g., a top-result policy for Tavily, a snippet-analysis policy for Brave).
For Providers: Search APIs for agents should optimize for actionability (calibrated snippets, stable IDs, low contradiction) rather than just relevance.
Limitations: The study is observational (agent actions were not randomized), relies on a single agent and judge, and uses a specific sample size. The authors explicitly state they do not claim one provider is universally superior, but rather that their surfaces induce different retrieval regimes.

The study concludes that evaluating search APIs for agents requires looking beyond answer accuracy to understand the retrieval budget and policy conditions under which grounded answers are generated.