🤖 AI

LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?

This paper reveals that current LLM-based search agents often rely on intrinsic knowledge rather than genuine web searching, prompting the introduction of LiveBrowseComp, a dynamic benchmark using recent, non-salient facts to effectively evaluate true evidence-driven discovery capabilities.

Original authors: HuiMing Fan, Xiao Wang, Zheng Chu, Qianyu Wang, Zhuoyao Wang, Ming Liu, Bing Qin, XingYu

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: HuiMing Fan, Xiao Wang, Zheng Chu, Qianyu Wang, Zhuoyao Wang, Ming Liu, Bing Qin, XingYu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Are They Exploring or Just Checking Their Homework?

Imagine you hire a super-smart detective (an AI agent) to find a specific, obscure fact. You give them a library card (internet access) and tell them, "Go find the answer."

The paper asks a suspicious question: Is this detective actually searching the library to find new information, or are they just opening a book they already memorized, finding the answer in their head, and then using the library card only to say, "See? I told you I was right"?

The authors call this behavior Intrinsic Knowledge Dependence (IKD). It's like a student who has already memorized the answers to a practice test. When they take the real test, they don't need to study; they just write down what they remember and use the "open book" rule to double-check that their memory is correct.

The Problem with Current Tests (The "Static" Benchmarks)

Currently, we test these AI detectives using "Static Benchmarks" (like BrowseComp). Think of these as old, dusty trivia quizzes.

The Issue: Because these quizzes have been around for a while, the AI models have likely already "read" the questions during their training.
The Result: The AI gets a high score, but it's not because it's good at searching. It's because it's good at remembering. It guesses the answer from its memory and uses the internet just to feel confident.

The Three "Lie Detector" Tests

To prove their suspicion, the researchers ran three simple experiments on existing AI models:

The "No-Tools" Test: They took away the internet.
- Result: The AIs still got about 44% of the answers right. This proved they were relying on their internal memory, not the search tool.
The "Broken Library" Test: They let the AI use the internet, but they secretly removed all the pages that actually contained the correct answer. The AI could only see irrelevant junk.
- Result: The AIs' scores crashed. Instead of ignoring the junk and sticking to their memory, they got confused and gave wrong answers. This showed they weren't using the search to discover truth; they were using it to confirm what they already guessed.
The "Trail of Clues" Test: They watched the AI's search history.
- Result: They found that over 50% of the search questions the AI asked were based on its own guesses, not on things it actually found on the web. It was chasing its own tail.

The Solution: Introducing "LiveBrowseComp"

To fix this, the researchers built a new test called LiveBrowseComp.

The Analogy:
If the old tests were like a Jeopardy! episode from last year (which the AI might have seen on TV), the new test is like a live news broadcast happening right now.

Freshness: The questions are based on facts published in the last 90 days. The AI couldn't have memorized these because they didn't exist when the AI was "born" (trained).
Obscurity: The facts aren't famous headlines (like "Who won the Super Bowl?"). They are "long-tail" facts, like "What was the specific earthquake magnitude in a small town in Peru three weeks ago?" or "What is the name of a niche video game released yesterday?"
The Goal: To force the AI to actually hunt for information, rather than just guessing from memory.

What Happened When They Ran the New Test?

The results were shocking:

Memory Fails: When the AI tried to answer these new questions without using the internet (Closed-Book), their score dropped to less than 2%. They knew almost nothing about these fresh facts.
Search Scores Dropped: Even when allowed to use the internet, the AI scores dropped by 25–40 points compared to the old tests.
Rankings Changed: The AI models that were "winners" on the old tests (because they had good memories) suddenly became average or poor on the new test. The models that were good at actually searching rose to the top.

The Human Comparison

The researchers also asked real humans to solve these questions.

The Finding: Humans took about the same amount of time to solve the "Old" questions and the "New" questions.
The Meaning: The new questions aren't "harder" or "smarter." They are just new. The only reason the AI struggled was that it couldn't rely on its memory shortcut.

The Bottom Line

The paper concludes that current AI evaluations are flawed because they reward memory instead of searching.

Old Way: "Can you guess the answer and then find a website to back it up?" (AI gets an A).
New Way (LiveBrowseComp): "Can you find an answer you have never seen before?" (AI gets an F).

The authors argue that to truly test if an AI is a good "Search Agent," we must test it on things it doesn't already know, forcing it to do the hard work of discovery rather than just verification.

Technical Summary: LiveBrowseComp

Problem Statement

The paper addresses a critical evaluation gap in the assessment of Large Language Model (LLM) based search agents. While agents demonstrate high performance on static benchmarks like BrowseComp, the authors question whether this success stems from genuine evidence-driven discovery or merely the verification of pre-existing parametric knowledge. The central problem identified is Intrinsic Knowledge Dependence (IKD): a failure mode where agents rely on information encoded in their parameters before retrieval, using search tools primarily to confirm hypotheses they have already generated internally. This conflation of "what agents know" with "what agents can find" leads to benchmarks that reward memory-backed verification rather than the ability to discover unknown information.

Methodology

The authors employ a two-phase methodology: a diagnostic pilot study on existing benchmarks and the construction of a new benchmark designed to neutralize IKD.

1. Diagnostic Pilot Study

To quantify IKD, the authors conducted three diagnostic experiments on four static benchmarks (BrowseComp, BrowseComp-ZH, HLE, GAIA) using a unified search-agent scaffold across multiple frontier models (e.g., DeepSeek, Kimi, MiniMax, GLM):

Closed-Book Coverage: Measuring performance with all search tools disabled. This establishes the baseline of intrinsic knowledge available to the agent.
Evidence-Blocked Search: Removing all answer-supporting documents from the retrieval index while keeping the search interface active. This tests whether agents can maintain performance when the "verification" channel is broken.
Trajectory Grounding: Analyzing search logs to determine if queries are seeded by retrieved evidence or by the model's own internal hypotheses.

2. LiveBrowseComp Construction

To evaluate agents beyond their intrinsic knowledge boundary, the authors introduced LiveBrowseComp, a deep-search benchmark with specific design constraints:

Temporal Recency: Questions are derived from facts published within the 90 days preceding benchmark construction, ensuring they fall outside typical training data cutoffs.
Long-Tail Obscurity: Sources (GDELT, TMDB, RAWG, CVE/NVD, SportsDB, USGS) are filtered to exclude globally salient events, retaining obscure but verifiable facts.
Stability and Uniqueness: Questions are filtered to ensure answers are stable (not changing rapidly) and unique.
Human Validation: A rigorous multi-stage process involving professional annotators and independent verifiers ensures questions require multi-step reasoning, cannot be solved via pre-existing knowledge, and have a single correct answer.

Key Contributions

Identification of Intrinsic Knowledge Dependence (IKD): The paper formally defines and empirically demonstrates that current search agents often function as verification interfaces for parametric knowledge rather than discovery engines.
Diagnostic Framework: The authors provide a set of diagnostics (closed-book, evidence-blocking, trajectory analysis) to detect IKD in existing evaluations.
LiveBrowseComp Benchmark: A new dataset of 335 human-authored questions specifically engineered to suppress IKD by relying on recent, long-tail facts.
Re-evaluation of State-of-the-Art: The paper re-ranks leading models on LiveBrowseComp, revealing that static benchmark rankings do not reliably predict performance on live search tasks.

Experimental Results

High Intrinsic Reliance: On static benchmarks, agents answered up to 44.5% of questions correctly in a closed-book setting (without tools).
Evidence-Blocking Collapse: When answer-supporting evidence was removed, performance dropped drastically. For example, MiniMax M2.5 fell from 44.5% (closed-book) to 8.0%, and Kimi-K2.6 from 25.5% to 2.3%. This indicates that search often degrades performance when it cannot confirm internal hypotheses.
Query Provenance: Over 50% of search queries were model-originated (seeded by internal reasoning) rather than retrieval-originated. Furthermore, agents used answer-supporting evidence in less than 33% of cases where it was retrieved.
LiveBrowseComp Performance:
- Closed-Book Accuracy: All evaluated models fell below 2% on LiveBrowseComp, confirming the temporal and long-tail constraints effectively neutralized intrinsic knowledge.
- Search-Augmented Scores: Scores dropped by 25–40 points compared to static benchmarks.
- Ranking Shift: The correlation between static benchmark scores (BrowseComp) and LiveBrowseComp scores weakened significantly (Pearson $r$ dropped from 0.79 to 0.53). Models that excelled on static benchmarks due to broad knowledge did not necessarily perform best on LiveBrowseComp.
- Behavioral Change: On LiveBrowseComp, the distribution of search turns shifted from short, verification-heavy trajectories to longer, more exploratory paths, indicating a shift toward genuine information seeking.

Significance

The paper argues that static benchmarks increasingly conflate a model's parametric knowledge with its search capabilities, creating a misleading picture of agent progress. LiveBrowseComp demonstrates that when the "memory backstop" is removed, current search agents struggle to discover information they do not already know. The authors conclude that future evaluation must prioritize dynamic, time-sensitive benchmarks to distinguish between "guess-and-verify" behavior and true evidence-led discovery, urging a shift in training signals to reward genuine search capabilities.