🤖 AI

Fetch-then-Explore: Decoupling Selection from Extraction over a Persistent Workspace for Search Agents

The paper introduces "Fetch-then-Explore," a search agent framework that decouples page selection from evidence extraction by maintaining a persistent filesystem workspace, enabling agents to revisit and extract information from previously fetched pages on demand, which significantly improves accuracy on complex search benchmarks compared to traditional transient interfaces.

Original authors: Qi Liu, Yiqun Chen, Zidan Chen, Yan Gao, Yi Wu, Yao Hu, Jiaxin Mao, Fengbin Zhu, Tat-Seng Chua

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Qi Liu, Yiqun Chen, Zidan Chen, Yan Gao, Yi Wu, Yao Hu, Jiaxin Mao, Fengbin Zhu, Tat-Seng Chua

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive mystery, but instead of a single crime scene, the clues are scattered across the entire internet. You have a super-smart assistant (an AI agent) who can talk to you, think through problems, and use tools to find answers. This is the world of AI search agents. These digital detectives don't just read a single book; they have to hunt down information across thousands of websites, piece together facts, and figure out the truth.

To do this, the agent needs two main skills: finding the right web pages and reading them. Think of "finding" as using a search engine to get a list of potential clues. "Reading" is the tricky part. In the past, when the agent clicked a link, it had to swallow the whole page whole, or at least a big chunk of it, immediately. It was like being handed a 500-page novel and told, "Read this right now and tell me the ending," even though you didn't know which chapter held the answer yet. If you missed a detail, you had to throw the book away, go back to the library, and check it out all over again. This paper explores a smarter way to handle these digital books so the detective doesn't have to keep re-reading the same pages.

The Problem: The "Read-It-Now-or-Lose-It" Trap

Imagine you are playing a high-stakes game of "Clue" with a friend who is an AI. You ask, "Who co-founded Acme Corp?" The AI searches the web and finds a list of links.

In the old way of doing things (called Visit-and-Read), the AI picks a link and immediately forces the entire page into its short-term memory. It's like the AI grabs a book, reads the first few pages, summarizes them, and then shoves the rest of the book into a trash can. If the AI later realizes, "Wait, I need to know the second founder, not the first," it can't look back at the book. The book is gone. The AI has to go back to the library, find the book again, and read it all over. This is slow, expensive, and often leads to the AI missing crucial details because it was forced to read the wrong part of the page too early.

Another method, called Browsing, is slightly better. It lets the AI open a page and look around before summarizing. But there's a catch: the AI can only hold one page open at a time. As soon as it opens a new page, the old one vanishes from its memory. It's like having a single window on your computer; if you open a new tab, the old one closes. If the AI needs to compare facts from Page A and Page B, it has to keep flipping back and forth, re-opening and re-reading them constantly.

The Solution: The "Digital Filing Cabinet"

The authors of this paper propose a new method called Fetch-then-Explore. They realized that the problem isn't just how the AI reads, but where it keeps the pages it finds.

Instead of forcing the AI to read a page immediately or holding just one page in a temporary window, they give the AI a persistent workspace—think of it as a personal digital filing cabinet on a hard drive.

Here is how it works:

Fetch (The Librarian): When the AI finds a promising link, it doesn't read it yet. It just "fetches" the page and saves the whole thing into its filing cabinet. It gets a tiny receipt that says, "Page saved at this location." This costs almost nothing.
Explore (The Detective): Later, when the AI has a specific question (like, "Who was the co-founder?"), it goes to the filing cabinet. It doesn't need to re-download the page. It just uses a tool called grep (like a super-fast search function) to find the exact line in the saved file that mentions "co-founder."

This is like having a library where you can check out a book, put it on your shelf, and come back to it days later without having to re-check it out. The AI can save dozens of pages, then come back to any of them anytime to ask, "What did you say about X?" or "Show me the part about Y."

What They Found: The Power of Going Back

The researchers tested this new method against the old ways using three different "brain" models (AI backbones) and two challenging internet search benchmarks: BrowseComp (which requires finding one very hard-to-find fact) and WideSearch (which requires filling out a big table with many facts).

The results were clear:

Better Accuracy: The "Fetch-then-Explore" method was the winner. On the hard fact-finding test (BrowseComp), it beat every other method across all three AI brains. For example, with the strongest AI brain, it achieved 70.5% accuracy, compared to 66.0% for the next best method.
The Secret Weapon: The biggest reason for this success was revisiting. The AI using the filing cabinet went back to pages it had already seen 12% to 20% of the time on hard tasks, and up to 75% of the time on broad tasks. The old methods, which threw pages away, rarely went back.
Efficiency: Because the AI could re-read saved pages instantly, it didn't have to waste time searching the web again and again. It actually used fewer search queries than the other methods while getting better answers.

Why This Matters

The paper shows that the way an AI "holds" information is just as important as how it "thinks." By separating the act of selecting a page (saving it to the cabinet) from the act of extracting the answer (reading it later), the AI becomes much more flexible. It can form a hypothesis, check a saved page, change its mind, and check the page again without losing its place.

The authors suggest that this "persistent workspace" approach is a game-changer for long, complex research tasks. It turns the AI from a frantic reader who forgets everything as soon as it moves on, into a careful investigator who keeps all its evidence organized and ready to be re-examined whenever the mystery deepens. The study confirms that giving an AI a place to store its findings—and the freedom to look back at them—leads to smarter, more accurate answers.

Technical Summary: Fetch-then-Explore

Problem Statement

Current long-horizon search agents, which answer complex questions requiring dozens of searches, predominantly rely on two document access interfaces that tie page content to the moment of retrieval.

Visit-and-Read: This interface fuses document selection and evidence extraction into a single action. Upon visiting a URL, the agent immediately receives a fixed rendering (e.g., a goal-chunked window, LLM summary, or truncated body) injected into the message history. This forces the agent to commit to a specific reading before it knows which specific facts it will need. Content falling outside this initial rendering is lost unless the page is fetched again.
Stateful Browsing (Open-and-Find): This approach decouples selection from extraction by opening a page into a transient session. However, it holds only one page at a time; opening a new page releases the previous one. Consequently, if an agent needs to revisit a page later to verify a claim or read from a new angle, it must re-fetch and re-render the page, incurring the cost of a fresh retrieval and losing the ability to accumulate evidence across documents.

Both paradigms suffer from the inability to retain selected pages for deferred, repeatable extraction. A page that becomes relevant many turns later must be fetched and rendered into the context window all over again, making long-horizon reasoning inefficient and prone to missing evidence.

Methodology: Fetch-then-Explore (FTE)

The authors propose Fetch-then-Explore, a paradigm that decouples page selection from evidence extraction and retains selected content in a persistent, external workspace.

Core Architecture

FTE operates within a standard ReAct agent framework but replaces the document-access toolset with a set of tools acting on a per-question filesystem cache ( $C_q$ ).

Persistent Workspace: Instead of injecting page bodies into the context window or a transient session, FTE stores the full text of fetched pages as plain-text files in a temporary directory on the filesystem.
Decoupled Tools:
- fetch(url): Retrieves a page, normalizes it to text, and stores it in $C_q$ . It returns only a lightweight record (path, size, status) rather than the content. Selection becomes "almost free" as it does not consume context tokens for page text.
- grep(pattern, path, context): Searches the stored files for regex matches. It can search across the entire cache (all fetched pages) or a specific file, returning line-numbered hits with optional context.
- read(path, offset, limit): Reads a bounded window of lines from a specific file, allowing the agent to expand on a grep hit.
- list_fetched(): Enumerates the cache contents.

Design Principles

Decoupling: Selection (fetch) is separated from extraction (grep/read). The agent can select a page based on a snippet and defer extraction until its hypothesis sharpens.
Retention: Pages are not evicted when the agent moves to a new task. The workspace grows over the trajectory, allowing evidence to accumulate across documents.
Deferred & Repeatable Extraction: The agent can query the same page multiple times with different hypotheses without re-fetching.

Experimental Setup

The study isolates the impact of the document interface by holding the search backend (Serper API), the agent harness (ReAct), and the backbone models fixed.

Backbones: Three models spanning capability tiers: Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, and DeepSeek-V4-Pro.
Benchmarks:
- BrowseComp: A deep-research benchmark requiring multi-hop reasoning to find a single hard-to-find fact (1,266 questions, sampled 200).
- WideSearch: A broad information-seeking benchmark requiring high-recall table filling from many pages (200 tasks).
Baselines:
- Snippet-only: No document access.
- Visit-Snippet / Visit-Summary: Standard visit-and-read with different compression levels.
- Open+Find: Stateful browsing with transient sessions.

Key Results

1. Performance Gains

BrowseComp: FTE achieves the highest accuracy across all three backbones (44.5%, 52.5%, and 70.5% respectively), outperforming the strongest baselines by 2–4.5 percentage points and the raw visit-snippet baseline by 7–14 points.
WideSearch: FTE generally matches or exceeds baselines. It leads on the 122B backbone and ties the strongest baseline on the 122B/Pro backbones, though the visit-summary baseline edges it slightly on the weakest backbone.
Search Efficiency: FTE agents issue fewer search queries than baselines. Because they can extract evidence from held pages on demand, they rely less on re-searching to find information they already possess.

2. Behavioral Analysis

Revisiting Pages: The defining behavioral difference is the ability to return to a page after leaving it. FTE revisits 12–20% of documents on BrowseComp and 67–76% on WideSearch, compared to 3–10% for transient interfaces.
Ablation Studies:
- Removing grep (forcing blind paging) or read significantly drops performance, confirming both tools are necessary.
- Restricting grep to a single file (removing cross-cache scope) causes a performance drop comparable to removing an extraction tool entirely. This indicates that persistence alone is insufficient; the ability to query the entire accumulated workspace is critical.

Significance and Contributions

The paper claims three primary contributions:

Identification of Persistence: It identifies "persistence" as an unexplored axis in search agent document interfaces. While existing agents either fuse selection/extraction or use transient sessions, FTE introduces a persistent, external workspace.
Controlled Isolation: By fixing the harness and search backend, the study isolates the value of the document interface. The results demonstrate that externalizing document access to a persistent cache leads to measurable gains in end-task accuracy, particularly for deep-research tasks (BrowseComp).
Mechanism of Gain: The analysis traces performance improvements to the specific capability of returning to a page after leaving it. This allows agents to correct incomplete hypotheses and recover missed evidence without the cost of re-fetching, a capability that transient interfaces fundamentally lack.

The authors conclude that decoupling operations from the medium holding their output raises structural questions about where an agent's working memory should live, suggesting that external, persistent substrates are a viable and effective alternative to context-window-based memory for long-horizon search.