Technical Summary: SAGE – Benchmarking and Improving Retrieval for Deep Research Agents

1. Problem Statement

Deep research agents have emerged as powerful systems capable of addressing complex queries by iteratively searching and synthesizing information across multiple sources. While Large Language Model (LLM)-based retrievers have demonstrated strong capabilities in instruction following and reasoning, a critical question remains: Can LLM-based retrievers effectively contribute to deep research agent workflows?

Existing commercial agents typically rely on proprietary search APIs over large web corpora, which often depend on surface-form matching. In contrast, LLM-based retrievers are designed for semantic understanding. However, it is unclear whether the specific query generation behaviors of deep research agents (which may differ from natural language queries) align with the strengths of semantic retrievers. Furthermore, existing datasets for scientific literature search often rely on outdated papers or fail to evaluate the multi-step reasoning capabilities required by modern agents.

2. Methodology

2.1 The SAGE Benchmark

To investigate this, the authors introduce SAGE (Scientific AGentic retrieval Evaluation), a benchmark designed for reasoning-intensive scientific literature retrieval.

• Corpus: A collection of 200,000 up-to-date, open-access scientific papers spanning four domains: Computer Science, Natural Science, Healthcare, and Humanities.
• Queries: 1,200 queries divided into two types:
  1. Short-form Questions: Require intensive reasoning across metadata, figures/tables, and inter-paper relationships (e.g., citation overlaps). Answers are verifiable and unique.
  2. Open-ended Questions: Grounded in real-world research scenarios (e.g., literature reviews), requiring the synthesis of multiple relevant papers. Answers have multiple ground truths weighted by relevance.
• Data Curation: Questions are generated by prompting LLMs (GPT-5-mini) to create tasks that necessitate synthesizing information from paper metadata, extracted figures/tables, and reference overlaps (papers sharing at least four common references).

2.2 Experimental Setup

The study evaluates the performance of deep research agents under two conditions:

1. Web Search: Evaluating six agents (including proprietary systems like GPT-5, Gemini-2.5-Pro, and open-source DR Tulu) using their native web search capabilities.
2. Corpus Search: Using DR Tulu as a controlled backbone agent, replacing its web search tool with specific retrievers:
   • BM25: A traditional sparse lexical retriever.
   • gte-Qwen2-7B-instruct: A general LLM-based retriever.
   • ReasonIR: An LLM-based retriever specifically trained for reasoning-intensive tasks.

2.3 Proposed Solution: Corpus-Level Test-Time Scaling

To address the observed performance gap, the authors propose a corpus-level test-time scaling framework. Instead of modifying the agent's query generation, this approach augments the document corpus itself.

• Mechanism: An LLM (Qwen3-Next-80B-A3B-Instruct) processes each paper to extract bibliographic metadata and eight topic-relevant keywords summarizing core contributions.
• Implementation: These keywords and metadata are prepended to the document text. This enriches the corpus with signals that make retrieval easier for off-the-shelf retrievers, particularly those relying on keyword matching.

3. Key Results

3.1 Agent Performance on Web Search

• Proprietary vs. Open Source: GPT-5 leads overall on short-form questions. However, the open-source DR Tulu is competitive, outperforming closed-source Gemini-2.5 series agents on short-form questions.
• Search Efficiency: Search quantity is not the primary driver of accuracy. Agents that perform better (like GPT-5) tend to use more accurate query decomposition and targeted evidence selection rather than brute-force searching.
• Query Decomposition: Proprietary models tend to generate phrasal, semantically structured queries, while DR Tulu often generates less structured, keyword-concatenated sub-queries.

3.2 Retriever Performance in Corpus Search

• BM25 Dominance: Surprisingly, BM25 significantly outperforms LLM-based retrievers by approximately 30% on short-form questions.
  • Reasoning: Deep research agents generate keyword-oriented sub-queries. This behavior aligns well with surface-form matching (BM25) but causes a "Query-Retriever Mismatch" for LLM-based retrievers, which are trained on natural language queries and falter when faced with fragmented keyword strings.
• Open-Ended Questions: The performance gap narrows for open-ended questions. BM25 and gte-Qwen achieve comparable results, while ReasonIR ranks last on both query types.
• Diversity Issues: LLM-based retrievers exhibit lower diversity in retrieved documents (Unique References per Search) compared to BM25, especially under fixed search budgets.

3.3 Impact of Corpus-Level Scaling

• Short-Form Gains: Applying the corpus-level scaling (adding keywords/metadata) yields an 8.18% improvement for BM25 on short-form questions. LLM-based retrievers show only modest gains (0.9%–2.5%).
• Open-Ended Gains: Improvements are marginal (1.7%–2.5%) across all retrievers. The authors attribute this to the low diversity in the agent's query decomposition, which limits the breadth of evidence retrieved regardless of corpus enrichment.

3.4 Ablation Study

• Information Sensitivity: The importance of query components (metadata, multimodal details, relationships) shifts based on the search method. Under web search, agents are sensitive to paper details; under corpus search, inter-paper relationships become dominant.
• Retriever-Awareness: The retrieval backend largely determines which parts of the query information drive performance.

4. Significance and Claims

The paper claims to provide the first systematic investigation into the collaboration between deep research agents and LLM-based retrievers within a controlled scientific domain.

• Critical Finding: The authors highlight a significant misalignment: existing deep research agents generate keyword-oriented sub-queries that are better served by traditional sparse retrievers (BM25) than by state-of-the-art LLM-based semantic retrievers.
• Practical Contribution: The proposed corpus-level test-time scaling offers a practical, training-free method to improve retrieval performance by adapting the corpus to the agent's query style, rather than attempting to retrain the agent or the retriever.
• Benchmark Utility: SAGE provides a rigorous, up-to-date benchmark for evaluating reasoning-intensive retrieval, addressing the limitations of previous datasets that relied on outdated literature or lacked multi-step reasoning requirements.

The authors conclude that effective collaboration between retrievers and agents requires further adaptation, specifically noting that current agents do not yet adapt their query generation strategies to the underlying retriever type.