💬 NLP

Revisiting Text Ranking in Deep Research

This paper investigates the effectiveness of text ranking methods in deep research by analyzing retrieval units, pipeline configurations, and query characteristics on the BrowseComp-Plus dataset, revealing that specific retriever types and passage-level units perform well while proposing a query-to-question method to mitigate query mismatch.

Original authors: Chuan Meng, Litu Ou, Sean MacAvaney, Jeff Dalton

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chuan Meng, Litu Ou, Sean MacAvaney, Jeff Dalton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a super-smart research assistant (an AI agent) to find the answer to a very tricky question, like "Which football match in 2020 had exactly 61,880 people in the stands?"

To solve this, your assistant can't just guess; it has to go out, search the internet, read articles, think about what it found, and search again. This process is called Deep Research.

This paper is like a mechanic inspecting the engine of that research assistant. Specifically, the authors wanted to understand how the "text ranking" part works. Text ranking is the tool the assistant uses to decide which search results are the most important to read.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Black Box" Mystery

Previously, most researchers let these AI assistants use a "black box" search engine (like a standard Google API). They didn't know how the engine was picking results. The authors wanted to pull back the curtain and test different search engines themselves to see which ones actually work best for these AI agents.

2. The First Discovery: "Chapters" vs. "Whole Books"

The authors tested two ways of giving information to the AI:

Document Level: Giving the AI the whole web page (the whole book).
Passage Level: Giving the AI just the specific paragraph or section that matters (the specific chapter).

The Finding: Giving the AI just the passages (chapters) worked much better.

Why? AI assistants have a limited "memory" (called a context window). If you feed them whole books, they run out of memory quickly and have to stop searching. If you feed them just the relevant paragraphs, they can fit more information in their memory, ask more questions, and find the answer more accurately.
Analogy: It's like trying to solve a puzzle. If you dump the whole box of 1,000 pieces on the table, you get overwhelmed. If you only give the AI the 10 pieces that actually fit together, it solves the puzzle faster.

3. The Second Discovery: The "Keyword" vs. "Essay" Mismatch

The authors noticed something funny about how the AI agents search.

How Humans Search: We usually type natural questions like, "Who won the game where the attendance was 61,880?"
How AI Agents Search: The agents act like old-school search engines. They type short, keyword-heavy strings like: "61,880" "football" "attendance".

The Finding:

Old-School Tools Win: Because the AI agents search using keywords, the old-school search tool called BM25 (which is great at matching exact words) actually beat the fancy, modern AI-based search tools in most cases.
The Mismatch: The fancy AI search tools were trained on natural language questions (essays). When the agent gave them keyword strings, the AI tools got confused and performed poorly. It was like asking a poet to write a grocery list; they know how to write, but the format is wrong for the task.

4. The Third Discovery: The "Editor" (Re-ranking)

After the search tool finds a list of results, a second tool called a Re-ranker acts like an editor. It looks at the list and says, "Actually, this one is the most important, move it to the top."

The Finding:

Editors are Essential: Using a re-ranker significantly improved the results. It helped the AI find the right answer faster and with fewer search attempts.
Deep Editing Helps: The deeper the editor looked (checking more results before picking the top one), the better the results were.
The "Reasoning" Editor Didn't Help: One specific editor was designed to "think" and "reason" about why a result was good. However, because the search queries were so short and keyword-heavy, this "reasoning" editor often got confused and made mistakes. It was like a detective trying to solve a case with no clues; the extra thinking didn't help because the input was too messy.

5. The Solution: The "Translator" (Q2Q)

To fix the problem where the AI's keyword searches confused the fancy AI tools, the authors built a Translator.

How it works: Before the search tool looks for answers, the Translator takes the AI's keyword string (e.g., "61,880" "football") and turns it into a natural question (e.g., "Which football match had an attendance of 61,880?").
The Result: This simple translation made the fancy AI search tools work much better again. It bridged the gap between how the agent asks and how the tools learned to answer.

Summary of the "Mechanic's" Report

Don't feed the AI whole books; give it the specific paragraphs (passages) so it doesn't get overwhelmed.
Old-school keyword matching (BM25) is surprisingly powerful for these agents because they search like keyword machines.
An "Editor" (Re-ranker) is crucial to pick the best results.
If you use fancy AI search tools, you must translate the agent's keyword searches into natural questions first, or the tools will get confused.

The paper concludes that while AI agents are powerful, they still rely heavily on these established, sometimes old-school, information retrieval methods to work effectively.

Technical Summary: Revisiting Text Ranking in Deep Research

Problem Definition

Deep research is an emerging task where Large Language Model (LLM)-based agents perform iterative reasoning and open-web exploration to answer complex, multi-hop queries. While search is central to this process, most prior work relies on opaque, black-box web search APIs. This lack of transparency hinders a systematic understanding of how established text ranking methods perform in this specific setting. Furthermore, existing research often overlooks critical variables: the granularity of retrieval units (documents vs. passages), the configuration of ranking pipelines (retrievers, re-rankers, and depths), and the potential mismatch between the "web-search-style" queries issued by agents and the natural-language questions used to train neural rankers.

Methodology

The authors conduct a comprehensive reproduction and evaluation of text ranking methods on BrowseComp-Plus, a deep research dataset featuring a fixed corpus and human-verified relevance judgments. The study involves:

Agents: Two open-source LLM-based agents: gpt-oss-20b (20B parameters) and GLM-4.7-Flash (30B parameters). Both are trained for tool use and chain-of-thought reasoning.
Retrievers: Five retrievers spanning four paradigms:
- Lexical sparse: BM25.
- Learned sparse: SPLADE-v3.
- Single-vector dense: RepLLaMA (7B) and Qwen3-Embed-8B.
- Multi-vector dense: ColBERTv2.
Re-rankers: Three pointwise re-rankers representing different efficiency/effectiveness trade-offs: monoT5-3B (non-reasoning), RankLLaMA-7B (non-reasoning), and Rank1-7B (reasoning-based, generating CoT traces).
Experimental Variables:
1. Retrieval Units: Comparing performance on a document corpus (full web pages, often truncated to 512 tokens to fit context windows) versus a passage corpus (segments of ~250 words).
2. Pipeline Configurations: Varying retrievers, re-rankers, and re-ranking depths (10, 20, 50).
3. Query Mismatch: Evaluating the impact of agent-issued queries (keyword-heavy, quoted) versus natural-language questions. The authors propose a Query-to-Question (Q2Q) method to reformulate agent queries into natural language, with variants using only the query ( $Q$ ) or the query plus the agent's reasoning trace ( $Q+R$ ).

Key Findings

1. Retrieval Units and Lexical Performance (RQ1)

Passage Superiority: Passage-level units consistently outperform document-level units (without a full-document reader) in answer accuracy. This is particularly pronounced for agents with smaller context windows (e.g., gpt-oss-20b), as passages allow for more search and reasoning iterations before hitting token limits.
BM25 Dominance: Contrary to trends in some neural retrieval domains, BM25 achieves the highest accuracy (0.572) when paired with the passage corpus and gpt-oss-20b. This is attributed to the "web-search-style" syntax of agent queries (keywords, exact matches via quotes), which favors lexical matching.
Document Truncation Issues: BM25 performs poorly on the document corpus under default parameters due to a lack of proper document-length normalization. The authors demonstrate that tuning BM25 parameters (specifically increasing $b$ for length normalization) or truncating documents to 512 tokens significantly improves performance.
Neural Retriever Generalization: Learned sparse (SPLADE-v3) and multi-vector dense (ColBERTv2) retrievers generalize better to web-search-style queries than single-vector dense models (RepLLaMA, Qwen3-Embed-8B), despite the latter having larger model sizes.

2. Re-ranking Effectiveness (RQ2)

Consistent Gains: Re-ranking consistently improves recall and answer accuracy while reducing the number of search calls.
Depth Matters: Deeper re-ranking (e.g., depth 50) yields better results than shallow depths.
Best Configuration: The pipeline combining BM25 (retriever) + monoT5-3B (re-ranker) with gpt-oss-20b achieves the best overall performance (0.716 recall, 0.689 accuracy), approaching the performance of commercial agents using more complex setups.
Reasoning Limitations: The reasoning-based re-ranker Rank1-7B shows no clear advantage over non-reasoning methods. It often misinterprets keyword-rich, phrase-driven queries (e.g., treating independent keywords as a semantic unit), suggesting that explicit reasoning is less effective when the input query format does not match the training distribution (natural language questions).

3. Query Mismatch and Q2Q (RQ3)

Distribution Shift: There is a significant performance gap for neural rankers when using raw agent-issued queries versus natural-language questions.
Q2Q Efficacy: The proposed Query-to-Question (Q2Q) method, specifically the variant incorporating the agent's reasoning trace ( $Q+R$ ), significantly mitigates this mismatch. It improves recall and accuracy for neural retrievers (SPLADE-v3, Qwen3-Embed-8B) and the reasoning-based re-ranker.
Lexical Sensitivity: Interestingly, applying Q2Q to BM25 slightly degrades performance, confirming that BM25 is better suited to the raw, keyword-heavy syntax of agent queries.

Contributions

The paper claims the following contributions:

First Comprehensive Reproduction: It is the first study to reproduce a comprehensive set of text ranking methods (retrievers and re-rankers) specifically within the deep research paradigm.
Passage Corpus Construction: The authors constructed a passage-level corpus from the BrowseComp-Plus document set to enable granular retrieval analysis.
Empirical Insights: The study reveals that passage-level units, web-search-optimized retrievers (like BM25), and re-ranking are critical components for deep research agents.
Query Mismatch Mitigation: The identification of the training-inference query mismatch and the proposal of the Q2Q method to address it.
Open Source: All code, data, and agent traces are open-sourced to facilitate further research.

Significance and Claims

The authors assert that established text ranking methods from the Information Retrieval (IR) community remain highly effective in deep research, provided they are configured correctly. Specifically:

BM25 is not obsolete: With appropriate parameter tuning and passage-level indexing, BM25 outperforms many large neural models in this setting.
Re-ranking is essential: It provides a high return on investment, improving accuracy and reducing search costs.
Query Format is Critical: The "web-search-style" queries generated by agents create a distribution shift that hinders neural rankers trained on natural language. Mitigating this via methods like Q2Q is crucial for unlocking the potential of neural ranking in agentic search.

The paper concludes that while deep research introduces new complexities, the core principles of IR (lexical matching, re-ranking, and unit granularity) are still the primary drivers of success, rather than solely relying on the scale of the LLM agent.