💬 NLP

When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models

The paper introduces ReaLM-Retrieve, a framework that optimizes retrieval timing for large reasoning models by detecting knowledge gaps at the reasoning-step level, which significantly improves answer accuracy and retrieval efficiency while reducing unnecessary retrieval calls compared to standard and fixed-interval approaches.

Original authors: Dongxin Guo, Jikun Wu, Siu Ming Yiu

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongxin Guo, Jikun Wu, Siu Ming Yiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Library Visit" Mismatch

Imagine you are a brilliant detective (the Large Reasoning Model) trying to solve a complex mystery. You have a notebook where you write down your thoughts step-by-step. Sometimes, you get stuck because you don't know a specific fact, like the name of a witness or the date of an event.

The Old Way (Standard RAG):
Currently, most systems act like a librarian who hands you a stack of books before you even start writing in your notebook. You read the books, then try to solve the whole mystery.

The Flaw: If you get stuck halfway through your 20-page thought process because you forgot a detail, you can't go back and ask for a new book. You are stuck with the information you were given at the start, even if it wasn't enough for the later steps.

The New Way (ReaLM-Retrieve):
This paper introduces a system called ReaLM-Retrieve. Instead of getting all the books at once, the detective is allowed to walk to the library in the middle of writing their notes, but only when they genuinely need it.

How It Works: The Three Magic Tools

The researchers built a system with three main parts to make this "mid-thought library trip" work perfectly.

1. The "Confidence Check" (Step-Level Uncertainty)

Imagine the detective pauses after every paragraph of their notebook. They ask themselves: "Do I actually know this, or am I just guessing?"

Old methods checked this every single word (too frequent) or every sentence (too rigid).
ReaLM-Retrieve checks at the logical step. It asks: "Did I just finish a complete thought? Do I feel unsure about the next logical jump?"
If the detective feels a "knowledge gap" (uncertainty), the system flags it. It's like a "Check Engine" light that only turns on when the car actually needs fuel, not just because the engine is running.

2. The "Smart Manager" (Learned Intervention Policy)

Just because the detective feels unsure doesn't mean they should run to the library every time. Sometimes they are just thinking deeply, not missing facts.

This system acts like a smart manager sitting next to the detective.
The manager looks at the "Confidence Check" and the history of the case. It decides: "Okay, we really need to look up that specific fact right now," OR "No, keep thinking, you'll figure it out."
It also helps write the perfect question for the librarian (the search query) so the detective gets exactly the right page, not a whole book of irrelevant info.

3. The "Speedy Delivery" (Efficient Integration)

Going to the library takes time. If the detective stops for 5 minutes every time they need a fact, the whole process becomes slow.

The researchers built a high-speed delivery service.
Instead of handing the detective a whole book, they only give them the exact paragraph they need (compressing the info).
They also use a "speculative" trick: If the detective is likely to need a fact about "Paris" next, the system grabs that info while the detective is still writing the current sentence. If they need it, it's already there instantly.
Result: The system is 3.2 times faster at getting information than older methods.

The Results: Smarter, Faster, Cheaper

The team tested this on three difficult puzzle games (MuSiQue, HotpotQA, and 2WikiMultiHopQA) that require connecting many dots to find an answer.

Better Answers: The system got the right answer 10% more often than the standard "get all books at once" method.
Fewer Trips: It made 47% fewer trips to the library. Instead of checking the library every few sentences (like the old IRCoT method), it only went when absolutely necessary.
The Sweet Spot: On the hardest puzzles (requiring 4 steps of logic), the new method was significantly better. It proved that fewer, well-timed trips to the library are better than frequent, random trips.

A Real-Life Example from the Paper

The paper gives an example of a question: "Who directed the film that won Best Picture in the year the Berlin Wall fell?"

The Detective's Thought:
1. "The Berlin Wall fell in 1989." (Confident, no library trip needed).
2. "The Best Picture of 1989 was Driving Miss Daisy." (Confident, no trip).
3. "Who directed it?" (Here, the detective feels a gap. The system says: GO TO LIBRARY).
4. The system fetches the director's name instantly.
5. The detective finishes the answer.
The Old Way: Would have fetched the director's name at the very beginning, even though the detective didn't need it until step 3, wasting time and potentially confusing the detective with too much info too early.

Summary

ReaLM-Retrieve teaches AI models to stop and ask for help exactly when they get stuck, rather than asking for help before they start or asking for help too often. It's like having a smart assistant who knows exactly when to open the encyclopedia, saving time and getting better results.

1. Problem Statement

The paper addresses a fundamental temporal misalignment between Large Reasoning Models (LRMs) like DeepSeek-R1 and OpenAI o1, and traditional Retrieval-Augmented Generation (RAG) systems.

The Mismatch: Standard RAG follows a "retrieve-then-generate" paradigm, injecting context before reasoning begins. However, LRMs generate extended chains of thought (12,000–25,000 tokens) involving multi-step inference, self-verification, and backtracking. Knowledge gaps often emerge mid-inference and cannot be anticipated at the query stage.
Limitations of Existing Iterative Methods: Current iterative retrieval methods (e.g., IRCoT, FLARE) are ill-suited for LRMs due to:
1. Granularity Mismatch: They trigger retrieval at the token or sentence level, whereas LRMs operate at the reasoning step level (logical units spanning multiple sentences).
2. Signal Unavailability: Methods relying on token probabilities (FLARE) or internal attention states (DRAGIN) fail with "completion-only" models (like o1) that do not expose these internals.
3. Efficiency Collapse: Fixed-interval retrieval (e.g., after every sentence) is prohibitively expensive for long chains, adding significant latency and token overhead.

2. Methodology: ReaLM-Retrieve

The authors propose ReaLM-Retrieve, a reasoning-aware framework that integrates retrieval dynamically during the inference chain. It consists of three core components:

A. Reasoning Step Segmentation

Instead of sentence-level segmentation, the system identifies logical reasoning steps.

Mechanism: A lightweight 3-layer transformer classifier detects step boundaries based on discourse markers ("therefore," "however"), logical connectives, topic shifts (via embedding similarity), and punctuation.
Training: Trained on human-annotated traces from DeepSeek-R1, achieving 94.2% F1 in identifying boundaries. This allows the system to treat the reasoning chain as a sequence of discrete logical units rather than a stream of tokens.

B. Step-Level Uncertainty Detection (RSUS)

The system computes a Reasoning Step Uncertainty Score (RSUS) to determine if a step requires external knowledge.

Composite Score: $RSUS(r_i) = \alpha \cdot U_{verb} + \beta \cdot U_{ent} + \gamma \cdot U_{cons}$ $R S U S (r_{i}) = α \cdot U_{v er b} + β \cdot U_{e n t} + γ \cdot U_{co n s}$
- $U_{verb}$ (Verbalized Uncertainty): The model is prompted to rate its confidence (0–100). For completion-only models, a learned MLP proxy estimates this based on surface features and entity counts.
- $U_{ent}$ (Entity Entropy): Measures the ambiguity/sparse coverage of named entities in the retrieved corpus. High entropy indicates the model lacks knowledge about specific entities.
- $U_{cons}$ (Consistency): Measures agreement across multiple sampled continuations of the current step.
Efficiency: This computation adds only ~8% overhead, avoiding the massive cost of semantic entropy or self-consistency sampling.

C. Learned Intervention Policy & Efficient Integration

Policy ( $\pi$ ): A contextual bandit model decides whether to retrieve, what query to formulate, and how to integrate the evidence. It is trained via REINFORCE to maximize answer accuracy while penalizing retrieval cost (latency and call count).
Query Formulation: The policy generates specific queries targeting the information need (e.g., "What is [entity]'s relationship to [concept]?") rather than just repeating the current step.
Efficiency Optimizations:
- Implicit Compression: Only retains sentences from retrieved documents exceeding an importance threshold, reducing context expansion by 73%.
- Speculative Caching: Pre-fetches documents for likely next-step queries in parallel; if the guess is correct (37% hit rate), retrieval latency is eliminated.
- KV-Cache Preservation: For open-weight models, preserves cached key-value states for unchanged context portions, reducing time-to-first-token by 2.1×.

3. Key Contributions

Step-Level Granularity: Shifts the retrieval trigger from token/sentence level to reasoning-step level, aligning with the logical structure of LRMs.
Black-Box Compatible Uncertainty: Introduces RSUS, a method that works on completion-only models (like o1) without requiring access to token probabilities or internal attention states.
Efficiency-Optimized Integration: Reduces per-retrieval overhead by 3.2× compared to naive integration through speculative caching and context compression, enabling practical deployment.

4. Experimental Results

Evaluated on MuSiQue, HotpotQA, and 2WikiMultiHopQA using DeepSeek-R1 and QwQ models.

Accuracy: ReaLM-Retrieve achieves an average 10.1% absolute improvement in F1 over standard RAG and 4.6% over the strongest prior method (ReARTeR).
- On MuSiQue (2–4 hop reasoning), it achieves 71.2% F1 (vs. 65.4% for IRCoT).
Efficiency:
- Fewer Calls: Reduces retrieval calls by 47% compared to fixed-interval approaches (1.8 calls vs. 3.4 calls for IRCoT on MuSiQue).
- Latency: Achieves 1.33× lower end-to-end latency than IRCoT and 1.50× lower than naive interleaving.
- Cost: Reduces output token costs by 16% due to shorter reasoning chains (evidence resolves uncertainty faster).
Retrieval Quality: Improves Recall@5 to 81.3% (vs. 72.6% for IRCoT), proving that the system retrieves better evidence at the right time.
Generalization: The policy generalizes well to different model architectures (e.g., QwQ) and transfers across benchmarks (trained on HotpotQA, tested on MuSiQue) with minimal performance drop.

5. Significance

Paradigm Shift: The paper demonstrates that "fewer, better-timed retrievals" outperform frequent, fixed-interval retrieval. This establishes a new Pareto frontier for the accuracy-efficiency trade-off in reasoning-intensive tasks.
Scalability: By operating at the reasoning-step level and supporting completion-only models, the framework is applicable to the most capable proprietary LLMs (like o1) which are currently inaccessible to many RAG research efforts.
System Design: It provides a blueprint for integrating retrieval into long-horizon reasoning tasks, moving RAG from a static pre-processing step to a dynamic, adaptive reasoning partner.

In conclusion, ReaLM-Retrieve solves the critical timing mismatch in LRM-RAG integration, delivering state-of-the-art accuracy with significantly reduced computational costs.