🤖 AI

Critic-R: Improving Agentic Search using Instruction-tuned Retrievers with Natural Language Introspective Feedback

The paper proposes Critic-R, a framework that enhances agentic search by introducing a critic model to provide natural language introspective feedback, enabling iterative query refinement and automatic retrieval model optimization without the need for manual relevance annotations.

Original authors: Md Zarif Ul Alam, Alireza Salemi, Hamed Zamani

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Zarif Ul Alam, Alireza Salemi, Hamed Zamani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very tricky puzzle, like figuring out the name of a movie director's alma mater. You have a brilliant Detective (the AI reasoning agent) who knows how to think, but they can't see the world outside their own mind. To solve the puzzle, they need to ask a Librarian (the retrieval model) to fetch books (documents) from a massive library.

In the old way of doing things, the Detective would ask the Librarian for a book, get whatever was handed over, and immediately try to solve the puzzle. If the book was the wrong one, the Detective would often just guess anyway, or get stuck. The Librarian was treated like a "black box"—you couldn't really fix them, you just had to hope they got it right.

The paper introduces a new system called Critic-R that changes this dynamic by adding a third character: The Critic.

The Three Characters

The Detective (The Reasoner): The smart AI that tries to answer the question. It thinks, "I need to find the director first," and asks the Librarian.
The Librarian (The Retriever): The system that searches the library and hands over the top few books.
The Critic (The New Addition): A separate, sharp-eyed editor who watches the whole process. The Critic doesn't just look at the books; they look at what the Detective says about the books.

How It Works: The Two-Part Strategy

The paper proposes two main ways to use this Critic to make the system smarter. Think of them as "Fixing it Right Now" and "Training for the Future."

1. Critic-R-Zero: The "Do-Over" Button (Inference-Time)

Imagine the Detective asks the Librarian for a book about "Christopher Nolan." The Librarian hands over a book about the plot of the movie, but it doesn't mention his school.

Without Critic: The Detective might get confused and give a wrong answer.
With Critic-R-Zero: The Critic looks at the Detective's internal thoughts ("I need the director's biography, but this book only has the plot") and says, "Stop! This book isn't helpful. Let's try again."

The Critic then rewrites the request for the Librarian, making it more specific: "Find the biography of Christopher Nolan, not the movie plot." The Librarian tries again, finds the right book, and the Detective can finally solve the puzzle.

This happens instantly while the AI is working. It's like having a coach standing next to the player, shouting, "That pass was bad, try a different angle!" without changing the player's actual training.

2. Critic-Embed: The "Training Camp" (Retriever Fine-Tuning)

The "Do-Over" method is great, but it takes extra time and computer power every single time you ask a question. To make the Librarian better permanently, the authors use the "Do-Over" sessions to create a Training Camp.

Every time the Critic says, "That book was bad, try again," the system saves that moment.
It saves the "bad book" as a Negative Example (Don't pick this!).
It saves the "good book" that finally worked as a Positive Example (Pick this one!).

The authors then take these saved examples and teach the Librarian how to pick the right books the first time, without needing a human to grade every single book. The Librarian learns from its own mistakes and successes, becoming a much better searcher on its own.

The Results: Why It Matters

The authors tested this system on very hard questions that require connecting dots across many different documents (like "Who directed the movie that won the award for the actor who played in this other movie?").

The Detective + The Critic (Zero): Even with a standard, untrained Librarian, adding the Critic's "Do-Over" loop made the answers much more accurate. It fixed mistakes on the fly.
The Trained Librarian (Embed): The Librarian that was trained using the Critic's feedback became so good that it outperformed other advanced systems, even without the "Do-Over" loop.
The Full Team (Critic-R): When they combined the super-trained Librarian with the Critic's "Do-Over" loop, they got the best results of all. It was like having a world-class Librarian who still had a safety net to catch any remaining errors.

The Bottom Line

The paper argues that in AI search, the Librarian (Retriever) is often the weak link, not the Detective (Reasoner). By adding a Critic that checks the work and provides feedback, you can fix mistakes instantly and train the Librarian to be better for next time.

Important Limitations Mentioned:
The paper notes that this system relies on the Detective being smart enough to realize when it has the wrong information. If the Detective is too simple or confused, the Critic won't get good signals to work with. Also, the tests were done on a static library (Wikipedia); the authors haven't tested this on a chaotic, real-time internet search yet.

Technical Summary: Critic-R

Problem Statement

Agentic search systems, which interleave internal reasoning with iterative retrieval actions, have emerged as a standard for solving complex, multi-hop queries. However, a critical bottleneck persists: while reasoning agents are often optimized via reinforcement learning, the retrieval models they rely on are frequently treated as frozen black boxes. Existing approaches that attempt to jointly optimize retrievers and agents often require heavy co-training, gold-standard passage annotations, or the ability to retrain the reasoning model itself. These requirements limit real-world applicability, particularly when the reasoning model is fixed, the retriever is externally provided, or high-quality relevance labels are unavailable. The authors argue that sub-optimal retrieval cannot be fully compensated for by query reformulation alone, necessitating a framework that explicitly closes the feedback loop between the agent and the retriever without relying on manual supervision.

Methodology

The authors propose Critic-R, a framework that introduces a dedicated critic model to evaluate the agent's introspective reasoning trace after it consumes retrieved evidence. The framework operates through two complementary mechanisms:

1. Critic-R-Zero (Inference-Time Refinement)

This component operates entirely at inference time without requiring gradient updates or retraining of the underlying models.

Mechanism: When an agent performs a search, the retrieved documents are speculatively fed to the reasoner to generate an introspective reasoning trace. A separate critic model ( $M_C$ ) analyzes this trace to determine if the retrieved context sufficiently resolves the agent's information need.
Feedback Loop: If the critic deems the evidence insufficient (negative feedback), it leverages the agent's explicit dissatisfaction in its reasoning trace to rewrite the search query and retrieval instruction. This refined query triggers a new retrieval attempt.
Termination: This loop repeats up to a predefined budget ( $K$ ) or until the critic confirms the evidence is satisfactory. The final selected documents are committed to the reasoning history.
Data Collection: This process naturally generates "positive" documents (those satisfying the agent) and "hard negative" documents (those rejected during unsuccessful refinement attempts), which are collected for training purposes.

2. Critic-Embed (Retriever Fine-Tuning)

To amortize the computational cost of iterative refinement and permanently improve retrieval quality, the authors utilize the trajectories generated by Critic-R-Zero as a source of automatic supervision.

Training Signal: Documents verified as useful by the critic are treated as positive examples ( $D^+$ ), while documents rejected during refinement are treated as hard intra-trajectory negatives ( $D^-$ ).
Contrastive Learning: The retriever is fine-tuned using an InfoNCE objective with these automatically constructed contrastive pairs. Crucially, this process does not require manual relevance annotations (gold passages).
Constraint: To ensure label quality, only trajectories where the final prediction is correct according to the downstream task metric are retained for training.

3. Unified System

The full Critic-R system combines both mechanisms: the Critic-Embed retriever provides a domain-aligned, high-quality initial retrieval, while the Critic-R-Zero loop provides an inference-time safety net to recover from complex edge-case failures.

Key Contributions

Feedback Loop Closure: The paper introduces a framework that explicitly closes the feedback loop between reasoning agents and retrievers during both inference and training, challenging the assumption that reasoning agents can solely compensate for retrieval failures.
Inference-Time Scaling for Retrieval: Critic-R-Zero demonstrates that allocating additional compute at inference time to iteratively refine queries and instructions can significantly mitigate retrieval failures without modifying the reasoning agent or the retriever.
Automatic Supervision for Retrievers: Critic-Embed presents a novel method for fine-tuning dense retrievers using intra-trajectory contrastive learning derived from agent-critic interactions, eliminating the need for expensive gold-passage annotations.
Modular Design: The framework is designed to be orthogonal to the reasoning agent's training objective, allowing it to be applied to arbitrary agents (e.g., Search-R1) and retrievers.

Experimental Results

The framework was evaluated on four challenging multi-hop QA benchmarks: HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle.

Inference-Time Refinement (RQ1): Using a frozen retriever, Critic-R-Zero significantly improved answer accuracy across agents of varying scales (3B to 14B). The method yielded an overall relative improvement of 12.4% in Exact Match (EM) and F1 scores. The authors note that scaling the critic model size (14B to 72B) does not strictly guarantee better performance, with a 32B critic often proving optimal for harder datasets.
Retriever Fine-Tuning (RQ2): The Critic-Embed retriever, trained solely on Critic-R-Zero trajectories, outperformed both off-the-shelf dense retrievers (Stella-400M) and retrievers co-trained end-to-end with search agents (Agentic-R). It achieved up to a 7.5% relative improvement in overall performance, demonstrating that the collected trajectories contain transferable supervision signals.
Combined System (RQ3): Integrating the trained Critic-Embed retriever with the Critic-R-Zero inference loop yielded the strongest results, achieving a 10.9% relative improvement overall. The authors observe that the two components are complementary rather than redundant, addressing different slices of retrieval failures.
Role of Introspective Feedback (RQ4): An ablation study removing the agent's introspective feedback ( $T_{i+1}$ ) from the critic's input resulted in a significant performance drop in the trained retriever. This confirms that the agent's internal reasoning trace is a primary source of the supervisory signal, not merely a marginal input.

Significance and Claims

The paper claims that explicitly modeling and optimizing retrieval quality from within the agentic loop is a powerful path toward more robust agentic search. The significance of Critic-R lies in its ability to:

Improve retrieval quality and downstream answer accuracy without requiring manual relevance annotations.
Function effectively in settings where the reasoning model cannot be retrained or the retriever is a fixed external component.
Provide a scalable, modular approach where inference-time refinement and offline retriever fine-tuning reinforce each other.

The authors conclude that while the framework shows substantial promise, its success relies on the reasoning agent's capacity to provide accurate introspective feedback, and its efficacy in highly dynamic environments (e.g., real-time web search) remains to be evaluated.