RVR: Retrieve-Verify-Retrieve for Comprehensive Question Answering

1. Problem Statement

The paper addresses the challenge of comprehensive retrieval in open-domain Question Answering (QA). Unlike standard retrieval tasks that seek a single "gold" answer, many real-world queries admit a wide range of valid answers (e.g., "Who are the directors of movies produced by Eric Newman?").

• The Gap: Traditional retrievers and even recent "agentic" search systems often fail to retrieve all valid documents for a single query. They tend to stop after finding the first few relevant documents or focus on multi-hop reasoning (chaining queries) rather than exhaustive coverage of a single query's answer space.
• The Goal: To maximize answer coverage (Recall) by retrieving a diverse set of documents containing all possible valid answers, rather than just the most relevant single document.

2. Methodology: Retrieve-Verify-Retrieve (RVR)

The authors propose RVR, a multi-round retrieval framework designed to iteratively uncover missing answers. The process consists of three main components:

A. Framework Workflow

1. Initial Retrieval ($f_i$): An initial retriever takes the original query $q$ and returns a candidate set of documents.
2. Verification ($g$): A verifier model (an LLM) examines the retrieved documents to identify a high-quality subset ($D_v$) that actually contains valid answers. This filters out noise and irrelevant documents.
3. Subsequent Retrieval ($f_r$): The query is augmented by concatenating the original query with the text of the verified documents from the previous round. This new context-aware query ($q_r$) is fed to a subsequent retriever.
   • Key Innovation: The subsequent retriever is trained to find documents that are complementary to the already verified ones, specifically targeting answers not yet covered.
4. Iteration & Aggregation: This process repeats for $T$ rounds. The final output is the union of all verified documents and the final retrieval set, deduplicated and ranked.

B. Training Strategy

• Initial Retriever ($f_i$): Trained using standard contrastive learning on the query and gold documents.
• Subsequent Retriever ($f_r$): Trained on a synthetic dataset generated by the RVR inference procedure.
  • Input: Query + Context (a subset of previously verified gold documents).
  • Target: Gold documents that were not in the context (i.e., missing answers).
  • Objective: The model learns to reason about what information is missing given the current context, rather than just matching the query.
• Verifier: Uses off-the-shelf LLMs (e.g., Qwen3) prompted to verify if a document contains a direct answer to the query. No fine-tuning is performed on the verifier.

3. Key Contributions

1. Novel Framework: Introduction of RVR, a framework that explicitly conditions subsequent retrieval rounds on verified evidence from previous rounds to maximize answer coverage.
2. Retriever Adaptation: Demonstrates that fine-tuning retrievers specifically for this "iterative inference scenario" (where the input includes previous context) yields significant gains over standard fine-tuning or off-the-shelf models.
3. Superiority over Agentic Search: Shows that while agentic search (LLMs generating new queries) is effective for multi-hop reasoning, it is less effective for comprehensive single-query coverage. RVR outperforms agentic baselines by focusing on the retrieval model's ability to find missing documents rather than relying on the LLM to formulate new search terms.
4. Verification Integration: Integrates verification within the retrieval loop (to guide the next retrieval) rather than just as a post-hoc filter.

4. Experimental Results

The method was evaluated on QAMPARI (a multi-answer retrieval benchmark) and generalized to QUEST and WebQuestionsSP.

• Performance on QAMPARI:
  • RVR achieved a 10% relative and 3% absolute gain in complete recall percentage (MRecall@100) compared to strong baselines, including fine-tuned retrievers and agentic search systems (Tongyi DeepResearch, SearchR1).
  • Using fine-tuned subsequent retrievers ($f_r$) provided the best results, outperforming even the "Oracle" (perfect verifier) in some configurations when combined with strong base models.
• Generalization:
  • RVR showed consistent improvements on out-of-domain datasets (QUEST and WebQuestionsSP), even when using off-the-shelf retrievers for the initial round.
• Efficiency:
  • RVR is significantly more efficient than agentic baselines. Agentic systems required ~15 search calls and hundreds of seconds per query, whereas RVR used only 2 retrieval calls and ~5–8 seconds (depending on the verifier budget).
• Ablation Studies:
  • Verifier Impact: Using an "Oracle" verifier (perfect knowledge of gold answers) significantly boosted performance, indicating that better verification models could further improve RVR.
  • Multi-turn: Performance gains plateaued after the second turn with LLM verifiers (due to redundancy), but continued to improve with Oracle verifiers, suggesting room for improvement in verification mechanisms.

5. Significance and Impact

• Paradigm Shift: The paper shifts the focus from "generating better queries" (agentic approach) to "adapting the retriever to the inference scenario." It proves that training retrievers to understand context and find missing information is a more robust path for comprehensive retrieval.
• Practical Application: RVR offers a cost-effective solution for applications requiring exhaustive information gathering (e.g., legal research, medical diagnosis, competitive intelligence) where missing a single valid answer is critical.
• Open Source: The authors released code and models, facilitating further research into iterative retrieval and coverage-optimized search.

In summary, RVR demonstrates that a simple, iterative loop of Retrieve $\to$ Verify $\to$ Retrieve, powered by retrievers trained to find complementary information, significantly outperforms complex agentic search methods for tasks requiring comprehensive answer coverage.