RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigm

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems struggle when operating across heterogeneous knowledge sources (e.g., private databases, global corpora, specialized APIs).

• The Core Challenge: Existing systems typically rely on an "LLM-as-a-Router" strategy. In this paradigm, an LLM predicts the single most appropriate data source for a sub-query before retrieval occurs.
• The Failure Mode: This predictive approach relies heavily on semantic boundaries between sources. In real-world scenarios, sources often have topical overlap or are artificially partitioned (e.g., Wikipedia shards). If the router makes an incorrect prediction (a "routing hallucination"), critical evidence is excluded from the context window, leading to catastrophic failure in multi-hop reasoning tasks.
• Limitations of Current Solutions: While reflection-based retries can mitigate errors, they introduce latency and remain constrained by the initial single-source commitment.

2. Methodology: The RealRoute System

RealRoute introduces a paradigm shift from predictive routing to a Retrieve-then-Verify mechanism. Instead of committing to a source before searching, it prioritizes evidence discovery followed by dynamic verification.

The system operates through three core phases (illustrated in Figure 1 of the paper):

Phase I: Context-Aware Parallel Multi-Source Retrieval

• Decomposition: The input query is decomposed into a sequence of dependent sub-queries.
• Variable Binding: Answers from previous sub-queries are injected into subsequent queries to maintain context.
• Parallel Retrieval: Unlike hard-routing systems, RealRoute simultaneously retrieves top-$k$ candidates from all available candidate sources (e.g., local corpus, global corpus, BioASQ, SciQ).
• Outcome: This creates a unified "candidate pool," eliminating the risk of missing evidence due to a premature routing decision.

Phase II: Dynamic Evidence Selection via Adaptive Cap

Once the candidate pool is constructed, the system filters evidence to fit within a strict context budget using an Adaptive Cap mechanism:

1. Preferred Source Prediction: A router still predicts a "preferred source" for the query.
2. Quota Allocation:
   • Candidates from the preferred source are allowed a higher token/cap limit.
   • Candidates from non-preferred sources are restricted to a lower limit.
3. Global Ranking & Selection: The surviving candidates are globally ranked to fit the final context budget. The system supports three plug-and-play selectors:
   • Score-based: Raw retrieval scores.
   • RRF (Reciprocal Rank Fusion): Fuses rankings to handle score scale discrepancies across heterogeneous sources.
   • LLM-as-a-Judge: Uses an LLM to evaluate fine-grained relevance.

• Benefit: Even if the initial routing preference is suboptimal, the system retains valuable evidence from non-preferred sources, preventing "single-point failure."

Phase III: Adaptive Generation and Reflection

• Synthesis: The generator constructs a prompt using the optimized evidence subset to answer the sub-query.
• Reflection Loop: If the model determines evidence is insufficient, it triggers an automated reflection loop. This updates a "failure history" to discourage repeating ineffective routing preferences in future retries, prompting the Adaptive Cap to explore alternative evidence allocation strategies.
• Fusion: Intermediate answers are fused into a comprehensive final response.

3. Key Contributions

1. Paradigm Shift: Moves from "Predict-then-Retrieve" (prone to hallucinations) to "Retrieve-then-Verify" (robust parallel retrieval).
2. Adaptive Cap Mechanism: A novel budgeting strategy that balances routing preferences with the necessity of cross-source evidence, ensuring high recall without context bloat.
3. Comprehensive Evaluation: The system is the first to simultaneously support multi-hop reasoning, multi-source retrieval, cross-domain generalization, and built-in knowledge verification (see Table 1 in the paper).
4. Open-Source Toolkit: Released with a user-friendly web interface allowing users to visualize real-time "re-routing," inspect verification chains, and upload custom datasets.

4. Experimental Results

The authors evaluated RealRoute against Hard Routing baselines and other RAG paradigms (e.g., DeepSieve, ReAct, Reflexion) on HotPotQA and biomedical benchmarks.

• Performance in Heterogeneous Settings:
  • In 2-source settings (semantically close), RealRoute improved Exact Match (EM) from 49.30 to 53.67 and F1 from 61.70 to 66.93, while reducing token usage by ~31% (due to better evidence filtering).
  • In 3-source and 4-source settings (highly heterogeneous), RealRoute maintained superior accuracy (e.g., EM 53.00 vs. 52.33 in 3-source) by dynamically allocating tokens to cover broader evidence, accepting a slight increase in token cost for significantly higher reliability.
• Paradigm Comparison: Compared to DeepSieve (a strong hard-routing baseline), RealRoute achieved higher EM (53.7 vs. 49.3) and F1 (66.9 vs. 61.7) while using fewer tokens (2711 vs. 3926).
• Efficiency: It outperforms complex agentic reasoning methods (like ReAct or Reflexion) which often suffer from high token costs and trajectory instability.

5. Significance and Impact

• Robustness: RealRoute effectively mitigates the "routing hallucination" problem, making RAG systems viable for complex, real-world deployments where data boundaries are ambiguous.
• Cost-Effectiveness: By decoupling evidence discovery from source prediction, it achieves a better accuracy-cost trade-off than both simple hard-routing and complex multi-agent reasoning frameworks.
• Practical Applicability: The system serves as a dependable default for heterogeneous multi-source RAG applications, offering a transparent, inspectable workflow that bridges the gap between theoretical retrieval optimization and practical deployment.

In summary, RealRoute solves the critical bottleneck of source selection in multi-source RAG by replacing brittle predictive routing with a robust, parallel retrieval and dynamic verification workflow, significantly improving accuracy and reliability in complex reasoning tasks.