💬 NLP

Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generation

This paper proposes a rule-driven agent framework for Hybrid-Source Retrieval-Augmented Generation that dynamically routes queries between relational databases and unstructured documents using adaptive rules and meta-caching to achieve higher accuracy and efficiency than existing static or learned baselines.

Original authors: Haoyue Bai, Haoyu Wang, Shengyu Chen, Zhengzhang Chen, Lu-An Tang, Wei Cheng, Haifeng Chen, Yanjie Fu

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoyue Bai, Haoyu Wang, Shengyu Chen, Zhengzhang Chen, Lu-An Tang, Wei Cheng, Haifeng Chen, Yanjie Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a super-smart but slightly forgetful librarian (the Large Language Model, or LLM). You know a lot of general facts because you've read millions of books, but you don't know the specific, up-to-the-minute details of your town's library, bank, or hospital records.

When someone asks you a question, you have two ways to find the answer:

The Document Shelf: You pull out general books and articles (unstructured documents). Great for stories, opinions, and broad explanations.
The Database Vault: You go to the secure, organized filing cabinets (relational databases). Great for exact numbers, dates, and strict facts like "What was the profit last quarter?"

The Problem: The "Kitchen Sink" Approach

Most current systems try to be safe by throwing everything at the librarian. They say, "Here are 5 books and the entire bank vault, please find the answer."

The paper argues this is a terrible idea for three reasons:

Confusion: If you ask, "What's the temperature?" and you hand the librarian a 500-page history book and a weather log, the history book might distract them.
Cost & Speed: Reading 500 pages takes time and money (computing power).
Mismatch: Some questions only need the bank vault (e.g., "How much is in account #123?"). Others only need the books (e.g., "Why is the sky blue?"). Mixing them often makes the answer worse.

The Solution: The "Smart Router"

The authors propose a new system called Learning to Route. Think of this as hiring a sharp, experienced Traffic Cop who stands at the entrance of the library.

Instead of letting every question go to the "Kitchen Sink" (everything), the Traffic Cop looks at the question and decides: "Do you need the books, the vault, or just the librarian's own brain?"

Here is how their system works, using a simple analogy:

1. The Traffic Cop (Rule-Driven Routing Agent)

This isn't a mysterious black box. It's a cop following a clear Rulebook.

Rule: "If the question asks for a number or a date, send it to the Database Vault."
Rule: "If the question asks 'Why' or 'How' about a concept, send it to the Document Shelf."
Rule: "If it's a simple definition, just let the Librarian answer from memory."

This is fast, transparent, and cheap. You can see exactly why the cop made a decision.

2. The Rulebook Editor (Rule-Making Expert Agent)

What if the Rulebook isn't perfect? Maybe the cop keeps sending "How much money?" questions to the wrong place.
The system has a Rulebook Editor who watches the results. If the cop makes a mistake, the Editor says, "Hey, I noticed that when people ask about 'revenue,' we should send them to the Vault, not the Books. Let's update the rule."
This allows the system to learn and get smarter over time without needing a massive amount of training data.

3. The "Cheat Sheet" (Path-Level Meta-Cache)

Imagine the Traffic Cop has to stop and think for every single question. That's slow.
But, people often ask the same questions or very similar ones (e.g., "What was the revenue in Q1?" vs. "What was the revenue in Q2?").
The system keeps a Cheat Sheet (Cache). If someone asks a question that sounds like one they asked yesterday, the cop just looks at the sheet and says, "Oh, we already decided this one goes to the Vault!"
Crucially: This cheat sheet remembers where to send the question, not the answer itself. This is important because if the bank vault updates its numbers today, the old answer is wrong, but the decision to check the vault is still correct.

Why This Matters

The paper tested this on three different types of questions (financial, general knowledge, etc.) and found:

Accuracy: It got more answers right because it didn't get confused by irrelevant information.
Speed & Cost: It used fewer "tokens" (words processed) because it didn't waste time reading the wrong books.
Reliability: It was more consistent than systems that just guess or try to use everything at once.

The Bottom Line

Instead of blindly throwing every question at a giant pile of information, this framework teaches the AI to think like a human expert: "Do I need a quick fact from a spreadsheet, or a detailed explanation from a book?" By using simple, clear rules to make that choice, the system becomes faster, cheaper, and much more accurate.

1. Problem Statement

Large Language Models (LLMs) struggle with domain-specific questions requiring accurate, up-to-date information. While Retrieval-Augmented Generation (RAG) addresses this by retrieving external knowledge, existing systems face two main limitations:

Source Bias: Most RAG systems rely solely on unstructured documents (e.g., Wikipedia), which can be imprecise or outdated. Conversely, relational databases offer precise, structured, and timely facts but are often ignored in RAG frameworks.
Inefficiency of Naive Hybridization: Simply combining both document and database sources for every query (Hybrid RAG) often introduces noise, increases token consumption (cost/latency), and can degrade accuracy due to context distraction.

Core Challenge: How to dynamically and intelligently route each query to the most suitable source (Database, Document, Hybrid, or Direct LLM) to balance accuracy, efficiency, and interpretability without relying on complex, black-box learned models.

2. Key Observations

The authors conducted preliminary experiments revealing three critical insights:

Complementarity: Databases and documents have distinct strengths. Fact-centric/numerical queries are best answered by databases, while open-ended/descriptive queries align better with documents. Neither source dominates all queries.
Noise in Hybridization: Blindly concatenating both sources often leads to incorrect answers compared to using the single optimal source, while significantly increasing inference costs.
Pattern Regularity: Query types exhibit consistent patterns (e.g., "How much" implies a number/database; "Why" implies explanation/document). These patterns can be captured by explicit rules rather than requiring heavy training.

3. Methodology: The Rule-Driven Agent Framework

The proposed framework consists of three core components designed to create an adaptive, interpretable, and efficient routing system.

A. Rule-Driven Routing Agent

Instead of a learned classifier, this agent uses explicit, interpretable rules to score candidate augmentation paths ( $P = \{Doc, DB, Hybrid, LLM\}$ ).

Mechanism: The agent evaluates a query against a set of rules (e.g., "If the query asks for a number, add +3 to the DB score").
Scoring: It calculates a score $S_p(q)$ for each path and selects the one with the maximum score: $p_q = \arg\max_{p} S_p(q)$ .
Advantage: This approach is training-free, transparent, and allows for easy domain adaptation by modifying rules.

B. Rule-Making Expert Agent

To prevent the rules from becoming static or suboptimal for specific datasets, a "Rule-Making Expert" refines the rule set iteratively.

Process: After processing a batch of queries, the system generates a diagnostic report (including path selection frequencies, rule triggers, and accuracy).
Refinement: The Expert Agent analyzes this feedback and updates the rule set (e.g., strengthening a rule that led to high accuracy or weakening one that caused errors). This acts as a "textual gradient," allowing the system to adapt to data distributions without retraining the routing model itself.

C. Path-Level Meta-Cache

To reduce the latency of calling the routing agent for every query, a caching mechanism is introduced.

Strategy: Unlike traditional answer caching (which fails in dynamic databases), this caches routing decisions.
Implementation: It stores the embedding of a query and the resulting path scores. For new queries, if the embedding similarity exceeds a threshold, the cached routing decision is reused. This bypasses the agent call for semantically similar queries, significantly reducing latency.

4. Key Contributions

Systematic Analysis: Provided empirical evidence that relational databases and documents are complementary and that naive hybridization is often detrimental, establishing the need for intelligent routing.
Rule-Driven Framework: Proposed a novel architecture integrating a Routing Agent, a Rule-Making Expert, and a Meta-Cache. This framework prioritizes interpretability and efficiency over black-box learning.
Performance & Efficiency: Demonstrated that rule-based routing can outperform learned routing baselines while maintaining lower computational costs and latency.

5. Experimental Results

The framework was evaluated on three datasets: TATQA (financial), FinQA (financial), and WikiQA (general knowledge), using various LLM backbones (LLaMA-3, Qwen2.5, GPT-4o, GPT-4.1).

Accuracy: The proposed method ("Ours") consistently outperformed static strategies (Basic, Doc-only, DB-only, Hybrid) and learned routing baselines (Adaptive-RAG, Agent-Based, Rule Agent).
- Example: On WikiQA with Qwen2.5, it achieved a state-of-the-art accuracy of 0.302, surpassing the best baseline (0.260).
Efficiency (Token Usage): The method achieved higher accuracy while using significantly fewer tokens than the Hybrid strategy. It avoided the "token bloat" of feeding all sources to the LLM.
Latency: By utilizing the Path-Level Meta-Cache, the routing time was drastically reduced (often negligible) compared to agent-based routing methods that took 2+ seconds per query.
Path Selection: Analysis showed the framework's path selection distribution closely matched an "Oracle" (ideal) distribution, effectively leveraging the strengths of DB, Doc, and Hybrid paths where appropriate, unlike baselines that skewed heavily toward one source.

6. Significance

This work shifts the paradigm in Hybrid-Source RAG from "more data is better" to "the right data is better."

Interpretability: By using rules, the decision-making process is auditable, which is crucial for high-stakes domains like finance and healthcare.
Cost-Effectiveness: It offers a practical solution for enterprise RAG systems where latency and token costs are critical constraints.
Adaptability: The feedback loop allows the system to evolve its routing logic without expensive retraining, making it robust to changing data distributions.

In conclusion, the paper demonstrates that a lightweight, rule-driven approach, enhanced by expert feedback and caching, can achieve superior performance and efficiency compared to complex learned routing mechanisms in hybrid retrieval environments.