💬 NLP

Talk to Right Specialists: Iterative Routing in Multi-agent Systems for Question Answering

The paper proposes RIRS, a training-free orchestration framework for multi-agent question answering that utilizes embedding-based query routing and iterative response aggregation to efficiently handle knowledge-sovereignty constraints and solve both single-hop and complex multi-step queries without broadcasting to all agents.

Original authors: Feijie Wu, Zitao Li, Fei Wei, Yaliang Li, Bolin Ding, Jing Gao

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Feijie Wu, Zitao Li, Fei Wei, Yaliang Li, Bolin Ding, Jing Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you work in a massive, futuristic library where the books aren't all in one building. Instead, the library is split into hundreds of tiny, specialized rooms scattered across different cities.

Room A only has books about cooking.
Room B only has books about 19th-century history.
Room C only has books about car repair.

Crucially, no one is allowed to move the books out of their rooms due to strict privacy rules (this is the "knowledge sovereignty" problem mentioned in the paper).

The Problem: The "Broadcast" Nightmare

Now, imagine a user walks up to the front desk and asks a complex question: "How did the invention of the steam engine (History) change the way we cook (Cooking) in the 1800s?"

In the old way of doing things (called "Broadcast-to-All"), the librarian would shout the question to every single room in the library at once.

The Waste: The car repair room and the gardening room have to stop what they are doing, read the question, realize they don't know the answer, and shout back, "I don't know!"
The Noise: The librarian gets hundreds of "I don't know" messages mixed with a few "I know!" messages. It's like trying to hear a whisper in a stadium full of people shouting.
The Delay: The librarian has to wait for the slowest room to reply before they can give an answer.

The Solution: RIRS (The Smart Concierge)

The paper introduces a new system called RIRS (Routing and Iterative Refining-Solving). Think of RIRS as a super-smart, invisible concierge who doesn't shout to everyone but knows exactly who to call.

Here is how RIRS works, broken down into two simple steps:

Step 1: The "Fingerprint" (Offline Preparation)

Before any questions are asked, the librarian doesn't look at the actual books in the rooms (to keep them private). Instead, each room sends the librarian a tiny, abstract "fingerprint" of what's inside.

Analogy: Imagine the Cooking room sends a note saying, "We have a cluster of knowledge about 'Spices' and another about 'Baking'."
The librarian puts these fingerprints on a map. Now, the librarian knows exactly which room holds which type of knowledge without ever seeing the actual books.

Step 2: The "Smart Hunt" (Online Solving)

When a user asks a question, the RIRS concierge does two things:

Precision Routing: Instead of shouting to everyone, the concierge looks at the question, matches it against the "fingerprints," and only knocks on the doors of the relevant rooms.
- Result: The Car Repair room never gets bothered. The Cooking and History rooms get the question immediately. This saves time and reduces noise.
Iterative Refining (The Detective Work): Sometimes the answer is too complex for one room to solve alone.
- Scenario: The History room says, "I know the steam engine was invented in 1712, but I don't know how it changed cooking."
- The Fix: The concierge doesn't give up. It takes that partial answer, rewrites the question to focus on the missing piece ("Okay, now that we know the date, how did 1712 steam engines change cooking?"), and sends only that new, specific question to the Cooking room.
- Result: The system builds the answer piece by piece, like a detective solving a mystery, rather than guessing the whole thing at once.

Why is this a big deal?

Privacy First: The rooms never have to share their actual books with each other or the central server. They only share the "fingerprints."
Speed: By ignoring the irrelevant rooms, the system is much faster.
Accuracy: Because the librarian isn't drowning in "I don't know" messages, the final answer is much more accurate. It avoids the "Lost in the Middle" problem where a model gets confused by too much irrelevant information.

The Bottom Line

RIRS is like upgrading from a chaotic town crier shouting to the whole village, to a highly efficient, private detective who knows exactly which expert to call, asks them the right follow-up questions, and pieces together the perfect answer without ever violating anyone's privacy.

1. Problem Statement

The paper addresses the challenge of deploying Retrieval-Augmented Generation (RAG) systems in enterprise environments where knowledge is distributed across multiple specialized agents (e.g., different business units, products, or teams) due to knowledge sovereignty constraints (privacy, compliance, and data ownership). These constraints prevent centralizing all data into a single repository.

Two primary failure modes arise in this distributed setting:

Routing Uncertainty: Users do not know which specific agent holds the relevant evidence for a query.
Complex Query Resolution: Complex, multi-hop questions often require evidence distributed across multiple agents and multiple reasoning steps.

Existing solutions suffer from significant drawbacks:

Broadcast-to-All: Sending queries to every agent is inefficient, causing high latency, excessive computation, and "lost-in-the-middle" problems where noisy contexts degrade answer quality.
Description-based Routing: Relies on manual text descriptions which often fail to capture the precise knowledge boundaries of agents with overlapping or complex subtopics.
Learning-based Routing: Requires training on task-specific data, which is difficult to construct when the agent's capability is determined by its non-parametric local corpus (which cannot be shared for training).

2. Methodology: RIRS Framework

The authors propose RIRS (Routing and Iterative Refining-Solving), a training-free orchestration framework designed to coordinate multiple RAG agents without centralizing raw data. The system operates in two distinct stages:

A. Offline Stage: Knowledge Summarization

Instead of sharing raw documents, each agent summarizes its local corpus into a compact representation in the embedding space.

Clustering: An agent splits its local corpus into chunks, encodes them into embeddings, and partitions these embeddings into $n$ coherent clusters.
Max-Min Objective: The clustering is optimized to maximize the worst-case within-cluster similarity (max-min objective). This ensures that every cluster is semantically tight, preventing "average" centroids that represent mixed topics.
Centroid Upload: The agent computes the centroid (mean vector) for each cluster and uploads only these centroids (along with an agent ID) to the central server. The raw data remains local, preserving sovereignty.

B. Online Stage: Query Solving

The server uses the uploaded centroids to route queries and iteratively solve problems.

Routing (Centroid Matching): The server embeds the user query and matches it against the uploaded centroids using cosine similarity. It selects the Top-K most relevant centroids and routes the query to the corresponding agents.
Response Evaluation: Selected agents generate responses with retrieved evidence. A server-side Response Evaluator (powered by an LLM) assesses each response based on relevance, evidence support, and logical coherence, labeling them as Fully Addressed, Partially Addressed, or Not Addressed.
Iterative Refining-Solving:
- If a response is Fully Addressed, the server synthesizes a final answer.
- If responses are Partially Addressed, the server aggregates the progress and uses a Question Simplifier to rewrite the query, focusing only on the unresolved aspects.
- The process repeats with the refined query until the question is answered or deemed unanswerable.

3. Key Contributions

Problem Definition: The paper formalizes the multi-RAG-agent setting where knowledge is distributed and cannot be centralized, highlighting the dual challenges of agent selection and evidence composition.
RIRS Framework: A novel, training-free framework that combines centroid-based knowledge summarization (offline) with iterative refining-solving (online). It avoids the "broadcast-to-all" inefficiency and the need for raw data sharing or router training.
Theoretical Guarantee: The authors provide a theoretical bound (Theorem 4.1) proving that centroid-based routing incurs a bounded error relative to fine-grained matching, provided the clusters are semantically coherent ( $s$ -coherent).
Empirical Validation: Extensive experiments demonstrate that RIRS outperforms state-of-the-art baselines in both routing accuracy and final QA performance, particularly for complex multi-hop queries.

4. Experimental Results

The authors evaluated RIRS on several benchmarks: Natural Questions (NQ), HotpotQA, 2WikiMultiHopQA, MuSiQue, and the Multi-Hop RAG (MHR) benchmark.

Performance vs. Baselines:
- RIRS consistently outperformed Chameleon (description-based) and RouterDC (learning-based) across all datasets.
- On HotpotQA and 2WikiMultiHopQA, RIRS improved accuracy by up to 8% over Chameleon and 15% over RouterDC, especially when using weaker backbone models (LLaMA-3.1-8B).
- RIRS approached the performance of GoldRouter (an oracle upper bound that knows the correct agents), significantly narrowing the gap compared to other methods.
Efficiency:
- RIRS reduced latency significantly compared to "All-agent Participation" (broadcast-to-all). For example, on NQ with Qwen-Plus, latency dropped from ~73s (broadcast) to ~23s (RIRS) while maintaining higher accuracy.
- It reduced the number of agents queried per round, minimizing token usage and interference from irrelevant agents.
Multi-hop Capabilities:
- The iterative strategy allowed RIRS to successfully decompose complex queries. Experiments showed that most questions were resolved within 1-2 rounds, and additional rounds did not necessarily improve accuracy (avoiding error accumulation).
Model Agnosticism:
- The framework supports a "Mixed" setting where lightweight models run on agents (for local retrieval/generation) while a stronger model (e.g., Qwen-Plus) runs on the server for routing and evaluation. This setup recovered most of the performance of using strong models everywhere while keeping agent costs low.

5. Significance and Impact

Practical Deployment: RIRS offers a viable path for deploying RAG in regulated industries (finance, healthcare, enterprise) where data cannot be centralized. It respects knowledge sovereignty while enabling collaborative problem-solving.
Scalability: By using centroid summaries, the system scales efficiently. The server maintains a lightweight index, and agents can update their knowledge locally without retraining a global router.
Robustness: The iterative refinement mechanism ensures that complex questions are handled systematically, filtering out hallucinations and irrelevant evidence at each step.
Theoretical Insight: The paper bridges the gap between theoretical clustering bounds and practical routing, providing a mathematical justification for why centroid-based routing works effectively in distributed settings.

In conclusion, RIRS represents a significant step forward in Agentic RAG, moving from centralized or naive distributed approaches to a sophisticated, privacy-preserving, and iterative orchestration framework that effectively mimics human "specialist consultation."