💬 NLP

SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach

This paper proposes a small language model trained via progressive supervised fine-tuning and reinforcement learning to dynamically route queries to specialized retrieval agents based on retrieval performance rather than just intent, achieving significantly higher relevance (NDCG@10 of 0.771) and lower latency (120.1ms) compared to large language model baselines.

Original authors: Gayathri V Kondapalli, Alexander Ng, Hirsh Pithadia, Rahul Monish, Harvey Yorke, Amir Kayhani

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gayathri V Kondapalli, Alexander Ng, Hirsh Pithadia, Rahul Monish, Harvey Yorke, Amir Kayhani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a massive, futuristic library where every single book is written by a different expert. There's a section for quantum physics, another for ancient history, a whole wing for cooking, and even a room dedicated to the secrets of the human genome. In this library, the librarians are super-smart robots. Some are geniuses at finding medical papers, while others are masters of financial reports. The problem? You don't know which robot to ask. If you ask the finance robot about a recipe for sourdough bread, it might try to find a connection to stock markets and give you a confusing, useless answer. If you ask the medical robot about a stock market crash, it might start diagnosing you with a rare disease.

This is the world of "Multi-Agent Retrieval." In the digital world, we have many specialized AI "agents" (think of them as those expert robots) designed to search for specific types of information. The big challenge is the "Router": the part of the system that decides which robot gets your question. For a long time, the best way to pick a robot was to guess what the question was about based on the words used. If you said "stocks," the system picked the finance robot. But this is like judging a book by its cover; sometimes a question about "stocks" is actually about the history of a company, not its current price, and the finance robot might miss the mark. The authors of this paper wanted to build a smarter Router that doesn't just guess the topic, but actually learns from the results to see which robot is truly the best fit.

The paper, titled "SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach," proposes a clever solution to this problem. The authors argue that simply guessing the topic isn't enough because it ignores whether the chosen robot actually finds good answers. Instead, they trained a tiny, super-fast AI model (called a Small Language Model, or SLM) to act as the ultimate traffic cop. They didn't just teach it to read questions; they taught it to learn from experience.

Here is how they did it, using a two-step training process that is like teaching a new employee. First, they used "Supervised Fine-Tuning" (SFT). Imagine showing the new employee thousands of examples of questions and the "correct" robot to pick. This gave the model a solid foundation, teaching it that "finance" usually means the finance robot and "genetics" usually means the science robot. However, the authors realized this wasn't enough. Just like a new employee who only memorizes a handbook, the model couldn't tell the difference between a question that looked like it belonged to one robot but was actually better suited for another.

To fix this, they added a second step: "Reinforcement Learning" (RL). This is where the model gets to play a game. It picks a robot, gets the results, and then a "judge" (another AI) scores how good those results were. If the model picked the wrong robot and got bad results, it got a "punishment" (a low score). If it picked the right robot or realized a specialized robot was a bad fit and switched to a general one, it got a "reward." Over time, the model learned to spot the tricky cases where a specialized robot would fail, even if the question seemed to match perfectly.

The results were impressive. The tiny model they trained, which has only 0.6 billion parameters (making it much smaller than the giant models it was compared against), became a better decision-maker than two very large, expensive AI models (Amazon Nova Lite and Claude Haiku 4.5). When tested on a standard scale of 0 to 1 (where 1 is perfect), the new router achieved an average score of 0.771, while the large models scored 0.594 and 0.552.

The real magic happened when they tested the router on the "tricky" questions—the ones where a specialized robot would get the topic right but the answer wrong. On these specific mismatched queries, the new router scored a massive 0.918, while the large models struggled with scores of 0.539 and 0.490. This proves that the new model learned to detect when a specialized agent was a poor fit, a skill the larger models missed because they only looked at the surface-level topic.

Speed was another huge win. In the real world, waiting for a decision is frustrating. The new router made its choices in an average of 120.1 milliseconds. That is 82.4% faster than the Amazon Nova Lite model, which took 683.6 milliseconds, and a staggering 95.1% faster than the Claude Haiku model, which took 2,457 milliseconds.

In short, the paper shows that you don't need a giant, slow, expensive brain to make smart routing decisions. By training a small, fast brain to learn from the quality of the results it helps generate, you can build a system that is not only smarter at picking the right expert but also lightning-fast. The authors suggest that this approach could revolutionize how we build AI systems that need to search for information quickly and accurately, proving that sometimes, a small model trained with the right feedback is better than a giant one guessing in the dark.

Technical Summary: SLMs as Multi-Agent Routers

Problem Definition
In multi-agent retrieval systems, selecting the optimal specialized agent for a given query is a critical determinant of retrieval quality. While specialized agents (e.g., for genomics, finance, or patents) typically yield higher-quality results than general-purpose search, selecting the correct agent remains an open challenge. Current approaches rely on intent-based routing, where a model infers the query's topic or intent to select an agent. However, this approach is fundamentally limited: it relies solely on intrinsic query features and never observes the actual retrieval outcomes. Consequently, intent-based systems cannot detect "agent-query mismatches"—situations where a query superficially aligns with a specialized agent's domain but falls outside its specific expertise, leading to low-relevance results. Furthermore, using Large Language Models (LLMs) for this selection layer introduces significant inference costs and latency, creating a bottleneck in production agentic systems.

Methodology
The authors propose a progressive training approach using a Small Language Model (SLM) (specifically a 0.6B parameter Qwen3 model) to perform multi-agent routing. The method consists of two distinct stages:

Supervised Fine-Tuning (SFT): The model is first trained on a dataset of 56,000 queries (derived from production logs and synthetic generation) to learn intrinsic query features. The output is a structured tuple comprising:
- A bitmask selecting one or more agents from a pool of 11 domains.
- Extracted salient keywords.
- Inferred temporal intervals (start/end dates).
  This stage establishes a strong foundation for query understanding and intent classification but remains limited by the lack of retrieval outcome feedback.
Reinforcement Learning (RL): The SFT checkpoint is further refined using RL (specifically REINFORCE++) to ground selection decisions in downstream retrieval quality.
- Process: For each query, the model generates a routing decision and parameters. The selected agent executes retrieval, and the results are evaluated.
- Reward Function: A hierarchical reward function ( $R$ $R$ ) guides the policy, balancing two signals:
  - Routing Quality ( $R_{src}$ ): Measures alignment with query intent, keyword precision, and date accuracy.
  - Content Quality ( $R_{con}$ ): Measures the relevance and trustworthiness of the retrieved results using an LLM-as-judge.
- Mechanism: A KL-divergence penalty constrains the policy to remain close to the SFT checkpoint, preventing catastrophic forgetting while allowing the model to learn when to redirect queries away from topically aligned but ineffective agents.

Key Contributions

Retrieval Quality via RL: The authors demonstrate that grounding agent selection in retrieval quality signals via RL produces superior outcomes compared to intent-only baselines.
Detection of Mismatches: The trained router successfully identifies agent-query mismatches invisible to intent-based routing. It learns to redirect queries to more suitable agents (e.g., general or alternative specialized agents) when a topically aligned agent fails to retrieve relevant content.
Efficiency: The approach achieves high selection quality with significantly lower latency and cost than larger LLMs, making it viable for real-time, large-scale agentic systems.

Experimental Results
The system was evaluated against two lightweight LLM baselines: Amazon Nova Lite and Claude Haiku 4.5.

Overall Retrieval Quality: The SLM (SFT+RL) achieved a mean NDCG@10 of 0.771, outperforming Nova Lite (0.594) and Haiku (0.552).
Routing Robustness (Mismatch Handling): On a targeted subset of queries where specialized agents were mismatched to the content, the SLM achieved an NDCG@10 of 0.918, compared to 0.539 for Nova Lite and 0.490 for Haiku. This highlights the model's ability to detect and correct routing errors that intent-based systems miss.
Latency and Cost: The SLM achieved a mean selection latency of 120.1ms, representing an 82.4% reduction over Nova Lite (683.6ms) and a 95.1% reduction over Haiku (2,457ms). The P99 latency remained under 200ms (179.4ms).
Variance: The SLM exhibited lower variance in performance ( $\pm0.372$ ) compared to the baselines, indicating more consistent routing behavior across query types.

Significance and Claims
The paper argues that agent selection quality is determined by the training signal, not model size. While larger LLMs possess more parametric knowledge, they cannot learn task-dependent agent suitability without feedback from retrieval outcomes. By combining SFT (for query understanding) with RL (for outcome-based refinement), a small model can outperform much larger models in both accuracy and efficiency.

The authors conclude that this progressive approach enables sub-150ms end-to-end routing at scale, eliminating the traditional trade-off between selection quality and operational latency. They note that while the current work focuses on a fixed pool of 11 agents, future work could extend this to dynamic agent pools by conditioning the router on explicit agent capability representations.

Technical Summary: SLMs as Multi-Agent Routers

More like this