🤖 AI

Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems

This survey provides a comprehensive overview of routing strategies in Large Language Model-based systems, analyzing how dynamically directing queries to appropriate models can optimize resource consumption, reduce costs, and enhance performance by addressing the inefficiencies of monolithic architectures.

Original authors: Clovis Varangot-Reille, Christophe Bouvard, Antoine Gourru, Mathieu Ciancone, Marion Schaeffer, François Jacquenet

Published 2026-07-16

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Clovis Varangot-Reille, Christophe Bouvard, Antoine Gourru, Mathieu Ciancone, Marion Schaeffer, François Jacquenet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where every time you asked a question, a team of experts rushed to answer it. Some are brilliant but slow and expensive, like a Nobel Prize-winning professor who charges by the hour. Others are quick and cheap, like a knowledgeable friend who can handle most things but might stumble on a really tricky riddle. In the world of artificial intelligence, these "experts" are Large Language Models (LLMs). Right now, most computer systems are built like a stubborn restaurant that sends every single order, from a simple glass of water to a complex five-course meal, to the most expensive chef in the kitchen. This works, but it's wasteful, slow, and burns a lot of energy. The big question scientists are asking is: How can we build a smarter "maitre d'" that looks at your order and instantly decides whether to send it to the quick, cheap helper or the expensive genius? This is the heart of a new field called "routing," and it's all about getting more done with less.

This paper, titled "Doing More with Less," is a massive tour of the different ways researchers are trying to build that smart maitre d'. The authors, a team from France, didn't just invent one new trick; they gathered and organized the entire landscape of strategies currently being tested. They found that the best way to save money and energy isn't to use one giant brain for everything, but to create a system that routes questions to the right tool based on how hard the question is. They discovered that you can often use a tiny, cheap model for simple greetings and save the big, expensive models only for the really tough logic puzzles. However, they also warn that this isn't a magic wand yet. Some methods are great at saving money but might miss the mark on accuracy, while others are too complicated to run in the real world. The paper suggests that the future lies in "adaptive" systems that can learn and change their minds as new tools become available, rather than being stuck with a rigid set of rules.

The Big Problem: The "One-Size-Fits-All" Chef

Think of a Large Language Model (LLM) like a super-smart chef. Some chefs are generalists; they can cook almost anything, from a sandwich to a soufflé. But they are expensive to hire and take a long time to work. Other chefs are specialists; maybe they are amazing at making pizza but terrible at baking cakes.

Right now, most AI systems are like a restaurant that ignores the menu. If you ask for a glass of water, the system calls the most expensive, powerful chef (like GPT-4 or Claude) to do it. This is a huge waste. It costs a lot of money, uses a ton of electricity, and takes longer than necessary. The paper argues that we need a "router"—a smart waiter—who looks at your request and decides: "Hey, this is just a simple question. Let's send it to the quick, cheap intern." But if you ask a complex math question, the waiter should say, "Okay, this needs the head chef."

The Three Big Questions the Paper Answers

The authors break down the problem into three simple questions: What should we optimize? When should we make the decision? and How do we actually build the router?

1. What are we trying to save?

Most people think the only thing that matters is money. But the paper points out that we also need to care about time (latency) and the environment (energy).

Money: Every time an AI answers a question, it costs a tiny bit of money per word (token).
Time: Waiting for a slow, powerful AI to answer a simple "Hello" is annoying.
The Planet: Big AI models use a lot of electricity. If we can use a smaller model for 80% of our questions, we save a massive amount of energy and reduce our carbon footprint.

2. When do we decide?

The paper identifies two main moments when the "waiter" can make a choice:

Before the answer is made (Pre-generation): The waiter looks at your question before anyone starts cooking. They guess, "This looks like a simple pizza order," and send it to the cheap chef immediately. This is fast and saves money because the big chef never gets involved.
After the answer is made (Post-generation/Cascade): The waiter lets the cheap chef try first. If the answer looks bad or the chef seems unsure, then the waiter sends the question to the expensive chef. This is like ordering a pizza, tasting a slice, and if it's burnt, calling the head chef to make a new one. This is safer but slower and can cost more if the cheap chef fails often.

3. How do we build the waiter?

This is the most fun part. The paper reviews dozens of different "waiter" strategies, which they group into four main families:

The "Look-Alike" Waiter (Similarity-based): This waiter keeps a scrapbook of past questions. If you ask a question that sounds like a question they asked last week, the waiter sends it to the same chef who answered that one. It's like saying, "Last time you asked about cats, the cat-expert was great, so I'll send this cat question to them too."
The "Teacher" Waiter (Supervised): This waiter is trained like a student. It learns to recognize patterns. For example, it learns that questions about "math" usually need a math-specialist, and questions about "jokes" need a creative model. It's like a student who memorized a textbook on "Which chef handles which dish."
The "Gambler" Waiter (Reinforcement Learning): This waiter learns by trial and error, like a video game character. It tries different chefs, sees if the answer was good, and gets a "reward" if it saved money and got a good answer. Over time, it learns the best strategy without needing a textbook.
The "Self-Reflecting" Waiter (Generative): This is where the AI talks to itself. The waiter asks the cheap chef, "Are you sure you can answer this?" If the chef says, "I'm not sure," the waiter immediately sends it to the big chef. It's like asking a friend, "Do you know this?" and if they hesitate, you call the expert.

What the Paper Rules Out (The "Don't Bother" List)

The authors are very clear about what doesn't count as a good routing strategy. They explicitly say that just gathering answers from five different chefs and picking the best one (called "ensemble methods") is not the same as routing. That's like ordering five pizzas and throwing away the bad ones; it's too expensive. The goal is to pick the right chef before you start cooking, not to cook everything and then choose.

They also note that some methods, like asking the AI to guess its own confidence, can be tricky. Sometimes the AI is overconfident and says, "I know this!" when it actually doesn't. This can lead to bad answers if the system trusts the wrong chef.

The Verdict: It's Promising, But Not Perfect

The paper concludes that routing is a powerful tool, but it's not a solved mystery yet.

It works: In many tests, smart routing can cut costs by half or more while keeping the answers just as good as using the big models.
It's tricky: The best strategy depends on the situation. Sometimes a "look-alike" approach works best; other times, a "teacher" approach is better.
The future: The authors suggest that the next big step is making these routers "adaptive." Right now, if you add a new chef to the kitchen, you often have to retrain the whole waiter system. The future goal is a waiter that can instantly understand a new chef's skills and start using them immediately, without needing a long training period.

In short, this paper is a guidebook for building smarter, cheaper, and greener AI systems. It tells us that we don't need to use a sledgehammer to crack a nut. By building a smart system that knows when to use a small hammer and when to use a big one, we can make AI faster, cheaper, and better for the planet.

Technical Summary: Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems

1. Problem Statement

Large Language Model (LLM)-based systems, particularly conversational agents, are frequently designed with monolithic, static architectures that rely on a single, general-purpose LLM to handle all user queries. This approach is often inefficient because different queries require varying levels of reasoning, domain knowledge, or pre-processing. While generalist models (e.g., GPT-4o, Claude-Sonnet) perform well across diverse tasks, they incur significant financial, energy, and computational costs that are disproportionate for simpler queries.

The paper defines a router as a component that directs an item (e.g., a user query) to the most suitable element in a pool of candidates to optimize resource consumption. This differs from Mixture-of-Experts (MoE) frameworks, where an internal gating mechanism selects subnetworks within a single model. In contrast, an external router operates independently of the LLM's latent forward process, functions under budget constraints, and can route to diverse candidates including smaller models, specialized models, workflows, or data sources.

Formally, the routing problem is defined as maximizing a scoring function $s$ (e.g., accuracy) for a query $q$ and a set of models $M$ , subject to a budget constraint $B$ :
$R_M(q) = \arg \max_{M \in M} s(q, M) \quad \text{s.t.} \quad C_M(q) \le B$
where $C_M$ is the cost (e.g., $/token) to call model $M$ .

2. Methodology and Taxonomy

This work provides a comprehensive survey and taxonomy of routing strategies, addressing three core questions:

What should routing optimize? (Performance vs. Cost)
When should routing take place? (Pre-generation vs. Post-generation)
How should routing be implemented? (Algorithmic strategies)

2.1 Optimization Objectives

Performance Metrics: The survey reviews methods to maximize scoring functions, ranging from exact/partial matching and ROUGE scores to semantic similarity and automatic evaluation using LLMs. It also notes the use of preference learning (e.g., Bradley-Terry models) based on human reviewer data.
Cost Metrics: Beyond financial costs (token price), the paper emphasizes minimizing latency and environmental costs (energy consumption, carbon footprint), noting that larger models generally require more floating-point operations and energy.

2.2 Timing of Routing

Pre-generation Routing: Decisions are made before any model generates a response. This minimizes latency. Strategies include inferring the query's domain or estimating its complexity to route to an appropriate model.
Post-generation (Cascade) Routing: Decisions are made after a model generates a response. If the output is deemed inadequate, the query is routed to a larger or different model. While flexible (e.g., multi-choice cascading), this approach incurs higher latency and cost due to multiple generations per query.

2.3 Implementation Strategies

The paper categorizes implementation strategies into four main groups:

A. Similarity-based Routing

Uses unsupervised or weakly supervised signals to route queries based on semantic similarity to past interactions.

Query Similarity: Computes cosine similarity between query embeddings (e.g., 1-Nearest-Neighbour). While lightweight, it often struggles with complex queries and generalization.
Clustering: Groups queries into semantic clusters (e.g., k-means) and routes based on historical performance per cluster. Some approaches (e.g., Jitkrittum et al.) allow for dynamic addition of new models without retraining the entire system.
Preference Similarity: Uses human preference data (e.g., from Chatbot Arena) to predict which model a user would prefer for a given query, often employing Bradley-Terry or ELO-based algorithms.

B. Supervised Routing

Treats routing as a classification or regression task once performance profiles are established.

Recommendation Systems: Uses matrix factorization to infer latent features and predict user preferences between models and queries.
Domain Classification: Routes queries to specialized models based on broad categories (e.g., math, medicine). While effective, it struggles with queries that do not fit predefined categories.
Query Complexity Inference: Classifies queries by difficulty level. This often outperforms domain-based routing but may suffer on out-of-distribution datasets. Approaches include inferring complexity via performance differences between model sizes or using regression to predict cost-quality scores.
Knowledge Graphs: Uses inductive graph frameworks (e.g., GraphRouter) to learn contextual relationships between tasks, queries, and models, improving generalization to new models without retraining.
Answer Confidence Inference: Estimates the probability of a generated answer being correct (e.g., FrugalGPT) to decide whether to escalate to a larger model.

C. Reinforcement Learning (RL) Routing

Frames routing as a sequential decision-making problem where the router learns from environmental feedback.

Stateless Algorithms: Directly map rewards to actions (e.g., Stochastic Learning Automata, Q-learning) without maintaining complex state. These are low-resource but struggle with evolving contexts.
State-based Algorithms: Incorporate contextual information (e.g., Contextual Multi-Armed Bandits) to determine which models perform best in specific situations.
Reward-based Inference: Uses separate reward models (trained via knowledge distillation) to predict which LLM will perform best, avoiding the need for direct interaction learning.

D. Generative Routing

Leverages the LLMs' own capabilities to make routing decisions.

Token/Sequence Probability: Measures output uncertainty (e.g., margin between top token probabilities) to decide if a larger model is needed.
Prompt-based Routing: Uses LLMs to classify queries or express "verbalized confidence" (pre- or post-generation). While accessible, LLMs often suffer from overconfidence.
LLM Fine-tuning: Fine-tuning models specifically for routing tasks (e.g., domain classification, complexity scoring, or API call generation).
Repeated Calls: Generates multiple responses to assess consistency; low consistency triggers routing to a larger model.
Code Execution: Uses successful code execution as an unambiguous confidence signal for routing in code-generation tasks.

3. Key Results and Findings

Efficiency vs. Performance: The survey demonstrates that routing can significantly reduce costs (up to 98% in some cases) while maintaining accuracy comparable to the best standalone models. However, the effectiveness of a strategy is highly dependent on the complementarity of the candidate models.
Strategy Trade-offs:
- Similarity-based methods are lightweight but often fail on complex or out-of-distribution tasks.
- Supervised methods generally offer higher accuracy but require labeled data and may need retraining for new models.
- Generative methods (like prompt-based) are flexible but can be resource-intensive or unreliable due to model overconfidence.
- RL-based methods offer adaptability but introduce complexity in training and state management.
Generalization: A recurring challenge is the inability of many transductive learning approaches to generalize to new routing candidates without retraining. Inductive approaches (e.g., GraphRouter, clustering-based performance vectors) show promise in addressing this.
Benchmarking: The paper notes a lack of standardized evaluation, with many studies using non-standard baselines. It highlights emerging benchmarks like RouterBench and MixInstruct.

4. Limitations and Challenges

The paper identifies several critical limitations in the current state of the field:

Cost Definition: Most studies focus solely on financial costs, neglecting computational and environmental impacts.
Standardization: There is a lack of standardized frameworks for evaluating routing strategies, making objective comparison difficult.
Candidate Diversity: Routing performance relies heavily on the complementarity of the candidate pool. Redundant models (e.g., same size, generalist training) offer limited gains.
Static Architectures: Most systems are static; adding new models or workflows typically requires retraining the entire routing infrastructure, hindering scalability in dynamic environments.
Overconfidence: Generative routing methods relying on verbalized confidence often suffer from LLM overconfidence, leading to suboptimal decisions.

5. Significance and Future Directions

This survey aims to provide a comprehensive overview and formalize routing as a performance–cost optimization problem. Its significance lies in:

Taxonomy: Providing a structured classification of routing strategies (pre/post-generation, similarity/supervised/RL/generative).
Industrial Transfer: Highlighting the gap between research and industry, where many companies still use simplistic conditional or prompt-based routing. The paper advocates for transferring lightweight, research-backed strategies to industrial applications.
Future Research: The authors call for:
- Incorporating computational and environmental costs into the optimization objective.
- Establishing standardized baselines (random, oracle, best standalone) for evaluation.
- Developing autonomous, adaptive routing strategies that can generalize to new components without retraining, treating the router as a dynamic agent within the system.
- Extending routing beyond the generation step to include embedding selection, retrieval strategies, and pipeline design.

The paper concludes that routing offers a practical pathway to improve the efficiency of LLM-based systems, enabling the development of adaptive, low-cost systems without sacrificing performance.