Most of the LLM routing gap is task type
This paper argues that the perceived gap in LLM routing performance is largely due to task type rather than model selection complexity, demonstrating that a simple static routing strategy based on task type and language outperforms learned routers and the best single model while significantly reducing costs.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, large language models act as powerful digital minds capable of answering questions, writing code, and solving puzzles. However, no single model is perfect at everything; some excel at mathematics while others struggle with creative writing, and a model that is brilliant in English might stumble when asked to speak Hindi. To get the best results, engineers have developed "routers," which are smart systems designed to look at a user's question and instantly decide which specific model in a pool is best suited to answer it. The goal is simple: route the question to the right expert to save money and get a better answer than any single model could provide alone. But for these routers to work, they must be able to distinguish between questions that are easy for one model and hard for another with perfect consistency.
A recent study by Janghoon Lee challenges the assumption that these routers are as sophisticated as we might hope. The researchers set out to understand exactly why current routing systems often fail to beat the simple strategy of just always using the strongest available model. They built a massive test matrix involving fourteen different AI models, seven distinct types of tasks ranging from coding to legal document extraction, and three languages: English, Korean, and Hindi. They asked every model to answer every single one of the 294 questions in their test set, and then they ran the entire experiment twice under identical conditions to see if the results would hold up. What they discovered was that the gap between the best possible outcome and what current routers achieve is not caused by a lack of complex algorithms, but by the fact that most of the "missed" questions are actually very easy to predict if you simply look at the type of task and the language involved.
The study began by running the same 4,116 model-question combinations twice. Even with the computer settings locked to be perfectly deterministic, the results were not identical. In about 5.37 percent of the cases, a model that got an answer right the first time got it wrong the second time, or vice versa. This fluctuation, which the researchers call the "reproducibility floor," means that any improvement a router claims to make must be larger than this natural noise to be considered real. When the researchers applied a strict rule—counting an answer as correct only if the model got it right in both runs—they found that only 29 questions out of the 294 could potentially be improved by routing. This is a very small number, and it represents the true "ceiling" of what a router could possibly achieve on this specific set of data.
The researchers then asked what these 29 difficult questions had in common. They found that the vast majority of them were determined by the type of task being performed. For instance, if a question was about coding, a specific model was almost always the best choice, regardless of the router's complex calculations. By simply assigning one specific model to handle all coding questions and a different model for all math questions, the researchers could recover 21 of those 29 difficult questions without using any learned intelligence or complex decision-making. Adding the language of the question to this simple rule recovered two more. This left only six questions out of the 294 that remained unoptimized, a number so small that it fell below the threshold of the natural noise the researchers had measured. In other words, the tiny amount of gain a sophisticated router might find was smaller than the random variation that occurs just by running the same test twice.
Furthermore, the study revealed that the identity of the "best" single model is not a fixed fact but depends entirely on how you measure success. Depending on whether you count a question as correct based on a single run or require it to be correct in two runs, or whether you exclude questions where the model ran out of memory, a different model would win the title of "best." In some cases, the difference between the top model and the runner-up was so small—less than one item out of 294—that it was indistinguishable from the natural noise of the system. This suggests that the question "which model is best?" does not have a single, stable answer on this dataset, making it difficult for a router to learn a reliable rule for picking a winner.
Instead of building a complex, learned router, the researchers adopted a static lookup table. This is a simple, pre-made list that says, "If the question is about coding in Korean, send it to Model A; if it is about math in English, send it to Model B." This static table, which required no training or learning, managed to answer 262 of the 294 questions correctly. More importantly, it did so at a cost of roughly $3.33 per run, whereas the best single model, which would have been the default choice without routing, cost $7.69 per run to answer the same questions. The static table was not only more accurate but also significantly cheaper, beating the best single model on both fronts.
The study concludes that the promise of routing is not in building increasingly complex systems to find hidden patterns, but in recognizing that the most valuable patterns are often visible on the surface. The distance between the best possible outcome and the current reality is largely made up of task types and languages that can be identified immediately. Once these are accounted for with a simple, static rule, the remaining room for improvement is so small that it is lost in the noise of the system itself. The researchers argue that before investing in complicated routing algorithms, we should first ask how much of the answer is already visible in the request. In this case, the answer was that most of the gain was already there, waiting to be captured by a simple table rather than a complex machine.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.