Task- and Session-Level Model Routing: A Common-Interface Hybrid Evaluation of Four Open-Source Routers Across Four Benchmarks
This paper introduces a common-interface hybrid evaluation protocol to compare four open-source model routers across four benchmarks, revealing that their performance gains are primarily driven by selected-tier composition rather than task-specific targeting, thereby establishing fixed-tier baselines as essential controls for future router assessments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, a new kind of software agent is emerging. These are not just programs that answer questions; they are systems that can plan, use tools, and navigate complex digital environments to complete multi-step jobs. To function effectively, these agents often rely on a "router," a decision-making component that acts like a traffic controller. Its job is to look at a specific request and decide which version of the underlying AI model should handle it. The logic is straightforward: some tasks are simple and can be solved by a fast, inexpensive model, while others are difficult and require a more powerful, expensive model. By sending the right task to the right model, developers hope to save money without sacrificing the quality of the work. However, the landscape of these routers is fragmented. Different research groups and companies have built their own versions, testing them on different sets of tasks with different rules, making it nearly impossible to compare them fairly or know which one actually works best in a real-world setting.
A team of researchers at Indiana University set out to bring order to this confusion. They designed a common testing ground where four different open-source routers could be evaluated side by side under identical conditions. Instead of letting each router run its own separate tests, the researchers created a controlled experiment involving four distinct benchmarks, which are standardized collections of tasks ranging from simple questions to complex web navigation and customer service simulations. They froze the tasks, meaning the questions and scenarios were fixed and did not change, and they locked the pool of available AI models to three specific options: a cheap, fast model; a mid-range model; and a powerful, expensive model. This setup allowed them to watch exactly how each router decided which model to use and whether those decisions actually led to better results.
The researchers ran 290 frozen tasks through this system, testing how each router performed across the different benchmarks. They discovered a surprising pattern: three of the four routers behaved almost exactly the same way, regardless of the task at hand. Two of them simply chose the cheapest model for every single request, while a third router, despite being designed to analyze the content of the request, ended up selecting the mid-range model for nearly every task. This happened because the third router's design relied on matching the request to a small set of pre-written examples, and when it couldn't find a match, it defaulted to the mid-range model. The fourth router was the only one that actually varied its choices based on the specific content of the prompt, shifting its selection between the cheap, mid-range, and powerful models depending on what it read.
When the researchers measured the success rates, the results challenged the assumption that smart, content-aware routing is always superior. The router that simply picked the mid-range model for everything performed just as well as the router that tried to be clever about its choices. In fact, on three of the four benchmarks, the simple "always pick the middle option" strategy matched the performance of the most sophisticated router exactly. On the fourth benchmark, the difference was so small it was statistically negligible. The router that varied its choices based on the prompt content did not show any clear advantage over a strategy that ignored the content entirely but used the same mix of models. The data suggested that the success of these systems was driven more by which models they happened to pick most often, rather than by their ability to correctly identify which specific task needed which specific model.
The study also looked at the cost and speed of these different approaches. The router that always chose the cheapest model was the most cost-effective, but it failed more often on difficult tasks. The router that always chose the mid-range model offered a middle ground, matching the success rate of the smartest router but without the complexity. The researchers found that the "smart" router did not save money or improve success rates enough to justify its complexity in these specific tests. The observed gains in performance were closely tied to the overall mix of models selected rather than to the router's ability to target specific tasks. For instance, on one benchmark involving web navigation, the router that varied its choices created a unique balance of cost and success that no single fixed model could replicate, but even there, it did not outperform a simple strategy that just happened to pick the right mix of models by chance.
This research highlights a critical need for better evaluation methods in the field of AI agents. The authors argue that to truly understand if a router is working, researchers must compare it against simple, fixed strategies, such as "always use the cheapest model" or "always use the best model." Without these baselines, it is easy to mistake a lucky outcome or a favorable mix of models for a genuine breakthrough in routing intelligence. The study concludes that under the specific conditions tested, the complex mechanisms designed to analyze and route tasks did not provide a measurable benefit over simpler, static rules. The findings are limited to the specific models and tasks used in the experiment, but they serve as a strong reminder that in the rush to build smarter AI systems, the simplest solution—picking the right tool for the job by default—often performs just as well as the most complicated one.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.