Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts
This empirical study reveals that reported "unsolvability ceilings" in multi-LLM routing are largely inflated by evaluation artifacts like judge bias and truncation, which distort router training signals and lead to significant opportunity costs, necessitating more reliable evaluation protocols to accurately assess cost-quality tradeoffs.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you run a busy restaurant with a team of chefs ranging from a quick, junior cook to a world-famous, expensive master chef. Your goal is routing: deciding which chef should cook each order. You want to use the cheapest chef possible, but only if they can actually make the dish correctly. If the junior cook messes it up, you lose money on a refund; if you send every order to the master chef, you lose money on their high hourly rate.
For a long time, researchers believed there was a huge "unsolvable ceiling"—a big pile of orders that the junior cooks simply couldn't handle, meaning you had to send them to the expensive masters. This belief suggested that smart routing could save you a fortune.
However, this paper argues that the "unsolvable" pile is mostly an illusion created by bad measuring tools.
Here is the breakdown of the study using simple analogies:
1. The Problem: The "Judge" is Biased
To decide which chef is best, the researchers used an AI "Judge" (a sophisticated model) to taste-test the dishes. They found the Judge was making three specific mistakes that made the junior cooks look worse than they actually were:
Mistake A: The "Style Over Substance" Bias (Evaluation Misalignment)
- The Analogy: Imagine a food critic who loves long, flowery descriptions. If a junior cook gives a short, direct answer that is 100% correct, the critic gives them a low score because it "lacked flair." But if a master chef writes a beautiful, 5-page essay that gets the wrong answer, the critic gives them a high score because the writing was so good.
- The Result: The researchers found the Judge often penalized correct answers just because they were short or didn't sound "expert," while giving high scores to wrong answers that sounded fancy. This made it look like the junior cooks were failing more often than they really were.
Mistake B: The "Time Limit" Trap (Truncation)
- The Analogy: Imagine you tell the chefs, "You only have 2 minutes to cook." The junior cooks are fast, but the complex dishes take 3 minutes. The chefs get cut off mid-sentence, leaving the plate half-empty. The critic sees the empty plate and says, "This is a failure!"
- The Result: The study set strict token (word) limits. Because the junior cooks were often cut off before they could finish their answer, the Judge marked them as failures, even if they knew the answer. This artificially inflated the number of "unsolvable" tasks.
Mistake C: The "Wrong Plate" Issue (Format Mismatch)
- The Analogy: The restaurant rule says, "Write the answer on a specific ticket." The junior cooks sometimes write the answer on a napkin or in a sentence like "The answer is A." The system can't read the napkin, so it throws the ticket away and counts it as a failure.
- The Result: Smaller models were more likely to write answers in the wrong format. The system couldn't read them, so it counted them as unsolvable, again making the junior cooks look worse.
2. The Discovery: The "Unsolvability Ceiling" is Fake
When the researchers fixed these three mistakes (by checking the actual correct answers directly instead of just listening to the biased Judge), they found something surprising:
- The "Unsolvability" was much lower than reported.
- Many tasks that looked impossible for the cheap models were actually solvable; they just looked broken because of the bad measuring tools.
- The "gap" between the cheap and expensive models wasn't as wide as everyone thought.
3. The Consequence: The "Lazy" Router
Because the data was flawed, the "Routing System" (the manager deciding which chef gets the order) got confused.
- The Collapse: The manager saw that 79% of orders were labeled as "solvable by the cheapest cook." So, the manager stopped thinking and just sent everything to the cheapest cook.
- The Cost: This "lazy" strategy worked okay for easy tasks, but it failed miserably on the hard tasks that did need the expensive chef. The study found this "lazy" approach cost them 13–17% more than it should have. They were either overpaying (sending easy stuff to the master) or under-serving (sending hard stuff to the junior who couldn't finish).
4. The Solution: Better Rules
The paper suggests three simple rules to fix this:
- Double-Check the Judge: Don't just trust one AI to grade the work. Use a "Judge" that checks for style and a "Judge" that checks for the exact correct answer. If they disagree, investigate why.
- Give Enough Time: Make sure the word limits aren't so tight that chefs get cut off mid-sentence.
- Teach the Manager to Care: Train the routing system to care about the hard tasks, not just the easy ones. If the system is told to prioritize the rare, difficult orders, it won't just default to the cheapest option for everything.
Summary
The paper claims that the industry has been overestimating how many tasks are "too hard" for cheap AI models. This overestimation was caused by bad grading habits (liking fancy words over correct answers, cutting off long answers, and rejecting weird formatting). Because of this, routing systems have been trained to be lazy, sending everything to the cheapest model and missing out on significant savings. By fixing the grading rules, we can build smarter systems that actually know when to use the cheap model and when to pay for the expensive one.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.