Target-Workload Auditing for Quality-Constrained LLM Routing

Imagine a world where artificial intelligence assistants are everywhere, ready to answer questions, write stories, or solve problems. To make these services affordable for everyone, companies often use a mix of different AI models. Some models are incredibly powerful but expensive to run, like a luxury car that guzzles fuel. Others are smaller, faster, and much cheaper, like an efficient economy sedan. The challenge for the companies running these services is deciding which "car" to use for each specific request. They want to save money by using the cheaper model whenever possible, but they cannot afford to let the cheaper model give a wrong answer when the expensive one would have been right. This balancing act between saving money and maintaining quality is the central problem this research addresses.

The researchers, Wenjie Li and Jufeng Li from the China University of Petroleum, set out to find a reliable way to make this decision. They focused on a specific strategy called "routing," where a system acts as a traffic cop, directing questions to either the cheap model or the expensive reference model. The goal was to create a method that guarantees the service stays within a strict budget for errors while maximizing savings. They were particularly interested in a tricky situation: what happens when the types of questions people ask change? A system trained on one set of questions might make different mistakes on a new set. The team wanted to know if they could use a small, labeled sample of the new questions to adjust their traffic cop's rules and ensure the savings remained safe and effective.

To test their ideas, the researchers built a rigorous experiment using two very different sets of data. First, they used a large, established collection of questions and answers involving a mix of a powerful AI and a slightly less powerful one. Then, they repeated the process with a newer, more advanced pair of models to see if their methods held up in a modern context. The core of their work involved a clever way of measuring risk. Instead of just looking at how often the cheap model got things wrong, they looked at two specific scenarios. The first was "harm," which happens when the cheap model gives a wrong answer but the expensive model would have been right. The second was "rescue," which happens when the cheap model gets an answer right even though the expensive model would have failed. By tracking both of these outcomes, they could calculate the true net effect of using the cheaper model.

The team developed a process that acts like a safety certification. Before deploying a routing rule to the real world, they would test it against a small, labeled group of questions from the target audience. This "audit" allowed them to set a threshold—a specific point where the system decides to switch from the cheap model to the expensive one. They compared several different ways to make this decision. Some methods relied on complex scores that tried to predict the quality of an answer, while others used a simple random chance to decide. They also tested a method that only looked at the risk of harm, ignoring the potential for the cheap model to rescue a difficult question.

The results revealed a nuanced picture of when and how to save money. In situations where the questions were very similar to what the system had seen before, the researchers found that they could save a significant amount of money without needing to audit the new questions at all. The rules learned from the old data worked perfectly well on the new traffic. However, when the questions changed significantly, such as moving from general knowledge to complex mathematics, the old rules often failed. In these cases, using a small sample of the new questions to recalibrate the system was essential. This small investment in checking a few answers allowed the system to safely use the cheaper model much more often, recovering the cost of the check and then some.

One of the most striking findings was the value of the "rescue" effect. When the researchers allowed their system to credit the cheap model for getting answers right that the expensive model missed, the system could safely use the cheaper model on a much wider range of questions. In one specific test with modern models, a method that accounted for these rescues saved nearly fifty percent of the total cost, whereas a method that only looked at avoiding mistakes saved less than six percent. This showed that recognizing when the cheap model is actually doing a better job than the expensive one is a powerful lever for cost reduction.

The study also compared their method against other recent approaches. They found that simply using a random gate to decide which model to use, once properly calibrated with a small audit, could be surprisingly effective, especially when the complex scoring systems failed to transfer well to new types of questions. This suggested that sometimes a simple, well-tuned rule is better than a complicated prediction model. The researchers concluded that there is no single "best" way to route traffic for all situations. Instead, the best approach depends on the specific mix of questions being asked and the stability of the data. If the data is stable, a pre-calibrated system works best. If the data shifts, a small, targeted audit is necessary to unlock the savings.

Ultimately, this work provides a clear, auditable framework for companies to manage their AI costs without guessing. It demonstrates that by carefully measuring both the risks and the unexpected wins of using cheaper models, services can operate within strict quality budgets. The research confirms that while the landscape of AI questions is constantly changing, a methodical approach to testing and calibration can ensure that the drive for efficiency does not come at the expense of reliability. The findings offer a practical path forward for the next generation of affordable and high-quality AI services.