💻 computer science

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

This paper introduces RouteJudge, an open online platform for evaluating LLM routing strategies through user preference-based pairwise comparisons, accompanied by ORBIT, a modular toolbox that standardizes the development, benchmarking, and integration of routing algorithms.

Original authors: Guannan Lai, Haoran Hu, Han-Jia Ye

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guannan Lai, Haoran Hu, Han-Jia Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-tech restaurant. You have a huge kitchen staffed by chefs of all different skill levels: some are lightning-fast but make simple dishes, while others are slow, expensive geniuses who can craft complex masterpieces.

The Problem: The "One-Size-Fits-All" Mistake
In the past, when people wanted to use AI (like Large Language Models), they often just picked the "best" chef (the most powerful AI) for every single order. But this is wasteful. If a customer just wants a quick sandwich, sending them to the expensive, slow genius chef is a waste of money and time. If they want a complex, 10-course meal, the fast chef might fail.

This is where LLM Routing comes in. It's like a smart host who looks at the customer's order and decides: "Okay, for this simple request, send it to the fast chef. For this hard request, send it to the genius."

The Old Way of Testing: The "Answer Key" Trap
Until now, scientists tested these "smart hosts" using a rigid answer key. They would give the host a question, see which chef they picked, and check if that chef's answer matched a pre-written "correct" answer.

The Flaw: Real life isn't a multiple-choice test. Sometimes there are many "correct" ways to answer a question. One person might want a short, funny answer; another might want a long, serious one. The old tests couldn't tell if the host actually picked the chef the customer would have liked best.

The New Solution: RouteJudge
The authors introduce RouteJudge, which is like a live, open-air food festival.

How it works: Instead of checking against a rigid answer key, RouteJudge lets real people taste the food.
The Setup: When a customer asks a question, several different "smart hosts" (routing strategies) make their own recommendations.
The Taste Test: The system takes the two best recommendations, hides the names of the chefs and the hosts, and asks the customer: "Which of these two dishes do you prefer?"
The Score: If the customer likes the dish, the system gives credit to the host who recommended that chef. It's not about who made the "perfect" answer; it's about who made the choice the human actually preferred.

The Toolkit: ORBIT
Building these smart hosts is hard because everyone builds them differently. To fix this, the authors also created ORBIT (Optimal Routing and Budgeted Inference Toolbox).

The Analogy: Think of ORBIT as a standardized "kitchen kit" or a universal recipe book. It gives every researcher the same measuring cups, the same ingredients list, and the same cooking instructions.
Why it matters: This ensures that when researchers build a new "smart host," they are all playing by the same rules. It makes it easy to test a new host in the kitchen (offline) and then send them out to the food festival (RouteJudge) to see how real customers react.

What They Found (The Snapshot)
The paper shows early results from this new system:

Offline vs. Online: A host that looks great in the kitchen (on paper tests) doesn't always win at the food festival. Sometimes, simpler, cheaper strategies that humans actually prefer beat out complex, expensive ones.
Cost Matters: The system shows that the "best" chef isn't always the most expensive one. The smartest host is the one that balances cost, speed, and what the customer actually wants.

In Summary
This paper builds a new way to test AI managers. Instead of asking, "Did they pick the right answer?" it asks, "Did they pick the answer the human actually liked?" It provides a standardized toolkit (ORBIT) to build these managers and a live platform (RouteJudge) to test them against real human preferences, ensuring AI is used efficiently and effectively in the real world.

Technical Summary: RouteJudge and ORBIT

1. Problem Statement

Large Language Model (LLM) routing aims to automatically select the most suitable model from a heterogeneous pool for each incoming query, optimizing for constraints such as cost, latency, and throughput. While existing evaluation protocols rely on static benchmarks, "golden" answers, or automated scoring metrics, these approaches suffer from a critical limitation: they impose a fixed notion of response quality. In real-world interactions, user preferences are pluralistic and context-dependent. For tasks like creative writing, translation, or tutoring, multiple outputs may be factually valid, yet users may prefer different responses based on tone, detail level, or cost sensitivity. Consequently, a router that performs well under offline, benchmark-aligned metrics may fail to select models that users actually prefer in deployment. This paper identifies a "pluralistic preference alignment problem," where current evaluation methods do not adequately capture whether routing decisions lead to user-preferred responses under realistic query distributions.

2. Methodology

The paper introduces a two-part ecosystem designed to address the gap between offline benchmarking and online user preference: RouteJudge and ORBIT.

RouteJudge: Online Preference Evaluation Framework

RouteJudge is an online platform that evaluates routing strategies through anonymous pairwise user preferences rather than fixed reference answers.

Evaluation Record: Unlike model-level leaderboards, RouteJudge stores "routing-centered" records ( $Z$ ) containing the user query, budget constraints, routing decisions made by multiple strategies, selected model responses, user preference labels, inference costs, latencies, and task metadata.
Workflow:
1. Submission: A user submits a query and a cost budget.
2. Router Recommendation: Multiple routing strategies ( $R$ ) independently select a candidate model from the budget-feasible model set ( $M_C$ ).
3. Duel Selection: The platform aggregates votes to select a "duel pair" of models ( $m_A, m_B$ ) with the highest vote counts.
4. Anonymous Judgment: Users compare the responses from $m_A$ and $m_B$ in a blinded interface, choosing between "A Win," "B Win," "Tie," or "Both Bad."
5. Router Attribution: The preference signal is attributed back to the routing strategies that selected the winning (or losing) models. A router receives a score only if its selected model participated in the judged duel; otherwise, it is marked as non-participating ( $\emptyset$ ).
Metrics: The system tracks router-level win rates, Elo ratings, participation rates, and cost-quality Pareto frontiers, enabling task-conditioned and budget-aware analysis.

ORBIT: Optimal Routing and Budgeted Inference Toolbox

ORBIT is a modular, extensible toolbox that standardizes the end-to-end workflow of LLM routing to support the continuous expansion of methods evaluated on RouteJudge.

Standardization: It provides unified interfaces for benchmark loading, query representation (via various embedding layers), router implementation (rule-based, learned, retrieval-based, etc.), and budget-aware evaluation.
Composability: Researchers can swap datasets, encoders, and routers independently without rewriting core logic.
Integration Layer: ORBIT serves as the submission interface for RouteJudge. Researchers implement routers within ORBIT, validate them offline, and submit compatible routers for online evaluation.
Two-Stage Evaluation Pipeline:
1. Historical Replay: Submitted routers are evaluated against historical RouteJudge records to determine if they would have selected the user-preferred model in past comparisons.
2. Online Evaluation: Validated routers are deployed to the live platform, participating in real-time anonymous pairwise comparisons and receiving preference attribution.

3. Key Contributions

RouteJudge Platform: The first open platform specifically designed to evaluate LLM routing systems based on pluralistic user preferences via anonymous pairwise comparisons, shifting the evaluation target from model-level response quality to router-level decision quality.
ORBIT Toolbox: A standardized development and integration layer that enables reproducible offline benchmarking and seamless submission of routing methods to RouteJudge, addressing the engineering fragmentation in current routing research.
Evaluation Protocol: A novel protocol that attributes user preferences back to routing strategies, accounting for budget constraints, task types, and participation rates, rather than relying on static "best model" assumptions.
Open Ecosystem: The combination of RouteJudge and ORBIT creates a closed loop where methods can be developed offline, validated against historical preferences, and tested in live user-facing environments.

4. Preliminary Results

The authors present initial empirical findings from both the offline ORBIT pipeline and the online RouteJudge platform (as of June 2026):

Offline Consistency: Using the RouterEval benchmark, ORBIT successfully demonstrated that different routing methods (e.g., EmbedLLM, Eagle, EquiRouter) can be compared under a unified protocol, revealing performance-cost trade-off curves and metrics like nAUC and Peak Score.
Online Differentiation: On RouteJudge, with 109 user-voted comparisons, the platform successfully differentiated between routing strategies. RouterLLM-MF achieved the highest Elo score (1278), while NIRT-Router showed the highest observed win rate (80.00%).
Divergence from Offline Metrics: The results indicate that strong offline designs do not always translate to online preference success. Some routers with explicit learned scoring mechanisms performed lower in user preference than simpler non-parametric or matrix-factorization-based methods.
Cost-Preference Trade-off: Analysis of model-level data showed that preference is not solely determined by cost; some low-cost models achieved competitive win rates, suggesting that effective routing requires adapting model selection to user budgets and task contexts rather than defaulting to the most capable models.

5. Significance and Claims

The paper claims that RouteJudge and ORBIT collectively address the "pluralistic preference alignment problem" in LLM routing. By moving beyond static benchmarks, the ecosystem allows researchers to evaluate whether routing decisions align with real user preferences under realistic constraints. The authors position this work not as a final ranking of all routing methods, but as a foundational step toward an open evaluation ecosystem. They argue that future routing research must consider not only benchmark-optimal model selection but also user-preferred, cost-aware, and deployment-sensitive behaviors. The system is designed to be continuously expandable, allowing the community to contribute new routers and benchmarks while maintaining reproducibility through the ORBIT standard.