Technical Summary: Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Problem Statement

The adoption of Large Language Models (LLMs) as automated judges ("LLM-as-a-Judge") has emerged as a scalable alternative to human evaluation. Concurrently, reasoning-capable LLMs (models that generate explicit reasoning chains before answering) have shown superior performance in problem-solving tasks. However, it remains unclear whether these reasoning capabilities transfer effectively to judgment tasks, nor is the cost-benefit trade-off well understood.

The paper identifies three core challenges:

1. Heterogeneous Value of Reasoning: While reasoning improves accuracy on complex tasks (e.g., math, coding), it offers marginal or even negative gains on simpler evaluations (e.g., factual recall, safety) while incurring significantly higher computational costs due to token consumption.
2. Distribution Shift: Existing routing methods, which select between reasoning and non-reasoning modes based on expected performance, are typically trained on static datasets. They often fail when deployed on out-of-distribution (OOD) data where user queries and cost structures shift, leading to budget violations or suboptimal judge selection.
3. Lack of Theoretical Guarantees: Prior routing strategies are largely empirical and heuristic, lacking formal convergence guarantees or uniqueness proofs for the optimal routing policy.

Methodology: RACER

The authors propose Robust Adaptive Cost-Efficient Routing (RACER), a framework that dynamically selects between reasoning and non-reasoning judges under a fixed computational budget while accounting for distribution shift.

1. Formulation as Constrained Distributionally Robust Optimization

RACER formulates router learning as a constrained optimization problem. Let $\pi$ be the routing policy mapping a prompt-response tuple $z$ to a probability of selecting the reasoning mode ($a=1$) or non-reasoning mode ($a=0$). The objective is to maximize expected reward (judgment accuracy) subject to a cost constraint $C$.

To address distribution shift, the problem is reformulated as a Distributionally Robust Optimization (DRO) problem using a Kullback-Leibler (KL) divergence uncertainty set $U(\rho_n, \delta) = \{ \tilde{\rho} : D_{KL}(\tilde{\rho} \| \rho_n) \le \delta \}$:

$$\begin{aligned} \max_{\pi \in \Pi} \quad & \min_{\tilde{\rho} \in U(\rho_n, \delta)} \mathbb{E}_{(z,l) \sim \tilde{\rho}, a \sim \pi} [r(z, a, l)] \\ \text{s.t.} \quad & \max_{\tilde{\rho} \in U(\rho_n, \delta)} \mathbb{E}_{(z,l) \sim \tilde{\rho}, a \sim \pi} [c(z, a)] \le C \end{aligned}$$

Here, $r$ represents the reward (correctness) and $c$ represents the cost (token consumption). The formulation explicitly optimizes for the worst-case reward and worst-case cost within the uncertainty set, ensuring robustness against shifts in data distribution.

2. Tractable Algorithm via Primal-Dual Updates

Solving the min-max problem directly is challenging due to the non-parametric nature of the uncertainty set. The authors leverage a theorem showing that the worst-case distribution within a KL ball can be approximated by reweighting the empirical data:

• For reward minimization (worst-case reward), samples with lower-than-baseline rewards are upweighted.
• For cost maximization (worst-case cost), samples with higher-than-baseline costs are upweighted.

This allows the transformation of the robust objective into a reweighted empirical objective. The authors introduce a regularized Lagrangian $L_\beta(\pi, \lambda)$ incorporating entropy regularization $H(\pi)$ to encourage exploration and ensure convergence. The solution is obtained via a primal-dual algorithm:

• Primal Update: The policy $\pi$ is updated to maximize the reweighted reward minus the Lagrange multiplier times the reweighted cost, plus an entropy term.
• Dual Update: The Lagrange multiplier $\lambda$ is updated via gradient ascent to enforce the cost constraint.

3. Theoretical Guarantees

The paper provides the first theoretical analysis for LLM routing policies, proving:

• Existence and Uniqueness: Under standard convexity assumptions, the saddle point of the regularized Lagrangian exists and is unique.
• Linear Convergence: The primal-dual iterates converge to the optimal policy at a linear rate (the error contracts by a constant factor at each iteration), dependent on the cost bound, density ratio bound, and regularization parameter.

Key Results

Empirical Findings on Reasoning vs. Non-Reasoning

• Task Dependency: Controlled comparisons on benchmarks (JUDGEBENCH, REWARDBENCH) reveal that reasoning judges significantly outperform non-reasoning judges on math and coding tasks (where intermediate step verification is crucial).
• Diminishing Returns: On safety and knowledge tasks, reasoning offers limited gains or can degrade performance due to "overthinking" and introducing irrelevant factors.
• Cost: Reasoning incurs a substantially higher token cost (e.g., cost ratios ranging from 3.4x to 11.2x depending on the model and dataset).

Performance of RACER

• Accuracy-Cost Trade-off: RACER achieves superior accuracy-cost trade-offs compared to baselines (All-Instruct, All-Reasoning, Random routing, and other advanced routers like RouterBench-KNN and RouteLLM-MF).
• Robustness to OOD: In experiments where training and test distributions differ (e.g., training on Magpie Ultra/WildGuardMix and testing on OffsetBias or standard benchmarks), RACER maintains performance where non-robust baselines fail.
  • When OOD queries require less computation, RACER-R (reward-robust) maximizes accuracy by utilizing the slack budget.
  • When OOD queries require more computation, RACER-C (cost-robust) prevents budget violations, maintaining reliability where other methods fail.
• Model Generalization: The method generalizes across model scales (1.7B, 4B, 8B) and model families (Qwen3, Llama-3.1), consistently outperforming baselines by 0.6 to 1.1 percentage points in accuracy under fixed budgets.

Significance and Claims

The paper claims to be the first to:

1. Systematically characterize the conditions under which reasoning capabilities transfer from problem-solving to judgment tasks, highlighting that reasoning is not universally beneficial and must be used selectively.
2. Propose a unified framework (RACER) that jointly addresses the accuracy-cost trade-off and robustness to distribution shift via constrained distributionally robust optimization.
3. Provide the first theoretical guarantees for LLM routing, specifically proving the uniqueness of the optimal policy and linear convergence of the learning algorithm.
4. Demonstrate that adaptive routing can effectively allocate reasoning resources to instances where they yield the highest marginal benefit, achieving better performance than simply increasing the overall reasoning rate.

The authors position RACER as a complement to human evaluation, promoting more efficient and sustainable use of computation by avoiding the indiscriminate application of expensive reasoning models. They acknowledge limitations, noting that overly large uncertainty sets can induce conservative routing, and suggest future work on extending the framework to richer routing schemes.