🤖 AI

ReCal: Reward Calibration for RL-based LLM Routing

ReCal is a reward calibration framework for RL-based LLM routing that enhances performance and training stability by introducing hierarchical reward decomposition with component-wise advantage estimation and a distribution-aware optimization strategy to address ambiguous credit assignment and optimization bias caused by heterogeneous task difficulties.

Original authors: Qihang Yu, Hanwen Tong, Zhengqi Zhang, Bo Zheng, Feng Wei, Shengyu Zhang, Zemin Liu, Fei Wu

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qihang Yu, Hanwen Tong, Zhengqi Zhang, Bo Zheng, Feng Wei, Shengyu Zhang, Zemin Liu, Fei Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a busy call center. You have a team of agents with different superpowers: Agent A is a genius at math but terrible at spelling; Agent B is a creative storyteller but slow at calculations; Agent C is fast but makes up facts.

Your job is LLM Routing: deciding which agent should handle each incoming customer call.

In the past, managers used simple rules (like "send all math questions to Agent A") or trained a supervisor to guess the best agent. Recently, managers started using Reinforcement Learning (RL). This is like giving the supervisor a video game controller where they get points for good outcomes. The supervisor learns by playing the game over and over, trying to get the highest score.

However, the authors of this paper found a major problem with how these "games" were being played. They call their solution ReCal (Reward Calibration). Here is how it works, using simple analogies:

The Problem: The "Blurry Scorecard"

In the old way of training these supervisors, the computer gave them a single, final score for every call. For example: "You got 8.5 points."

But this scorecard was blurry and unfair in two ways:

The "Smoothie" Problem (Entangled Signals):
Imagine the score was a smoothie made of three ingredients: Correctness, Reasoning, and Formatting.
- Scenario A: The agent gave a perfect answer but forgot to use a comma. Score: 8.5.
- Scenario B: The agent gave a wrong answer but wrote a beautiful, perfectly formatted essay. Score: 8.5.
- The Confusion: The supervisor sees the same score (8.5) for two totally different behaviors. They don't know if they should stop making formatting mistakes or stop giving wrong answers. The signal is "entangled" (mixed up), making it hard to learn what actually matters.
The "Loud Crowd" Problem (Heterogeneous Distributions):
Some questions are easy (everyone agrees on the answer), and some are hard (agents disagree wildly).
- Easy Questions: The score is always high and consistent.
- Hard Questions: The scores vary wildly.
- The Bias: In the old training method, the "loud" hard questions (with high variance) or the "easy" questions (with huge rewards) would drown out the quiet, informative middle ground. The supervisor would get obsessed with the easy wins or the chaotic hard ones, ignoring the subtle, important lessons in between.

The Solution: ReCal (The "Smart Scorecard")

ReCal fixes this by changing how the supervisor sees the score. It acts like a two-step calibration process:

Step 1: Unmixing the Smoothie (Hierarchical Reward Decomposition)

Instead of giving one blurry score, ReCal breaks the score down into separate, clear categories before the supervisor learns.

The Analogy: Instead of saying "You got 8.5," the computer says:
- "Your Answer was 10/10."
- "Your Reasoning was 4/10."
- "Your Formatting was 2/10."
The Benefit: Now the supervisor knows exactly what to fix. They can see that the "Reasoning" part needs work, even if the "Answer" part was perfect. This is called Component-wise Advantage Estimation. It stops the "smoothie" from hiding the specific ingredients that need changing.

Step 2: Balancing the Volume (Distribution-Aware Optimization)

ReCal also fixes the "Loud Crowd" problem. It realizes that some questions are naturally more chaotic than others.

The Analogy: Imagine a classroom where some students are shouting and some are whispering. If the teacher just listens to the loudest voices, they miss the quiet geniuses.
The Fix: ReCal uses Variance-Aware Reweighting. If a group of agents is very confused about a question (high variance), ReCal turns the volume up on that lesson because it's a valuable learning moment. If the agents are all agreeing (low variance), it turns the volume down because there's nothing new to learn.
The Fix 2: It also uses Per-Dataset Normalization. If one dataset (like a math test) gives scores from 0 to 100, and another (like a history quiz) gives scores from 0 to 10, ReCal normalizes them so neither dataset dominates the training. It ensures the supervisor learns from all types of questions equally.

The Result

When the authors tested this new system (ReCal) on seven different types of questions (from general trivia to complex multi-step logic puzzles), it worked better than the old methods.

Better Learning: The supervisor learned faster because the feedback was clear.
More Stable: The supervisor didn't get confused by easy or chaotic questions; it stayed focused on the hard, informative ones.
Generalization: Even when they added new, unseen agents to the team during the test, the supervisor knew how to handle them, proving it learned the principles of routing, not just memorized the training data.

In short: ReCal stops the AI from guessing what a "good score" means by giving it a detailed report card and ensuring it pays attention to the right lessons, regardless of how loud or quiet the questions are.

Technical Summary: ReCal – Reward Calibration for RL-based LLM Routing

1. Problem Statement

Large Language Model (LLM) routing aims to dynamically select the most appropriate model or reasoning strategy for a given query to leverage the complementary strengths of multiple LLMs. While Reinforcement Learning (RL) has emerged as a promising paradigm for optimizing these routing policies, current methods face significant bottlenecks related to the quality and comparability of learning signals, particularly under heterogeneous tasks with varying difficulty.

The paper identifies two primary technical issues in existing RL-based routing frameworks:

Entangled Learning Signals: Routing policies are typically optimized using scalarized rewards that aggregate multiple objectives (e.g., answer correctness, reasoning quality, format accuracy) into a single score. This aggregation obscures credit assignment, making it difficult for the policy to distinguish which specific routing decision led to a reward improvement, as similar scalar improvements can arise from entirely different behavioral changes.
Heterogeneous Reward Distributions: Reward signals exhibit significant variability across instances and datasets. Some instances produce highly dispersed rewards due to model disagreement (high uncertainty), while others yield concentrated rewards (low uncertainty). Furthermore, different datasets induce substantially different reward scales and variances. This heterogeneity biases optimization toward easier cases or datasets with larger reward scales, causing informative but uncertain routing decisions to receive insufficient learning attention.

2. Methodology: ReCal Framework

To address these challenges, the authors propose ReCal, a Reward Calibration framework that treats policy learning as a two-stage calibration process: restructuring reward signals for clarity and adjusting their contribution to policy updates under heterogeneous distributions.

2.1 Learning Signal Clarity Calibration

Instead of estimating advantages directly from aggregated scalar rewards, ReCal employs a Hierarchical Reward Decomposition mechanism:

Component-wise Reward Definition: Rewards are decomposed into specific objectives, including answer correctness ( $r_{ans}$ ), reasoning quality ( $r_{info}$ ), format accuracy ( $r_{format}$ ), routing efficiency ( $r_{route}$ ), and router balance ( $r_{balance}$ ).
Hierarchical Activation: Auxiliary rewards are activated only when the response satisfies a correctness threshold, ensuring that optimization focuses on valid answers first.
Component-wise Advantage Estimation (CAE): Rather than aggregating rewards before advantage calculation, ReCal computes group-relative advantages independently for each reward component. These component-wise advantages are then weighted and aggregated. This preserves objective-specific relative improvements, allowing the policy to receive clearer, more discriminative supervision regarding which specific objectives are being optimized.

2.2 Distribution-aware Optimization

To mitigate optimization bias caused by heterogeneous reward distributions, ReCal introduces a Distribution-aware Optimization strategy:

Variance-aware Reweighting: At the instance level, the framework measures routing uncertainty using the variance of rewards within a sampled response group. Cases with higher variance (indicating greater disagreement among candidate models and higher uncertainty) are dynamically reweighted to receive stronger optimization signals. This ensures that informative, uncertain routing decisions are not ignored.
Per-dataset Normalization: At the dataset level, advantages are normalized independently within each dataset. This prevents datasets with disproportionately large reward scales or variances from dominating the policy updates, ensuring comparability across heterogeneous data sources.

2.3 Overall Training Objective

The final calibrated advantages are integrated into a Group Relative Policy Optimization (GRPO)-style objective. The training process maximizes the expected value of the clipped policy ratio multiplied by the calibrated advantage, explicitly separating objective-level supervision from distribution-level optimization variability.

3. Key Contributions

The paper outlines four primary contributions:

Learning-Signal Calibration Perspective: A novel viewpoint for RL-based LLM routing that jointly models learning signal clarity and optimization comparability under heterogeneous tasks.
Hierarchical Disentangled Reward Mechanism: A method that performs component-wise advantage estimation before aggregation, enabling clearer credit assignment and reducing interference among competing objectives.
Distribution-aware Optimization Strategy: A technique that reduces optimization bias through variance-aware reweighting and per-dataset normalization, ensuring stable learning across diverse datasets.
Empirical Validation: Extensive experiments demonstrating consistent improvements in routing performance and training stability over baselines.

4. Experimental Results

The authors evaluated ReCal on seven question-answering (QA) datasets, covering both General QA (Natural Questions, TriviaQA, PopQA) and Multi-Hop QA (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle).

Performance: ReCal achieved the best average Exact Match (EM) score of 0.481, outperforming the strongest RL-based baseline (Router-R1) by 6.7% (0.414). The improvements were particularly significant on challenging multi-hop benchmarks, such as a jump from 0.476 to 0.560 on 2Wiki and from 0.424 to 0.568 on Bamboogle.
Generalization: The method demonstrated strong generalization to out-of-domain datasets and unseen model pools. When tested with additional unseen LLMs (DeepSeek-V3 and Qwen2.5-72B) during inference, the policy maintained strong performance, suggesting it does not overfit to a fixed routing space.
Stability: Ablation studies confirmed that removing either the clarity calibration (LSC) or distribution-aware optimization (DAO) degraded performance. Training curves showed that ReCal exhibits smoother entropy convergence and more stable reward improvements compared to Router-R1, which suffered from oscillations and cross-dataset optimization interference.

5. Significance and Claims

The paper claims that ReCal provides a promising direction for future RL-based LLM routing by addressing the fundamental disconnect between entangled reward signals and heterogeneous data distributions. The authors argue that improving the clarity of learning signals and calibrating optimization variability is critical for enabling routing policies to learn reliable behaviors for challenging cases.

The work suggests that the proposed calibration mechanisms improve the generalization ability of routing policies rather than simply overfitting to specific training distributions. While the current evaluation is focused on QA-oriented routing, the authors posit that the framework offers a robust foundation for broader routing scenarios.

Limitations: The authors acknowledge that the evaluation is primarily limited to QA tasks, with effectiveness in multimodal systems remaining unexplored. Additionally, the current reliance on manually designed reward components and weighting coefficients may require tuning when adapting to new tasks or model pools.