Technical Summary: Calibrating LLMs with Semantic-Level Reward

Problem Statement

As Large Language Models (LLMs) are deployed in high-stakes domains such as medical question answering and legal reasoning, the ability to estimate when outputs are likely to be correct is critical. Standard Reinforcement Learning with Verifiable Rewards (RLVR) optimizes for binary correctness but fails to account for model confidence. This binary reward structure is indifferent to confidence levels, structurally incentivizing overconfident guessing because expressing certainty carries no penalty. Consequently, RL-trained models often exhibit increased hallucination rates and poor calibration.

Recent attempts to address this involve training models to produce verbalized confidence scores (e.g., "I am 80% sure") alongside answers. However, the paper argues that verbalized confidence is an unreliable proxy for uncertainty. Empirical analysis shows that these scores are highly sensitive to prompt phrasing and answer text variations, even when the underlying semantic meaning remains identical. Verbalized confidence often fails to track the actual distribution of meanings produced by the model, calibrating a reporting interface rather than the model's true semantic uncertainty.

Methodology: Calibration with Semantic Reward (CSR)

The authors propose Calibration with Semantic Reward (CSR), a framework that calibrates LLMs directly in semantic space, bypassing the need for explicit confidence-reporting interfaces.

Core Concept

CSR frames calibration in terms of semantic-level uncertainty: the distribution of meanings a model produces when sampled repeatedly from the same question. A model is considered well-calibrated if:

1. Correct predictions form a concentrated semantic cluster (high semantic agreement).
2. Incorrect predictions yield dispersed, semantically diverse outputs (low semantic agreement).

Uncertainty is estimated post hoc from the answer distribution using Semantic Entropy, without requiring the model to generate a confidence token.

Reward Structure

CSR utilizes a dual-reward system optimized via Group Relative Policy Optimization (GRPO):

1. Verifiable Correctness Reward ($r_{RLVR}$): A standard binary reward ($1$ if the rollout matches the ground truth, $0$ otherwise). This shifts probability mass toward correct semantic modes.
2. Semantic Calibration Reward ($r_{calibration}$): A novel reward that aligns pairwise semantic agreement with correctness.
   • For a group of $K$ rollouts, the reward penalizes a rollout $C^{(j)}$ if its agreement with other rollouts $C^{(i)}$ does not match its correctness relative to the ground truth.
   • Specifically, it minimizes the binary cross-entropy between the pairwise agreement indicator $J(C^{(j)}, C^{(i)})$ and the correctness indicator $J(C^{(j)}, C^*)$.
   • Effect: This encourages correct rollouts to agree with each other (forming tight clusters) and incorrect rollouts to disagree (promoting exploration away from incorrect modes).

Training Objective

The authors observe that optimizing the calibration reward in isolation leads to degenerate failure modes (either collapsing to a single incorrect answer or generating maximally diverse incorrect answers). To prevent this, they employ a Curriculum Learning schedule:
$$r_{CSR}^{(j)}(t) = r_{RLVR}^{(j)} + \lambda(t) r_{calibration}^{(j)}$$
where $\lambda(t)$ is a non-decreasing function (linear schedule used in experiments). This ensures the model first establishes non-zero accuracy via $r_{RLVR}$ before the calibration signal shapes the distribution structure.

Integration

CSR integrates seamlessly with GRPO. Since GRPO already samples a group of rollouts to compute advantages, CSR utilizes this same group to evaluate pairwise semantic agreement, incurring no additional sampling overhead.

Key Contributions

1. Semantic Calibration Formulation: A shift from verbalized confidence to semantic-level uncertainty, providing a principled alternative to prompt-dependent confidence interfaces.
2. CSR Framework: A reinforcement learning method that aligns pairwise semantic agreement with correctness, enabling models to learn calibrated uncertainty without explicit confidence supervision.
3. Empirical Validation: Demonstrated across three model families (Llama, Qwen, Mistral) and four QA datasets (HotpotQA, TriviaQA, MSMARCO, NQ-Open).

Experimental Results

Experiments were conducted on 10,000 HotpotQA training examples, with evaluation on 1,000 examples per dataset (including out-of-distribution sets).

• Calibration Performance: CSR consistently achieved the lowest Expected Calibration Error (ECE) and highest Area Under the ROC Curve (AUROC) across nearly all settings.
  • ECE Reduction: Up to 40% reduction compared to verbalized-confidence baselines (RD, RLCR).
  • AUROC Improvement: Up to 31% improvement over verbalized-confidence baselines.
  • Generalization: CSR maintained strong calibration performance on out-of-distribution (OOD) datasets, whereas verbalized methods showed inconsistent gains.
• Accuracy: CSR maintained competitive accuracy with standard RLVR, avoiding the accuracy collapse seen when optimizing calibration rewards in isolation.
• Efficiency:
  • Token Cost: CSR's token cost was comparable to Base and RLVR models and significantly lower (2.7x–5.7x) than RLCR, which requires sequential reasoning and confidence generation.
  • Judge Efficiency: The authors explored replacing the LLM-based semantic judge with a lightweight token-level F1 threshold. While the LLM judge was more robust, the F1 surrogate preserved the bulk of CSR's calibration gains, offering a compute-efficient alternative.
• Robustness: Calibration gains persisted across different rollout budgets ($K$) and were robust to the choice of curriculum schedule.

Significance and Claims

The paper claims that CSR addresses the fundamental limitation of verbalized confidence methods: their reliance on a reporting interface that is sensitive to surface form and prompt variations. By calibrating the model's semantic distribution directly, CSR achieves:

• Robust Generalization: Calibration behavior transfers effectively to new datasets with different lexical content but similar semantic uncertainty structures.
• Reliability: It provides a more reliable uncertainty signal for downstream decision-making in high-stakes settings.
• Practicality: It avoids the parsing failures and format constraints associated with verbalized confidence (e.g., RLCR's reasoning traces or RD's integer bins), achieving a 100% parsing rate across all tested configurations.

The authors conclude that semantic-level calibration represents a promising direction for reliable uncertainty estimation, though they acknowledge the computational overhead of semantic equivalence judging as a current limitation, suggesting future work on rule-based consistency metrics.