💬 NLP

Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

This paper introduces RLCR (Reinforcement Learning with Calibration Rewards), a novel training approach that augments binary correctness rewards with a Brier score to simultaneously improve the accuracy and calibrated confidence of language models, thereby mitigating the hallucination and calibration degradation often caused by standard reinforcement learning.

Original authors: Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, Jacob Andreas

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, Jacob Andreas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a very smart student to take a difficult exam. The goal isn't just for them to get the right answers, but for them to know when they are guessing and when they are sure.

This paper, titled "Beyond Binary Rewards," introduces a new way to train these AI "students" (Language Models) so they become both smarter and more honest about their confidence.

The Problem: The "Guessing Game" Trap

Currently, when we train AI to reason through hard problems (like math or trivia), we use a simple scoring system called Binary Rewards.

The Rule: If the answer is right, you get a point. If it's wrong, you get zero.
The Flaw: This system doesn't care how the AI got the answer. It gives the same point whether the AI solved it with deep logic or just guessed wildly and got lucky.
The Result: The AI learns to be a "confident gambler." It starts guessing answers with 100% confidence even when it has no idea what it's talking about. In the real world, this is dangerous because the AI might "hallucinate" (make things up) and sound very sure of its lies.

The Solution: RLCR (The "Honesty Coach")

The authors propose a new training method called RLCR (Reinforcement Learning with Calibration Rewards). Think of this as hiring a coach who grades the student on two things at once:

Did you get the answer right? (Accuracy)
Was your confidence level accurate? (Calibration)

How it works:
Instead of just saying "Right" or "Wrong," the AI is now forced to say: "Here is my answer, and here is a number from 0 to 1 representing how sure I am."

The coach uses a special scoring rule (called the Brier Score) to grade the confidence:

If the AI says "I'm 90% sure" and it's right, it gets a great score.
If the AI says "I'm 90% sure" and it's wrong, it gets a terrible penalty.
If the AI says "I'm 50% sure" (unsure) and it's wrong, it gets a smaller penalty.

This teaches the AI a crucial lesson: It is better to be unsure and wrong than to be confidently wrong.

The Analogy: The Weather Forecaster

Imagine two weather forecasters:

Forecaster A (Old Method): Always says, "It will definitely rain tomorrow!" If it rains, they are a genius. If it's sunny, they are just wrong, but they never admit they were guessing. They are "overconfident."
Forecaster B (RLCR Method): Says, "There is a 60% chance of rain." If it rains, they were right. If it's sunny, they admit, "Well, I said 60%, so I was wrong, but I knew there was a chance."

The paper shows that Forecaster B (the RLCR model) is much more reliable. They don't just get the right answers more often; they also tell you exactly when they are guessing, so you know when to trust them and when to double-check.

What the Experiments Showed

The researchers tested this on difficult questions about facts (like "Who won the Eurovision in 1969?") and math problems.

Better Honesty: The RLCR models stopped guessing wildly. When they were unsure, they lowered their confidence score.
Still Smart: Crucially, making the AI "honest" didn't make it "dumb." It kept its high accuracy on the questions it knew.
Generalizing: Even when the AI faced questions it had never seen before (out-of-domain), it remained much more honest about its uncertainty compared to the old "confident guesser" models.
Test-Time Boost: Because the AI's confidence scores were now trustworthy, the researchers could use them to improve the final answer. For example, if the AI generated 10 different answers, they could pick the one with the highest confidence score, and it was more likely to be correct.

The Bottom Line

This paper proves that by teaching AI to "think about its own thinking" and reward it for being honest about its uncertainty, we can build reasoning systems that are not only accurate but also reliable. They stop pretending to know everything and start telling you when they are just guessing.

). 2. A final answer (). 3. An analysis of uncertainty (). 4. A numerical confidence score (`).

The training objective combines a standard correctness reward with a Brier score, a proper scoring rule for confidence estimation. The reward function $R_{RLCR}$ is defined as:
$R_{RLCR}(y, q, y^*) = \mathbb{1}_{y \equiv y^*} - (q - \mathbb{1}_{y \equiv y^*})^2$
Where:

$y$ is the model's predicted answer.
$y^*$ is the ground-truth answer.
$q$ is the verbalized confidence score ( $q \in [0, 1]$ ).
$\mathbb{1}_{y \equiv y^*}$ is the indicator function for correctness.

Theoretical Guarantees

The paper proves that this reward function incentivizes both accuracy and calibration. Specifically, the expected reward is maximized when:

The model outputs the answer with the highest probability of being correct ( $y = \arg\max_{y'} p(y' \equiv y^*)$ ).
The confidence score $q$ accurately reflects the true probability of correctness ( $q = p(y \equiv y^*)$ ).

The authors demonstrate that this holds for any bounded, proper scoring rule. They contrast this with the Logarithmic score (log-loss), which, while a proper scoring rule, is unbounded. Theoretical analysis and toy experiments show that using an unbounded score like log-loss combined with a correctness term can lead to "reward hacking," where models output incorrect answers with zero confidence to maximize the calibration term at the expense of accuracy.

Key Contributions and Results

1. Improved Calibration Without Sacrificing Accuracy

Experiments on HotpotQA (multi-hop reasoning) and Big-Math (mathematical reasoning) demonstrate that RLCR matches the task accuracy of standard RLVR while substantially improving calibration.

In-Distribution: On HotpotQA, RLCR reduced the Expected Calibration Error (ECE) from 0.37 (RLVR) to 0.03. On math datasets, ECE dropped from 0.26 to 0.10.
Out-of-Distribution (OOD): While standard RLVR often worsens calibration on OOD tasks, RLCR improves it. For example, on OOD benchmarks, RLCR achieved an ECE of 0.21 compared to 0.46 for RLVR, while maintaining comparable accuracy.

2. The Role of Explicit Uncertainty Reasoning

Ablation studies reveal that RLCR's success stems from two independent components:

Calibration-Aware Reward: Training with the Brier score reward is the primary driver of improved calibration.
Explicit Uncertainty Reasoning: Prompting models to generate an <analysis> of their uncertainty further improves calibration, particularly for smaller models or when capacity is limited. However, even without explicit analysis, RLCR significantly outperforms RLVR in calibration.

3. Test-Time Scaling and Self-Consistency

The paper shows that verbalized confidence scores from RLCR can be leveraged at test time to improve performance:

Confidence-Weighted Scaling: Using verbalized confidence to weight majority votes or select the best-of-N responses yields higher accuracy and better calibration than likelihood-based selection or vanilla majority voting.
Self-Consistency: RLCR models exhibit lower variance in confidence scores across multiple reasoning chains for the same answer, indicating more stable and self-consistent uncertainty estimation.

4. Generalization

The approach was validated across different model families (Qwen2.5-7B, OlMo-2-7B, and Qwen3-8B), showing consistent improvements in calibration over RLVR baselines.

Significance and Claims

The paper claims that explicitly optimizing for calibration via proper scoring rules is a straightforward modification to existing reasoning training pipelines that yields more reliable reasoning models. The authors argue that:

Calibration is learnable: Models can be trained to not only answer correctly but also to accurately assess their own certainty.
Robustness: RLCR produces models that are more robust to distribution shifts, maintaining calibration where standard RL models fail.
Practical Utility: The verbalized confidence scores are not just diagnostic; they are actionable, enabling test-time scaling methods that improve both accuracy and reliability without requiring external reward models or additional supervision.

The authors remain modest about the results, acknowledging that while RLCR significantly reduces calibration error, out-of-domain calibration errors can still be high in absolute terms, and models may occasionally assign high confidence to contradictory answers. Nevertheless, the work establishes a path toward reasoning systems that are both accurate and capable of communicating uncertainty.