💬 NLP

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

The paper introduces CALIBER, a method that improves the calibration of reasoning language models by distinguishing between pre-reasoning and post-reasoning confidence states and supervising each with a target aligned to its specific information context, thereby significantly reducing calibration error across various benchmarks and model sizes.

Original authors: Conor Finlay, Joshua Kurien, Saurabh Dash, Marzieh Fadaee, Beyza Ermis

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Conor Finlay, Joshua Kurien, Saurabh Dash, Marzieh Fadaee, Beyza Ermis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a very difficult math test. You have two moments where you might want to ask yourself, "How sure am I that I'm going to get this right?"

Before you start thinking: You look at the question and guess, "Hmm, this looks hard. I might get it right, or I might not."
After you've solved it: You have written down your answer and checked your work. Now you ask, "Okay, looking at the specific answer I just wrote, is it actually correct?"

The paper CALIBER argues that most current AI models make a mistake by treating these two moments as the same thing. They try to give you one single "confidence score" that is supposed to do both jobs at once. The authors say this is like asking a weather forecaster to predict both "Will it rain today?" (before looking at the sky) and "Did it rain this afternoon?" (after the storm has passed) using the exact same single number. It's confusing and leads to bad guesses.

The Problem: Mixing Up the "Before" and "After"

Think of an AI model like a detective solving a mystery.

The "Before" Confidence: This is the detective looking at the crime scene and saying, "Based on how messy this is, there's a 60% chance I can solve this case." This is about the difficulty of the puzzle.
The "After" Confidence: This is the detective looking at their final conclusion and the evidence they gathered and saying, "Based on the specific clues I found, I am 95% sure my conclusion is right." This is about the quality of the specific answer.

Existing methods often force the AI to give just one number. If the AI says "I'm 80% sure," is that because the puzzle was easy, or because the AI found great clues? The paper argues you can't know, and that's why the AI's confidence is often "miscalibrated" (meaning it says it's sure when it's wrong, or unsure when it's right).

The Solution: CALIBER (Calibration Before and After Reasoning)

The authors created a new system called CALIBER. Instead of asking for one guess, they ask the AI for two guesses at different times:

The Pre-Think Guess: Before the AI starts its "thinking" process, it gives a confidence score. This score is trained to predict: "What are the odds that I can solve this type of problem at all?"
The Post-Think Guess: After the AI has done its thinking and written an answer, it gives a second confidence score. This score is trained to predict: "Is this specific answer I just wrote actually correct?"

The Secret Sauce: The paper found that these two guesses need to be "taught" differently.

The Pre-Think guess is taught by looking at a group of attempts. If the AI gets the answer right 7 out of 10 times on similar questions, it should learn to say "70%" before it even starts.
The Post-Think guess is taught by looking at the specific answer. If the AI got this specific answer right, it should learn to say "100% sure." If it got it wrong, it should say "0% sure."

By separating these two jobs and teaching them with the right "homework" (data), the AI becomes much better at knowing what it knows.

What Happened When They Tested It?

The researchers tested this on math problems and general knowledge questions (like trivia). They compared CALIBER against other methods that only ask for one confidence score.

The Result: CALIBER was much better at being honest.
- When CALIBER said it was "90% sure," it was actually right 90% of the time.
- Other methods often said they were "90% sure" but were only right 60% of the time (overconfident) or said they were "50% sure" when they were actually right 90% of the time (underconfident).
The "Out-of-Distribution" Test: They also tested the AI on questions it had never seen before (like switching from math puzzles to trivia). Even on these new, tricky tasks, CALIBER stayed much more honest about its confidence than the other methods.

The "Gap" is Useful

One cool thing the paper mentions is the gap between the two numbers.

If the AI starts with a low confidence (e.g., "I'm not sure I can solve this") but ends with a high confidence (e.g., "But now that I've solved it, I'm sure!"), that gap tells us the thinking process helped.
If the AI starts high but ends low, it means the thinking process revealed a mistake.

Summary

In simple terms, CALIBER fixes AI confidence by realizing that "guessing if a problem is solvable" and "guessing if a specific answer is right" are two different skills. By teaching the AI to do these two things separately and correctly, the AI becomes a much more reliable partner that knows when to trust itself and when to be careful.

Technical Summary: CALIBER – Calibrating Confidence Before and After Reasoning in Language Models

Problem Statement

Reasoning language models are increasingly tasked with not only solving complex problems but also estimating the likelihood of their own success. Existing calibration methods typically elicit a single confidence estimate at one specific point in the generation process: either before the model begins thinking (pre-thinking) or after it has produced a final answer (post-thinking).

The authors argue that confidence in reasoning models is state-dependent.

Pre-thinking confidence conditions only on the prompt ( $x$ ). It should estimate the probability that the model's current policy can solve the prompt, regardless of the specific reasoning trace that will be generated.
Post-thinking confidence conditions on the prompt, the realized thinking trace ( $z$ ), and the final answer ( $\hat{y}$ ). It should estimate the correctness of that specific realized response.

Current methods often conflate these distinct information states by applying a single supervision target to a single estimate, or by using a target mismatched to the information state (e.g., training a pre-thinking estimate with an instance-level correctness target). This mismatch between what the model is asked to estimate and what it is trained to predict leads to suboptimal calibration, particularly under distribution shift.

Methodology: CALIBER

The authors introduce CALIBER (Calibration Before and After Reasoning), a framework that elicits two distinct confidence estimates within a single response and supervises each with a target matched to its information state.

1. State-Dependent Calibration

CALIBER structures the model's output to include:

Pre-confidence ( $p_{pre}$ ): Emitted before the thinking trace. It is conditioned only on the prompt.
Thinking Trace & Answer: The standard reasoning process.
Post-confidence ( $p_{post}$ ): Emitted after the thinking trace and answer. It is conditioned on the full realized response.

2. Supervision Targets

The framework assigns specific calibration targets based on the position of the estimate:

Pre-confidence Target (Group-wise): Supervised against the empirical success rate of the current policy on the prompt ( $\hat{p}(x)$ ), calculated as the average correctness of $N$ sampled rollouts for that prompt. This reflects the solvability of the prompt.
Post-confidence Target (Instance-wise): Supervised against the binary correctness of the specific rollout ( $c_i$ ). This reflects the correctness of the realized answer.

3. Training Objective

The training utilizes Reinforcement Learning with Verifiable Rewards (RLVR). The total reward for a rollout $i$ is a sum of four equally weighted components:

Accuracy Reward ( $R_{acc}$ ): Binary correctness of the answer.
Format Reward ( $R_{fmt}$ ): Ensures the model outputs the required tags and valid confidence values.
Pre-confidence Calibration Reward: $1 - (p_{pre} - \hat{p}(x))^2$ .
Post-confidence Calibration Reward: $1 - (p_{post} - c_i)^2$ .

To address optimization difficulties where the low-variance group-wise pre-confidence reward is dominated by high-variance instance-wise rewards, CALIBER employs a two-stage training schedule:

Warmup: Optimizes only the pre-confidence group-wise reward to establish diverse estimates.
Joint Training: Optimizes pre-confidence, post-confidence, and accuracy rewards simultaneously.

Key Contributions

State-Dependent Framing: The paper frames confidence in reasoning models as distinct quantities based on information state: pre-thinking confidence corresponds to question solvability (group-wise), while post-thinking confidence corresponds to answer-level correctness (instance-wise).
CALIBER Framework: A unified protocol that elicits both estimates and aligns each with its appropriate supervision target, avoiding the mismatch found in prior single-confidence methods.
Controlled Comparison: The authors isolate the contribution of position–target alignment by comparing CALIBER against a controlled set of baselines (RLCR, COREA-lite, CoCA-lite, DCPO-lite) under a shared training and evaluation protocol, removing confounding design choices like segmented credit assignment.
Diagnostic Gap: The framework allows for the analysis of the gap between pre- and post-confidence, serving as a per-example signal of how the model's uncertainty evolves during reasoning.

Results

In-Domain Performance (BigMathDigits)

7B Model: CALIBER reduces Expected Calibration Error (ECE) by 52.5% compared to the strongest single-confidence baseline (DCPO-lite). It achieves the best ECE, Brier score, and AUROC while maintaining accuracy within 2.1 percentage points of the best accuracy.
30B Model: CALIBER achieves the best ECE (0.022) while retaining competitive AUROC (0.797), distinguishing itself from methods like CoCA-lite that achieve low ECE but poor discrimination (AUROC ~0.5).

Out-of-Distribution (OOD) Generalization

Evaluated on GPQA, TriviaQA, and SimpleQA:

Calibration: CALIBER consistently achieves the best ECE and Brier scores on GPQA and TriviaQA for both model sizes.
SimpleQA: On this extreme distribution shift, CALIBER achieves the best ECE and Brier score for the 7B model. For the 30B model, while DCPO-lite achieves slightly lower ECE, CALIBER achieves substantially higher AUROC, indicating a better calibration-discrimination tradeoff.
Reliability: Reliability diagrams show CALIBER avoids severe overconfidence and aligns well with the diagonal in high-confidence regions.

Ablation Studies

An ablation swapping the targets (training pre-confidence with instance targets and post-confidence with group targets) revealed:

In-Domain: Results were mixed; the 30B model benefited significantly from correct alignment, while the 7B model showed mixed metrics.
OOD: Position–target alignment proved critical. CALIBER consistently outperformed the swapped baseline across all OOD datasets and model sizes, significantly reducing ECE. This suggests that matching the supervision target to the information state is essential for robust calibration under distribution shift.

Significance and Claims

The paper claims that CALIBER demonstrates that calibration for reasoning models should be treated as a dynamic problem rather than a static scalar attached to an answer.

Alignment is Key: The primary significance is the demonstration that the "position–target alignment" (matching pre-thinking to group success and post-thinking to instance correctness) is the critical factor driving calibration improvements, especially under distribution shift.
Diagnostic Utility: The gap between pre- and post-confidence provides a meaningful diagnostic of how the model's uncertainty changes after reasoning, rather than just a static confidence score.
Safety and Trust: By improving calibration without substantially sacrificing task accuracy, CALIBER supports safer deployment of reasoning models in settings where incorrect answers are costly, as it allows users to better trust the model's self-assessment.

The authors conclude that future work should extend this state-dependent approach beyond two points, test the prompt-level solvability hypothesis more directly, and validate these findings across additional model families and task domains.