🤖 machine learning

Reported Confidence in LLMs Tracks Commitment More Than Correctness

This paper demonstrates that verbal confidence reports in large language models primarily reflect an internal "commit-readiness" state driving the decision to answer rather than the actual correctness of the response, distinguishing them fundamentally from log-probabilities which track answer evidence.

Original authors: Dharshan Kumaran

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dharshan Kumaran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, but slightly mysterious, robot assistant to solve a riddle. You ask it, "What is the capital of France?" It thinks for a moment and says, "Paris." Then, you ask it, "How sure are you?"

The robot replies, "I am 100% certain."

In the world of Artificial Intelligence (AI), we usually assume that when the robot says "100% certain," it means, "I have checked my internal database, and I am statistically sure this answer is correct."

However, a new study from Google DeepMind suggests we might be misunderstanding the robot's confidence. The researchers found that when an AI says it is "confident," it isn't necessarily saying, "My answer is right." Instead, it is often saying, "I am ready to stand by this answer, even if it might be wrong."

Here is a breakdown of their findings using simple analogies.

The Two Types of "Confidence"

The study compared two different ways the robot measures its own certainty:

The "Math Confidence" (Log-Probability): This is like a calculator. It looks at the raw numbers and probabilities of its options. If the math says "Paris" is 99% likely to be the answer, the calculator is confident.
- The Finding: This type of confidence is a good judge of truth. If the math says it's confident, the answer is usually right. If the math says it's unsure, the answer is usually wrong.
The "Verbal Confidence" (The Robot's Words): This is when you ask the robot directly, "How confident are you?" and it answers in words or a number scale (like "Very Sure").
- The Finding: This type of confidence is a bad judge of truth, but a fantastic judge of commitment. When the robot says, "I am very sure," it doesn't mean the answer is definitely correct. It means the robot has made up its mind and is willing to commit to that answer in front of a user.

The "Commit or Quit" Game

To figure this out, the researchers set up a two-stage game, inspired by how scientists study decision-making in animals:

Stage 1: The robot answers a question and gives a confidence rating (e.g., "I'm 90% sure").
Stage 2: The robot is shown its own answer and asked: "Do you want to Commit (show this answer to a human) or Abstain (say 'I don't know')?"

The Surprise Result:
The robot's verbal confidence in Stage 1 was a much better predictor of what it would do in Stage 2 than it was a predictor of whether the answer was actually right.

If the robot said, "I'm super confident," it almost always chose to Commit in Stage 2, even if the answer was actually wrong.
If the robot said, "I'm not sure," it almost always chose to Abstain in Stage 2.

The Analogy:
Imagine a gambler at a casino.

Math Confidence is like the gambler checking the odds on a piece of paper. If the odds are good, the gambler knows they are likely to win.
Verbal Confidence is like the gambler shouting, "I'm going all in!"
The study found that the gambler shouting "I'm going all in!" is often just a signal that they are ready to bet, not a guarantee that they have the winning hand. They might be bluffing or just feeling bold. The "Math Confidence" (the odds) is the only thing that actually tells you if they will win.

The "Commit-Readiness" Switch

The researchers dug deeper into the robot's "brain" (its internal code) to see what was happening. They found a specific moment right after the robot gives an answer, but before it is asked to decide whether to commit.

At this exact moment, the robot's internal state is organized around one question: "Am I ready to stand by this?"

The robot has a "Commit/Abstain" switch that is very loud and clear in its brain.
The robot has a "Right/Wrong" switch that is much quieter and fuzzier.

When the robot generates a verbal confidence report, it is essentially reading the loud "Commit" switch, not the quiet "Right/Wrong" switch. It's like a person who feels a strong urge to speak up; they might feel very confident speaking, even if what they are saying is factually incorrect.

Why This Matters

For a long time, people have treated an AI's verbal confidence ("I'm 95% sure") as a direct measure of accuracy. We assumed that if the AI says it's sure, we can trust the answer.

This paper argues that this is a mistake.

Verbal Confidence is a behavioral signal. It tells you: "I am willing to show this answer to a human."
Math Confidence is a truth signal. It tells you: "This answer is likely correct."

If you rely on the robot's words to decide if an answer is true, you might get fooled. The robot might be very eager to commit to a wrong answer because its internal "commitment switch" is flipped, even though the "truth switch" is flickering.

The Bottom Line

The study concludes that we need to stop treating an AI's verbal confidence as a simple "truth meter." Instead, we should understand it as a "commitment meter." It tells us when the AI is ready to take a stand, but it doesn't necessarily tell us if that stand is on solid ground. To know if the answer is actually right, we need to look at the underlying math (the log-probabilities), not just the robot's enthusiastic words.

Technical Summary: Reported Confidence in LLMs Tracks Commitment More Than Correctness

Problem Statement

Large Language Models (LLMs) are frequently required to report confidence in their answers to guide user trust and downstream decision-making (e.g., deferring to humans or abstaining). While verbal confidence (VC)—explicit self-reports of certainty—is widely used, particularly in black-box settings where internal probabilities are inaccessible, it remains unclear whether these reports function as accurate estimates of answer correctness or as signals of a different internal state. The prevailing assumption is that confidence should track the probability of correctness; this paper challenges that assumption, proposing instead that verbal confidence may primarily function as a "commit-readiness" signal, indicating the model's behavioral disposition to stand behind an answer rather than its objective truth.

Methodology

The authors employed a two-stage abstention paradigm, inspired by neuroscientific studies of post-decision confidence in rodents and primates, to decouple answer generation from the decision to commit.

Experimental Design:
- Phase 1 (Answer & Confidence): Models answered questions (multiple-choice or freeform) and, in a separate step, reported verbal confidence (Class-VC on a 10-class scale or Num-VC on a 0–100 scale). Crucially, this phase occurred without any reference to a subsequent abstention task.
- Phase 2 (Commit/Abstain): Models were shown their Phase 1 answer and asked to make a binary decision: Commit (output the answer to a user) or Abstain (output "I don't know").
- Comparison Signal: The study compared VC against Calibrated Log-Probability (Cal-LP), a confidence signal derived from the temperature-scaled softmax distribution of answer tokens, which serves as a proxy for answer evidence.
Models and Benchmarks:
- Non-Reasoning Models: Gemma 3 (27B), Gemma 4 (31B, no-think mode), GPT-4o, and Qwen 3 (235B, no-think mode).
- Reasoning Models: Gemini Flash 3, Kimi K2 Think, GPT-Oss-120B, and Qwen 3 (235B, think mode).
- Datasets: SimpleQA, MMLU-Pro, SuperGPQA-hard, and HLE (Humanities Last Exam), covering varying difficulties and formats.
Analytical Approaches:
- Decision–Truth Gap: Defined as $\Delta_{DT} = \text{AUROC}_{\text{abstention}} - \text{AUROC}_{\text{correctness}}$ . This metric quantifies whether a signal predicts the commit/abstain decision better than it predicts objective correctness.
- Residualisation Analysis: To determine if VC's alignment with behavior is reducible to its alignment with truth, the authors regressed VC on Cal-LP (the best available correctness proxy) and analyzed the residual. They tested whether the residual VC predicted abstention or correctness.
- Mechanistic Analysis (Gemma 3 & 4): Linear probes were trained on post-answer (PA) residual-stream activations to decode correctness and the future commit/abstain decision.
- Causal Steering: In Gemma 4, the authors constructed steering vectors from PA activations and injected them into the residual stream during Phase 2 to test if manipulating the "commit-readiness" state causally altered abstention behavior.

Key Results

1. The Decision–Truth Gap

Across four non-reasoning models and two datasets, verbal confidence consistently exhibited a positive decision–truth gap, predicting the Phase 2 commit/abstain decision substantially better than it predicted Phase 1 correctness.

VC Performance: Class-VC and Num-VC showed median gaps of +0.13 and +0.20, respectively. In nearly all cells, VC predicted abstention significantly better than correctness.
Cal-LP Performance: Cal-LP showed near-zero or negative gaps (median +0.018), indicating its ability to predict abstention was tightly coupled to its ability to predict correctness. It did not show a systematic advantage in predicting behavior over truth.

2. Variance Decomposition and Residualisation

Variance Structure: ANOVA analysis revealed that the variance in VC was dominated by the later commit/abstain decision (partial $\eta^2 \approx 0.11–0.48$ ) rather than correctness ( $\eta^2 \approx 0.00–0.05$ ). Conversely, Cal-LP retained stronger correctness-related structure.
Residual Analysis: When the variance in VC shared with Cal-LP was removed, the residual VC remained a strong predictor of abstention (median AUROC = 0.64) but lost its ability to predict correctness (dropping to near-chance levels, median AUROC = 0.49).
Interpretation: This suggests VC contains a unique component aligned with commitment behavior that is independent of the answer-evidence signal captured by Cal-LP. The reverse analysis (residualising Cal-LP on VC) showed Cal-LP remained coupled to both truth and behavior, consistent with it being a primary answer-evidence signal.

3. Generalization to Reasoning Models

The decision–truth dissociation generalized to four reasoning models across four benchmarks. Despite the complexity of reasoning traces, VC continued to show a robust positive gap (median +0.16), while Cal-LP (where available) showed no comparable pattern. In reasoning models, Cal-LP was less effective at discriminating correctness, likely because uncertainty is distributed across the reasoning trajectory rather than concentrated at the final token.

4. Mechanistic Evidence in Gemma 4

Internal State Encoding: Linear probes on post-answer (PA) activations (before the abstention prompt was given) could decode the future commit/abstain decision with high accuracy (AUROC $\approx$ 0.94), far exceeding the decoding of correctness (AUROC $\approx$ 0.66).
Orthogonality: The direction in activation space encoding the commit/abstain decision was approximately orthogonal to the direction encoding correctness.
Causal Control: Steering along a VC-specific activation direction (orthogonal to Cal-LP) at the PA position causally shifted the model's subsequent abstention behavior in Phase 2. Low-confidence steering increased abstention rates, confirming that the internal state underlying VC is causally operative in commitment decisions.

Significance and Claims

The paper concludes that verbal confidence and log-probability confidence are not interchangeable uncertainty measures.

Nature of Verbal Confidence: VC is best understood not as a noisy estimate of correctness, but as a behavior-facing readout of an internal "commit-readiness" state. It reflects the model's disposition to stand behind an answer, a signal that is only partially anchored to objective truth.
Nature of Cal-LP: Cal-LP behaves more like a truth-evidence signal, where its relevance to behavior is inherited from its ability to track correctness.
Implications:
- Interpretation: Users and downstream systems should not treat verbal confidence as a direct proxy for the probability of correctness. A model may express high confidence in an incorrect answer if it is behaviorally "ready" to commit to it.
- Training: Current training objectives may reward confidence-like signals that regulate response style and caution without anchoring them tightly to correctness. Future work may need to target the internal post-answer representation to better align confidence with truth while preserving adaptive behaviors like abstention.
- Metacognition: The findings suggest that LLM confidence reports may function similarly to second-order metacognitive signals in humans, constructed downstream from the initial choice and other cues (e.g., fluency), rather than being a direct readout of the evidence that drove the choice.

The authors emphasize that while VC does track correctness above chance, its disproportionate alignment with commitment behavior is a fundamental feature of its internal organization, distinguishing it from option-distribution-derived measures.