🤖 AI

Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration

This paper demonstrates that LLM confidence calibration is highly sensitive to measurement protocols—specifically conditioning context, token-probability readout, and answer selection—revealing that verbalized confidence often reflects answer plausibility rather than correctness and urging the adoption of standardized reporting checklists for reliable evaluation.

Original authors: Hankyeol Kim, Pilsung Kang

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hankyeol Kim, Pilsung Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out how sure a large language model (LLM) is about its answers. You have two ways to ask:

The "Whisper": You ask the model to say a number out loud, like "I am 85% sure." (This is called verbalized confidence).
The "Internal Monologue": You look at the math inside the model's brain to see how likely it thought each word was to appear. (This is token probability).

For a long time, researchers thought these two methods were like looking at the same object from two slightly different angles. They assumed that if you compared them, you'd get a stable, reliable picture of the model's "self-knowledge."

This paper says: Stop. The picture changes completely depending on how you hold the camera.

The authors argue that comparing these two signals is less like taking a photo of a mountain and more like trying to measure the height of a building while standing on a trampoline. The result depends entirely on where you stand, what you're standing on, and which ruler you use.

Here is the breakdown of their findings using simple analogies:

1. The "Where You Stand" Problem (Conditioning Context)

This is the biggest surprise. Imagine you ask a student, "How sure are you that Paris is the capital of France?"

Scenario A: You ask them this question while they are just sitting at their desk.
Scenario B: You ask them this question after they have already written a long essay about Paris.

The paper found that if you measure the model's internal math (token probability) before it starts talking (Scenario A) versus after it has been prompted to give a confidence score (Scenario B), the results flip.

In one setup, the model seems perfectly calibrated (its spoken confidence matches its internal math).
In the other setup, the model looks wildly overconfident or underconfident.

The Takeaway: The "context" (what the model has just read or been asked to do) changes the math so much that it can reverse the conclusion. It's like measuring a shadow: the shadow's length depends entirely on where the sun is, not just on the object.

2. The "Which Answer Are We Talking About?" Problem (Answer Provenance)

Sometimes the model generates its own answer, and sometimes you feed it an answer and ask, "How sure are you this is right?"

Self-Generated: The model thinks of the answer itself.
Supplied: You give the model a correct answer and ask for a confidence score.

The paper found that when you feed the model a plausible but wrong answer (one that sounds smart but is factually incorrect), the model gives it almost the same high confidence score as a correct answer.

Analogy: Imagine a weather forecaster. If you show them a fake weather map that looks very realistic, they might say, "I'm 90% sure this is accurate," even though it's wrong. They are judging how plausible the story sounds, not whether it is true.

3. The "Which Ruler" Problem (Token Readout)

When calculating the internal math, do you look at the probability of the first word of the answer, or the average probability of the whole sentence?

The paper found that changing this tiny detail (like switching from a ruler in inches to a ruler in centimeters) changes the results enough to flip the sign of the conclusion in many cases. It's a small technical tweak, but it matters a lot.

4. The "Which Calculator" Problem (Estimator Choice)

Researchers use different mathematical formulas (estimators) to calculate "calibration error."

The Good News: Changing the calculator (the formula) didn't change the results much.
The Bad News: Changing where you stand (context) or what you measure (answer source) changed the results massively.

The Big Conclusion

The paper concludes that neither the spoken confidence ("I'm 90% sure") nor the internal math is a direct, unchangeable truth about the model's mind.

Instead, they are behavioral measurements. They are like a person's reaction to a specific question in a specific situation. If you change the situation (the prompt, the context, the answer source), the reaction changes.

The "Checklist" Metaphor:
The authors suggest that if you want to report how confident an AI is, you can't just say, "Our model is 90% calibrated." That's like saying, "The building is 100 feet tall" without saying if you measured it from the basement or the roof.

You must report the "Protocol":

Who wrote the answer? (Did the model write it, or did you give it to it?)
Where were you standing? (Did you measure the math before or after the model started talking?)
How did you read the math? (Did you look at the first word or the whole sentence?)

Summary:
Don't trust a single number that claims to show how "sure" an AI is. That number is fragile. It depends entirely on the specific rules of the game you are playing. If you change the rules, the score changes. Therefore, we need to be very careful about how we measure and report these numbers, especially if we want to use them for important decisions.

Problem
Large Language Model (LLM) confidence calibration is critical for safe deployment in high-stakes domains. Current evaluations typically compare two signals: token-probability scores (derived from the model's internal logits) and verbalized confidence (where the model explicitly reports a probability in text). While prior work has established that verbalized confidence is sensitive to prompt elicitation methods, this paper argues that the comparison between verbalized and token-based confidence is also highly sensitive to "measurement choices" that are often left implicit. Specifically, the quantity being measured depends on: (1) which answer string receives the token-probability score (self-generated vs. supplied), (2) how that score is read from the answer tokens (e.g., first token vs. span mean), and (3) the conditioning context under which the probability is measured (bare query vs. prompt prefix used for verbalization). The authors posit that without explicit protocol specification, observed calibration gaps may reflect measurement artifacts rather than intrinsic model properties or alignment gains.

Methodology
The authors conduct a controlled sensitivity analysis across four QA benchmarks (TriviaQA, SciQ, TruthfulQA, MMLU) using three open 7–8B base/Instruct model families (Llama-3-8B, Mistral-7B-v0.3, Qwen2.5-7B), with robustness checks on larger Qwen2.5 variants (14B, 32B, 72B).

The study holds the verbalized-confidence elicitation fixed (a single continuous 0.0–1.0 prompt template) while systematically varying the "protocol axes" defining the token-probability comparison:

Answer Provenance: Comparing confidence elicited on self-generated answers versus supplied answers (gold, plausible wrong, and off-type wrong).
Scored Answer: Comparing token probabilities assigned to the model's generated answer versus a teacher-forced gold reference.
Token Readout: Comparing the answer-span geometric mean ( $T_1$ ) versus the first answer-token probability ( $T_2$ ).
Conditioning Context: Comparing token probabilities measured under a bare query versus under the verbalized prompt prefix.
Estimator: Comparing standard 10-bin Expected Calibration Error (ECE) against bin-free estimators (KDE-ECE, KS-Cal, Smoothed ECE).

The primary metric is the signed calibration gap $g = \text{ECE}(c_{\text{verb}}) - \text{ECE}(c_{\text{tok}})$ , where negative values indicate verbalized confidence is better calibrated.

Key Contributions

Protocol Sensitivity with Fixed Elicitation: The authors demonstrate that even with a fixed verbalized-confidence prompt, the comparison between verbalized and token signals is unstable. Conditioning context and token readout choices materially change the observed ECE gap, often reversing its sign. In contrast, changing the ECE estimator has negligible effect.
Answer Provenance and Plausibility: The study reveals that verbalized confidence tracks answer plausibility and provenance rather than correctness alone. When supplied with surface-plausible wrong answers, models assign confidence levels nearly identical to those for supplied gold answers, while clearly rejecting off-type mismatches.
Reporting Checklist: The paper proposes a reporting checklist for confidence-signal comparisons covering elicitation provenance, scored answer, token-probability readout, and conditioning context to ensure measurement reproducibility and interpretability.

Results

Instability of the Calibration Gap: Under the default "generated-answer, bare-context" protocol, Instruct models show near-parity between verbalized and token confidence (mean gap $g_A \approx -0.029$ ), contradicting claims of large calibration gains for verbalized confidence. However, shifting the conditioning context (measuring token probability under the verbalized prefix) shifts the gap by $\approx -0.090$ on Instruct models, frequently flipping the sign of the gap.
Magnitude of Protocol Effects: The magnitude of the shift caused by conditioning context (mean absolute shift 0.28) is comparable to the variation caused by changing the model family (0.37) or the dataset (0.17). Token readout also induces significant shifts (0.11), whereas estimator choice induces minimal shifts (0.01).
Provenance Sensitivity: In supplied-answer analyses, the difference in confidence between supplied gold answers and plausible wrong answers is marginal ( $\approx +0.021$ ), whereas the difference for off-type wrong answers is large. This suggests verbalized confidence reflects the plausibility of the provided text rather than a ground-truth correctness estimate.
Robustness: These findings hold across different model scales (up to 72B), different prompt scales (integer 0–100), and various calibration metrics (Brier, NLL, AUROC).

Significance and Claims
The paper argues that neither verbalized confidence nor token probability should be treated as a direct, context-free readout of model uncertainty. Instead, both are behavioral measurements whose interpretation is inextricably linked to the measurement protocol. The authors claim that scalar claims about alignment improving self-reported uncertainty, or the superiority of verbalized confidence over token probabilities, are only meaningful when the specific measurement configuration (answer provenance, readout method, conditioning context) is explicitly specified. The study serves as a cautionary analysis, urging the community to treat calibration comparisons as protocol-dependent sensitivity analyses rather than causal decompositions of model capabilities.

1. The "Where You Stand" Problem (Conditioning Context)

2. The "Which Answer Are We Talking About?" Problem (Answer Provenance)

3. The "Which Ruler" Problem (Token Readout)

4. The "Which Calculator" Problem (Estimator Choice)

The Big Conclusion

More like this