💻 computer science

Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations

This paper introduces a novel evaluation framework for language model confidence estimation that assesses robustness, stability, and sensitivity to language variations, revealing that existing methods often fail to distinguish semantically different answers despite their alignment with correctness.

Original authors: Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, but sometimes overconfident, robot assistant a question. You ask, "What is the capital of France?" and it says, "Paris," with a confidence score of 99%. That seems great. But what happens if you ask the same question slightly differently, like "Tell me the capital of France, please"? Does the robot still say "99%"? Or does it suddenly drop to "45%"?

This paper is about checking if that robot's "confidence meter" is actually reliable when the way you talk to it changes. The authors argue that just checking if the robot is right (calibration) isn't enough. We also need to check if its confidence stays steady when the language changes, and if it knows when it's actually wrong because the meaning changed.

Here is a breakdown of their findings using simple analogies:

The Problem: The "Shaky Compass"

The authors say current methods for checking AI confidence are like checking a compass only when the ship is perfectly still. They check if the compass points North when the ship is stationary. But in the real world, the ship rocks (the prompt changes), and the compass might spin wildly even if it's pointing North.

They found that many AI confidence checkers are well-calibrated (they are right on average) but shaky (they give different confidence scores for the same answer just because you asked the question differently).

The Three New Tests

To fix this, the authors introduced three new tests for these confidence meters:

Robustness (The "Same Answer, Different Wording" Test)
- The Analogy: Imagine you ask a weather forecaster, "Will it rain?" and they say "80% chance." Then you ask, "Do you think it's going to rain?" If the forecaster suddenly says "20% chance" just because you changed the words, their meter is broken.
- The Finding: Most AI confidence methods passed this test. They are good at ignoring minor changes in how you phrase your question.
Stability (The "Same Meaning, Different Words" Test)
- The Analogy: Imagine the robot answers "Paris" and then, a second later, answers "The City of Light." Both mean the same thing. If the robot says "99% confident" for the first and "10% confident" for the second, it's being inconsistent.
- The Finding: Most methods passed this too. They give consistent scores when the answer means the same thing, even if the words are different.
Sensitivity (The "Different Meaning, Different Score" Test)
- The Analogy: This is the big failure. Imagine the robot answers "Paris" (99% confident) and then answers "London" (99% confident). Both are given the same high confidence score, even though one is right and the other is wrong. The robot's confidence meter is numb. It doesn't feel the difference between a right answer and a wrong one if the words look similar.
- The Finding: This is where almost all methods failed. They are terrible at noticing when the meaning of the answer changes. They keep giving high confidence scores even when the answer is nonsense, as long as the nonsense looks like a normal sentence.

Why Does This Happen?

The authors found that the methods which look at the question but ignore the answer are the most "numb."

The "Blind" Method: Some methods only look at the question (like a student who reads the test question but doesn't look at their own answer before grading it). They can't tell if the answer is wrong because they aren't looking at it.
The "Attentive" Method: Methods that actually read the generated answer are slightly better at spotting differences, but still not good enough.

The "Bigger is Better" Myth

There is a common belief that bigger AI models are always smarter and more reliable. The authors tested this.

The Finding: Making the model bigger (scaling up) helped a little bit, but it didn't fix the "numbness" problem. A giant model is still just as likely to be confidently wrong about a different answer as a small model.

The Bottom Line

The paper concludes that we cannot just pick the "best" confidence checker for every job. It depends on what you need:

If you need a system that doesn't freak out when you rephrase a question, you want Robustness.
If you need a system to pick the best answer out of ten different options (like a judge), you desperately need Sensitivity (the ability to tell the difference between right and wrong answers).

Currently, most systems are great at the first two but terrible at the third. The authors warn that if we use these systems for high-stakes decisions (like medical advice or legal judgments) without fixing this "numbness," we might trust a confidently wrong answer just as much as a right one.

In short: The AI's confidence meter is good at ignoring your typos, but it's terrible at realizing when it's talking nonsense. We need to teach it to listen to its own answers, not just the questions.

Technical Summary: Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations

Problem Statement

Confidence Estimation (CE) is critical for establishing user trust in Large Language Models (LLMs), particularly in high-stakes domains like medicine and law. Current evaluation paradigms primarily focus on calibration (the alignment between stated confidence and actual accuracy, measured by metrics like ECE and Brier score) and discrimination (the ability to rank correct answers higher than incorrect ones, measured by AUROC).

However, these existing metrics overlook a fundamental property of language: variability. Natural language expressions can vary significantly in surface form (word choice, phrasing) while retaining identical semantic meaning, or conversely, vary slightly in wording while representing completely different meanings. Existing CE methods are often evaluated under static conditions, failing to assess whether confidence scores remain consistent under semantically equivalent prompt perturbations or whether they appropriately shift when the semantic meaning of an answer changes. A method may be well-calibrated on average yet produce fluctuating confidence scores for semantically identical inputs, or fail to distinguish between semantically distinct answers, leading to unreliable behavior in real-world applications where prompt engineering and answer variations are common.

Methodology

The authors propose a novel evaluation framework based on three complementary properties that specifically address language variation. These metrics do not require ground-truth correctness labels, relying instead on multiple generated responses under different conditions.

1. Proposed Metrics

Robustness to Prompt Perturbations (P-RB): Measures whether a CE method outputs the same confidence score when the input prompt is altered (e.g., different instructions or phrasing) while the generated response remains constant. This isolates prompt variance from generation noise.
Stability Across Semantically Equivalent Answers (A-STB): Evaluates whether the CE method assigns consistent confidence scores to different generated responses that are semantically equivalent (e.g., "Concave lens" vs. "Concave").
Sensitivity to Semantically Different Answers (A-SST): Measures whether the CE method assigns distinct confidence scores when the meaning of the answer changes. A sensitive method should detect semantic shifts even if the surface form is similar.

2. Experimental Setup

Models: 11 LLMs across five major families (Mistral, Llama 3, Qwen 2.5, OLMo 2, GPT-4o) ranging from 7B to 123B parameters.
Datasets: Four open-ended QA benchmarks (Natural Questions, SciQ, TriviaQA, PopQA) with 1,000 test samples each.
CE Methods: 10 diverse methods categorized into:
- Logit-based: Sequence Likelihood, Boosted Probabilities, Platt Scaling, P(True).
- Linguistic: Verbalized Confidence.
- Auxiliary: Calib-1 (trained classifier).
- Internal States: Attention Score, Hidden Score, SAPLMA, P(IK).
Procedure:
- Robustness: Greedy decoding with 10 semantically equivalent prompt variations.
- Stability/Sensitivity: Stochastic decoding (temperature 0.7) to generate 10 sampled answers per prompt. Semantic equivalence is determined using an LLM-as-a-judge (GPT-4o).

Key Results

1. Independence from Traditional Metrics

The proposed metrics (P-RB, A-STB, A-SST) are largely independent of traditional calibration and discrimination metrics (ECE, Brier, AUROC). Correlation analysis (Kendall's $\tau$ ) shows weak correlations (e.g., $\tau=0.26$ between Sensitivity and AUROC), indicating that the new metrics capture previously overlooked aspects of confidence quality.

2. Performance of CE Methods

High Robustness and Stability: Most CE methods achieve high scores in robustness and stability, suggesting they are generally consistent when prompts or semantically equivalent answers vary.
Critical Lack of Sensitivity: The most significant finding is that almost all methods struggle with sensitivity. They fail to adjust confidence scores when the answer meaning changes.
- Methods relying on generation-side information (e.g., P(True), SAPLMA, Sequence Likelihood) show higher sensitivity.
- Methods relying solely on question-side information or final question tokens (e.g., P(IK)) are almost completely insensitive to answer changes (A-SST $\approx$ 0).
- Post-hoc scaling methods (e.g., Platt Scaling) tend to compress confidence differences, further reducing sensitivity.

3. Scaling Laws

While larger models generally show improved calibration, the improvement in the proposed metrics (robustness, stability, sensitivity) with model scale is mild and inconsistent. Scaling alone does not resolve the sensitivity issue, suggesting that model size is not a sufficient proxy for handling language variation in confidence estimation.

4. No Universal Best Method

No single CE method outperforms others across all metrics and datasets. For instance, P(IK) achieves the best overall Mean Reciprocal Rank (MRR) but performs poorly on robustness and sensitivity. This highlights a trade-off: methods optimized for calibration often lack sensitivity, and vice versa.

Significance and Claims

The paper argues that calibration is not sufficient for evaluating confidence estimators in real-world applications. The authors claim:

Overlooked Limitations: Current evaluations fail to detect that CE methods often produce unreliable confidence scores under language variations, specifically failing to distinguish between semantically different answers.
Guidance for Application: The choice of a CE method must be application-specific.
- User-facing QA systems may prioritize robustness to maintain consistent behavior under prompt variations.
- Multi-candidate reranking or LLM-as-a-judge settings require sensitivity to effectively distinguish between competing answers.
- Controlled environments with fixed prompts may still rely on traditional calibration metrics.
Future Direction: The observed lack of sensitivity suggests that CE methods must better leverage generation-side information (e.g., the full generated text or intermediate states) rather than relying solely on prompt-based or question-token representations.

The authors conclude that their framework provides a necessary, complementary lens for evaluating CE, exposing limitations that traditional metrics miss and offering practical guidance for selecting estimators based on specific deployment needs.