💻 computer science

On Verbalized Confidence Scores for LLMs

This paper investigates the reliability of Large Language Models verbalizing their own uncertainty via confidence scores, demonstrating that while effectiveness varies by prompting strategy, specific methods can yield well-calibrated, model-agnostic uncertainty quantification with low overhead.

Original authors: Daniel Yang, Yao-Hung Hubert Tsai, Makoto Yamada

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Yang, Yao-Hung Hubert Tsai, Makoto Yamada

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, well-read friend for advice. Sometimes they are absolutely sure about their answer, and sometimes they are just guessing. The problem with today's Large Language Models (LLMs) is that they often sound confident even when they are wrong. They are like a friend who never says, "I'm not sure," even when they have no idea what they are talking about.

This paper, "On Verbalized Confidence Scores for LLMs," investigates a simple idea: What if we just ask the AI to tell us how sure it is?

Instead of looking at the AI's internal code or running it a hundred times to see if it gives the same answer, the researchers simply asked the AI to add a "confidence score" to its response, like saying, "The answer is X, and I am 85% sure."

Here is a breakdown of their findings using everyday analogies:

1. The Goal: A "Trust Meter"

The researchers wanted to see if this "trust meter" (the confidence score) actually works. They wanted a system that:

Is Honest: If the AI says it's 90% sure, it should be right 90% of the time.
Is Flexible: It should work no matter what question you ask or which AI model you use.
Is Fast: It shouldn't take extra time or computing power to get the score.

2. The Experiment: Testing Different "Ways of Asking"

The team realized that how you ask the AI matters a lot. It's like asking a child for a guess.

If you ask, "What is the capital of France? Guess!" they might just say "Paris" and feel confident.
If you ask, "What is the capital of France? Guess, and tell me how sure you are on a scale of 0 to 100," they might think harder.

The researchers tested 17 different ways of phrasing the question (prompts) across 11 different AI models and 10 different types of questions (like science trivia, logic puzzles, and common sense).

3. The Big Discovery: It Depends on the AI's "Brain Size"

The most interesting finding is that the best way to get a honest answer depends on how "smart" (or large) the AI model is.

For "Tiny" AI Models (The Novices):
Think of these as students who are still learning. If you give them a complex, fancy prompt with lots of examples and instructions, they get confused and start making things up.
- The Paper's Finding: Simple prompts work best. Just asking them simply, "How sure are you?" works better than giving them a long list of rules.
For "Large" AI Models (The Experts):
Think of these as PhD graduates. They have a lot of knowledge but can be overconfident. If you just ask them simply, they might still guess with 100% confidence even when wrong.
- The Paper's Finding: They need a more complex "nudge." The researchers found that combining several techniques—like giving them a few examples of how to answer, asking them to explain their reasoning, and explicitly telling them to consider the difficulty of the task—made them much more honest.
- The Result: With the right complex prompt, these large models could produce confidence scores that were accurate within about 7% of their actual success rate. This is a huge improvement over just guessing.

4. The "Combo" Prompt

The researchers created a "super-prompt" (called combo) for the large models. It was like giving the AI a checklist:

Think about the question.
Consider how hard it is.
Look at your own knowledge.
Give your best guess.
Give a probability score (0.0 to 1.0).

When they used this "combo" prompt, the large models became much better at admitting when they were unsure.

5. What This Means (According to the Paper)

The paper concludes that asking an AI to "speak its confidence" is a very promising tool. It is:

Simple: You don't need to see inside the AI's brain.
Fast: It only takes a few extra words to generate.
Effective: If you know how to ask the right question, the AI can tell you when it's guessing and when it knows the answer.

However, there is a catch: You can't just ask any AI any way you want. You have to tailor the question. If you ask a small AI a complex question, it will fail. If you ask a large AI a simple question, it might lie about being confident. But with the right "recipe" for the prompt, we can get these models to be much more trustworthy.

In short: The AI can tell you how sure it is, but you have to know how to ask it the right way to get a truthful answer.

Technical Summary: On Verbalized Confidence Scores for LLMs

Problem Statement

The increasing integration of Large Language Models (LLMs) into daily life has created a critical need for trustworthiness mechanisms. Unlike traditional search engines or Q&A forums that utilize human voting or relevance ranking to indicate reliability, LLM responses often lack inherent trust indicators. Blind reliance on these systems poses risks, necessitating robust Uncertainty Quantification (UQ) methods.

Existing UQ approaches typically rely on:

Internal Knowledge: Sampling multiple responses to measure consistency or utilizing internal token logits. These methods often fail to be model-agnostic (logits are inaccessible in black-box models) or incur high computational overhead (sampling).
External Knowledge: Using proxy models or knowledge bases. These often lack generalization across different prompts and tasks.

The paper investigates verbalized confidence scores, a method where the LLM is prompted to explicitly output a confidence score alongside its answer. While this approach is prompt-agnostic, model-agnostic, and incurs low overhead, its reliability is contested. Previous studies have reported conflicting results regarding the calibration of these scores, suggesting that the method of prompting may be the deciding factor.

Methodology

The authors conduct a comprehensive empirical analysis to determine how dataset characteristics, model capacity, and prompt engineering affect the reliability of verbalized confidence scores.

Experimental Setup

Datasets: 10 diverse datasets (e.g., ARC, MMLU, TruthfulQA, CommonsenseQA) covering various domains (science, logic, social commonsense) and difficulty levels. The evaluation is restricted to closed-book, closed-ended, objective questions to ensure ground-truth correctness can be determined.
Models: 11 LLMs ranging from tiny open-source models (e.g., Gemma 2B, Llama 3 8B) to large models (e.g., Llama 3 70B, Qwen 110B, GPT-4o).
Prompt Methods: 17 distinct prompt variations categorized by:
- Score Range: Percentages (0-100), decimals (0-1), discrete letters (A-E), or text labels.
- Score Formulation: Asking for "confidence" vs. "probability of correctness."
- Advanced Description: Explicitly instructing the model to consider prompt uncertainty, task difficulty, and knowledge availability.
- Few-Shot Prompting: Providing 1 or 5 examples of answer-confidence pairs.
- Complexity: Variations including Chain-of-Thought (CoT) or ranking multiple guesses.

Evaluation Metrics

The reliability of the scores is assessed using three high-level criteria:

Calibration: Measured by Expected Calibration Error (ECE) and SmoothECE (smECE). A calibrated score $C$ implies $P(\text{correct} | C=c) = c$ .
Informativeness: Measured by the variance and number of distinct values in the predicted confidence scores. A model that always outputs the same score is uninformative.
Meaningfulness: Measured by the Kullback-Leibler (KL) divergence between the confidence distribution of a specific dataset and the global distribution. This assesses whether the model adjusts its confidence based on task difficulty.

Key Results

1. Dependence on Prompting Strategy

The study confirms that the reliability of verbalized confidence scores is not intrinsic to the model but heavily dependent on how the model is asked.

Disagreement Resolution: The paper resolves conflicting literature (e.g., Tian et al. vs. Xiong et al.) by showing that different prompt formulations yield vastly different calibration results.
Tiny vs. Large Models:
- Tiny LLMs: Benefit most from simple prompt formulations. Complex methods like few-shot prompting or ranking multiple guesses significantly degrade reliability. The "probability score" formulation (asking for a 0-1 probability) provided the best improvement.
- Large LLMs: Benefit from complex prompt methods. Combining probability formulation, advanced descriptions (explicitly mentioning uncertainty sources), and few-shot examples yielded the best results.

2. Calibration Performance

Overconfidence: Most models exhibit overconfidence, with confidence scores remaining high even as accuracy drops on difficult datasets.
Model Capacity: While accuracy increases with model size, calibration improvements are primarily driven by increased accuracy rather than a reduction in overconfidence.
Best Performance: For large LLMs, the "combo" method (probability formulation + advanced description + few-shot examples) achieved an average deviation of 7% from empirical accuracy (ECE $\approx$ 0.07), representing a significant qualitative improvement in calibration behavior.

3. Informativeness and Meaningfulness

Simple prompts often result in low variance (informativeness), with models clustering around high confidence.
Complex prompts for large models successfully encouraged the generation of diverse confidence scores that better reflected the difficulty of the underlying dataset (higher meaningfulness).

Key Contributions

Intuitive Uncertainty Decomposition: The authors propose a decomposition of LLM uncertainty into Input, Model, and Output uncertainty, focusing specifically on the output uncertainty regarding objective correctness.
Precise Reliability Specification: The paper defines and evaluates reliability through a triad of metrics: Calibration, Informativeness, and Meaningfulness, moving beyond simple accuracy checks.
Empirical Insights: The study provides a granular analysis of how dataset difficulty, model capacity, and specific prompt engineering techniques interact to determine the quality of verbalized confidence.
Open Source: The evaluation code and prompt templates are made publicly available to facilitate further research.

Significance and Claims

The paper argues that verbalized confidence scores represent a promising, simple, and versatile method for uncertainty quantification that satisfies the requirements of being prompt-agnostic, model-agnostic, and low-overhead.

However, the authors maintain a modest stance regarding the current state of the technology:

Not a Silver Bullet: The reliability of these scores is currently "greatly influenced by the way of asking." They are not yet fully reliable for real-world deployment without careful prompt engineering.
Future Potential: The authors suggest that while LLMs can be taught to express diverse and meaningful confidence scores through simple prompting, achieving full calibration and self-awareness may require further advancements in prompt optimization, LLM-guided engineering, or fine-tuning.
Scope Limitation: The current work is limited to objective correctness. It does not address subjective questions or open-ended generation tasks where "correctness" is ill-defined.

In conclusion, the paper establishes that verbalized confidence is a viable path forward for LLM trustworthiness, provided that the prompting strategy is carefully tailored to the specific model size and task complexity.