Technical Summary: Confidence Calibration in Large Language Models

Problem Statement

The utility of Large Language Models (LLMs) as guides, teachers, and advisers is fundamentally undermined by "hallucination"—the confident reporting of falsehoods. While an ideal LLM would provide only truthful information, irreducible uncertainties and limited access to ground truth make this unrealistic. A more attainable goal is well-calibrated confidence, where an LLM faithfully reports the probability of its correctness conditional on its limitations. Without such calibration, users cannot distinguish between high-confidence accurate outputs and high-confidence errors, creating significant epistemic risks.

Existing literature indicates that human judgment is plagued by overconfidence and a "hard-easy" effect (overconfidence on difficult tasks, underconfidence on easy ones). It remains unclear whether LLMs exhibit similar biases or if their confidence signals are merely noisy indicators of accuracy, as observed in deep neural networks. Furthermore, prior methods for assessing task difficulty in calibration studies rely on subjective human assessment, LLM-as-a-judge, or context scaling, all of which introduce bias or fail to provide a continuous measure of difficulty.

Methodology

The authors conducted a preregistered study evaluating the confidence calibration of 11 popular open- and closed-source LLMs across five distinct reasoning tasks and a newly developed test.

Models Evaluated

The study compared five "reasoning" models (DeepSeek-R1, Gemini 2.5 Pro, GPT-o3, Claude Sonnet 4, Claude Sonnet 3.7) against six "chat" models (DeepSeek-V3, Gemini 2.5 Flash, GPT-4o, Claude Haiku 3, Llama 3.1 8B, Llama 3.1 70B).

Question Sets

Five established datasets were used to test calibration across different domains:

1. BoolQ: True/False trivia questions.
2. SciQ: Scientific multiple-choice questions.
3. LSAT-AR: Logical reasoning requiring multi-step inference.
4. SAT-EN: Reading comprehension requiring nuanced textual inference.
5. HaluEval: A hallucination detection task where models evaluated pre-generated answers (half truthful, half hallucinated).

LifeEval: A New Benchmark

To isolate the effect of task difficulty from other variables, the authors developed LifeEval. This task asks models to predict a person's age at death given their sex and minimum age, and to state the probability that their estimate falls within a specific radius (1, 5, 10, or 20 years) of the true value.

• Difficulty Manipulation: Difficulty is controlled exogenously by varying the radius and the minimum age, holding the nature of the question constant.
• Ground Truth: Actual probabilities were calculated using U.S. Social Security Administration Period Life Tables, allowing for a precise comparison between stated confidence and empirical probability.
• Metric: Task difficulty is defined as $1 - \text{Maximum Achievable Score (MAS)}$, where MAS is the theoretical ceiling of accuracy for a given question.

Experimental Protocol

• Prompting: Models were instructed to output JSON with their answer and a numerical confidence score (0.0 to 1.0). Chain-of-thought prompting was used for most tasks.
• Scoring: For multiple-choice questions, a response was correct if the option with the highest assigned probability matched the ground truth.
• Metrics:
  • Expected Calibration Error (ECE): Measures the misalignment between predicted confidence and empirical accuracy across bins.
  • Overconfidence: Calculated as $\text{Mean Confidence} - \text{Mean Accuracy}$.
  • Hard-Easy Effect: Measured via the regression coefficient between task difficulty and overconfidence.

Key Results

1. General Overconfidence

Across all question sets, LLMs are, on average, overconfident. Models reported an average confidence of 88% in their favored answers, while their actual accuracy was 79%. This 9% gap indicates a systematic tendency to be "too sure they are right."

2. The Hard-Easy Effect

The study confirmed that the hard-easy effect, well-documented in human psychology, also exists in LLMs:

• Difficult Tasks: Overconfidence is greatest on difficult tasks (e.g., LSAT-AR, HaluEval).
• Easy Tasks: Models exhibit substantial underconfidence on easier tasks (e.g., SciQ, SAT-EN).
• LifeEval Findings: As task difficulty increased (smaller radius), overconfidence increased. Conversely, as tasks became easier (larger radius), models became underconfident. For the most difficult LifeEval tasks (1-year radius), models reported 34.2% confidence against a 9.6% actual probability. For the easiest (20-year radius), they reported 80.5% confidence against a 92.0% actual probability.

3. Reasoning vs. Chat Models

• Nuance: Reasoning models provided more nuanced confidence estimates. Only 61.1% of their confidence scores were rounded to multiples of 5%, compared to 91.4% for chat models.
• Correlation: Reasoning models showed a stronger correlation (average 0.94) between stated confidence and actual probability compared to chat models (0.48).
• Calibration: While reasoning models were generally better calibrated, they still exhibited the hard-easy effect.

4. Contamination Analysis

A post-hoc analysis of LifeEval revealed that some models (notably DeepSeek-R1, Gemini 2.5-Pro, and GPT-o3) showed high rates of "strong evidence" of having memorized the underlying SSA life tables used to generate the ground truth. When restricting the analysis to responses with no evidence of memorization, the hard-easy effect and overconfidence trends persisted, confirming that the findings are not solely artifacts of data leakage.

Key Contributions

1. Empirical Evidence of Hard-Easy Effect in AI: The paper provides the first systematic documentation of the hard-easy effect in LLM confidence, demonstrating that overconfidence rises with difficulty and underconfidence emerges on easy tasks.
2. LifeEval Benchmark: The introduction of LifeEval offers a novel framework for evaluating model calibration with a continuous, exogenous measure of task difficulty, overcoming the subjectivity of prior difficulty assessment methods.
3. Differentiation of Model Types: The study highlights that "reasoning" models offer more nuanced and better-calibrated confidence estimates than standard "chat" models, though neither is perfectly calibrated.
4. Quantification of Overconfidence: The study quantifies the average overconfidence gap (9%) and identifies specific tasks (logical reasoning, hallucination detection) where models struggle to recognize their own fallibility.

Significance and Claims

The paper claims that the presence of the hard-easy effect in AI confidence mirrors human judgment, suggesting that confidence in LLMs may be a "noisy signal of accuracy" similar to human cognition. This has critical implications for the deployment of autonomous systems:

• Trust and Safety: If users cannot trust a model's stated confidence to indicate accuracy, they may either trust faulty outputs or doubt accurate ones.
• Limitations of RLHF: The authors note that Reinforcement Learning with Human Feedback (RLHF) might inadvertently increase overconfidence if human users prefer models that express confident assurance.
• Path Forward: The authors suggest that strategies effective for human debiasing—such as encouraging reflection on why an answer might be wrong—could be adapted for LLMs. The superior performance of reasoning models suggests that prompts or training regimens encouraging self-criticism and reflection may improve calibration.

The paper concludes that while LLMs are improving, their confidence remains insufficiently sensitive to task difficulty, posing a barrier to reliable autonomous action. Future research should focus on distinguishing between forms of overconfidence (overestimation vs. overprecision) and developing methods to mitigate these biases.