Technical Summary: Large Language Models Are Overconfident in Their Own Responses

Problem Statement

While base (pre-trained) Large Language Models (LLMs) generally exhibit well-calibrated probabilities for next-token prediction, instruction-tuned (post-trained) LLMs suffer from significant miscalibration. They tend to be overconfident, assigning high confidence scores to incorrect answers. Prior research has established that post-training processes (Supervised Fine-Tuning and Reinforcement Learning from Human Feedback) degrade calibration, yet the specific contribution of the chat template—the conversational format used to interact with these models—remains under-explored. Furthermore, it is unclear whether the miscalibration stems from the training algorithm itself or the conversational prompting style, and whether specific confidence elicitation methods can mitigate this issue without retraining.

Methodology

The authors conducted a comprehensive investigation across six recent open-weight LLM families (Llama 3.1, Qwen3, and Gemma 3), utilizing both base and instruction-tuned versions of varying scales (4B to 70B). The evaluation was performed on the MMLU benchmark, a dataset of 57 multiple-choice disciplines.

The research proceeded in four distinct phases:

1. Decoupling Training and Format: The authors isolated the effects of instruction tuning and the chat template by comparing three model variants:

   • Base models (pre-trained, no instruction tuning).
   • Instruction-tuned models invoked without the chat template.
   • Instruction-tuned models invoked with the standard chat template.
     Calibration was measured using Expected Calibration Error (ECE) and Brier Score, derived directly from model logits.

2. Eliciting Explicit Confidence: To determine if the miscalibration persists when models are explicitly asked for their certainty, the authors tested three confidence elicitation methods:

   • P(True): Asking the model to predict the probability of a "True" token for a given answer.
   • Verbalized Percentage: Asking for a 0–100% confidence score.
   • Verbalized Linguistic: Asking the model to select a qualitative confidence level from a Likert scale (e.g., "very low" to "very high").

3. Testing the "Ownership Bias" Hypothesis: The authors hypothesized that LLMs exhibit an "ownership bias," being significantly more confident in answers they generate themselves compared to identical answers provided by a user. To test this, they prompted models with the same question and answer pair, varying only the source of the answer:

   • Assistant Condition: The answer was presented as part of the model's own response.
   • User Condition: The answer was presented as part of the user's input.
     The model was then asked to evaluate the correctness of the provided answer.

4. Proposed Mitigation Strategy: Based on the findings, the authors proposed an inference-time strategy: framing the model's own answer as user input during confidence elicitation to bypass the ownership bias.

Key Results

1. Drivers of Miscalibration

• Instruction Tuning: Instruction tuning fundamentally harms calibration. On average, it increased ECE by 13.1% and the Brier score by 6.5% compared to base models, despite improving accuracy by 3.7%.
• Chat Template: The chat template further aggravates the issue. While it provided a modest accuracy gain (+1.1%), it increased ECE by an additional 2.74% and the Brier score by 1.5%.
• Conclusion: While instruction tuning is the primary source of miscalibration, the conversational chat format significantly exacerbates the problem.

2. Confidence Elicitation Methods

Explicitly asking for confidence did not resolve the underlying miscalibration. Instruction-tuned models remained significantly worse calibrated than base models across all three elicitation methods (P(True), Verbalized Percentage, and Verbalized Linguistic), confirming that the post-training process is the root cause.

3. Evidence of Ownership Bias

The experiments confirmed a strong "ownership bias":

• Overconfidence: Models assigned significantly higher confidence to answers they generated (Assistant condition) compared to identical answers provided by the user (User condition).
• Magnitude: The difference in confidence was substantial. For the Verbalized Linguistic method, models showed an average increase in confidence of 26.8% when the answer was their own. Even for the P(True) method, the average increase was 15.8%.
• Calibration Impact: This bias led to worse calibration metrics. The average increase in ECE for assistant-generated answers was 9.8% (P(True)) and up to 26.1% (Linguistic) compared to user-provided answers.
• Statistical Significance: These differences were statistically significant (p < 0.01) across the vast majority of models and methods.

4. Effectiveness of the Proposed Strategy

By reframing the model's answer as user input during confidence elicitation, the authors demonstrated that:

• Overconfidence Reduction: The strategy significantly reduces the model's inherent overconfidence.
• Calibration Improvement: This approach improves calibration by up to 26% without any retraining.
• Gap Narrowing: It effectively narrows the calibration gap between base and instruction-tuned models, sometimes allowing instruction-tuned models to achieve calibration comparable to or even surpassing that of base models in specific configurations.

Significance and Claims

The paper claims to identify a fundamental mechanism behind the miscalibration of instruction-tuned LLMs: an ownership bias where models are inherently overconfident in their own outputs. This finding contrasts with "sycophancy" (where models align with user beliefs), suggesting instead a form of self-consistency where the model assumes its own generation implies correctness.

The primary contribution is a simple, inference-time strategy that requires no retraining, complex reward modeling, or auxiliary models. By simply altering the prompt structure to present the model's answer as a user query, practitioners can significantly improve the reliability of confidence estimates. This offers a practical, low-cost solution for deploying safer and more trustworthy AI in high-stakes applications where accurate uncertainty quantification is critical.