💬 NLP

Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning

This paper investigates how supervised fine-tuning degrades the correlation between confidence scores and output quality in language models, demonstrating that these metrics become unreliable off-the-shelf due to factors like training distribution similarity and highlighting the urgent need for more robust uncertainty quantification methods.

Original authors: Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot assistant. Before you ask it to do a specific job, you might ask it to translate a sentence, solve a math problem, or answer a trivia question.

Now, imagine this robot has a little "confidence meter" on its dashboard. When it gives you an answer, the meter lights up.

Green (High Confidence): "I'm 99% sure this is right!"
Red (Low Confidence): "I'm not really sure about this one, you might want to double-check."

This "confidence meter" is incredibly useful. It helps you decide when to trust the robot and when to intervene. But here is the problem this paper investigates: What happens when you teach the robot a new skill?

The "Training Camp" Effect

In the world of AI, teaching a robot a new skill is called Supervised Fine-Tuning (SFT). It's like sending your robot to a specialized training camp.

Before Camp: The robot is general. It knows a bit about everything. Its confidence meter is usually pretty honest.
After Camp: The robot is an expert in one specific area (like math or translation). It has practiced thousands of examples.

The researchers asked: "Does the robot's confidence meter stay honest after this training?"

The Big Discovery: The Meter Gets Broken

The answer is: Not always. In fact, the training often breaks the meter in two very specific, funny ways.

1. The "Overconfident Know-It-All" (Probability-Based Metrics)

Some confidence meters work by looking at how "smooth" the robot's words feel. If the words flow easily, the robot thinks, "I'm great at this!"

The Analogy: Imagine a student who memorized the answer key for a specific test. When they see a question that looks like the ones they practiced, they scream, "I know this!" even if they are actually wrong.
What happened in the study: After training, these robots became overconfident. They would give a wrong answer with a bright green "99% Sure" light. They got so used to the style of the training data that they forgot to check if the answer was actually correct. They confused "familiarity" with "correctness."

2. The "Humble Bragger" (Self-Consistency Metrics)

Other confidence meters work by asking the robot to think of the same answer five different times. If all five answers are the same, it's confident. If they are all different, it's unsure.

The Analogy: Imagine a student who is so nervous after studying that they second-guess themselves. Even when they know the answer is right, they think, "Hmm, maybe I'm wrong? Maybe I should change it?"
What happened in the study: These robots became underconfident. They would give a correct answer but light up the "Red" warning light, saying, "I'm not sure," even though they were right.

The "Training Data" Trap

Why does this happen? The paper found that the robots are sensitive to how much the question looks like the training data, not just whether the answer is right.

The Metaphor: Imagine a chef who only practiced cooking with tomatoes. If you ask them to cook a tomato dish, they are very confident. But if you ask them to cook a potato dish that looks like a tomato dish (maybe it's red), they might get confused.
The Reality: The robots' confidence scores were reacting to how "familiar" the question felt to their training, rather than the actual quality of the answer. This is why the meter gets broken: it's measuring familiarity, not truth.

Why Should You Care?

The researchers tested this on a real-world task: Question Answering.

They used the robots to decide which answers were good enough to show a human.

Before Training: The robot was good at filtering out bad answers.
After Training: The robot started letting bad answers through (because it was overconfident) or blocking good answers (because it was underconfident).

The Result: The robot became less useful. It failed to catch its own mistakes about 47% of the time after training.

The Takeaway

You cannot just take a robot, train it on a new job, and assume its "confidence meter" still works.

Don't trust the dashboard blindly: Just because the robot says "I'm sure!" doesn't mean it's right, especially if it just finished a training course.
Test before you trust: If you want to use an AI for something important, you have to check its confidence meter after you've taught it your specific task.
New tools needed: We need to invent new kinds of confidence meters that don't get confused by the training process. They need to measure "truth" instead of just "familiarity."

In short: Teaching a robot a new skill can make it either a cocky liar or a shy genius. You need to recalibrate its confidence meter every time you send it to training camp.

1. Problem Statement

Uncertainty Quantification (UQ) techniques are critical for measuring the confidence of Large Language Models (LLMs) to detect hallucinations, select high-quality outputs, and trigger human review. For these scores to be useful, they must be correlated with output quality (i.e., high confidence should align with high accuracy).

However, a significant gap exists in understanding how Supervised Fine-Tuning (SFT) affects this correlation. While off-the-shelf UQ metrics are designed to work with any white-box model, recent evidence suggests SFT can induce overconfidence. The paper investigates whether the correlation between confidence scores and output quality remains stable after SFT, or if it degrades due to factors unrelated to actual output quality (e.g., similarity to the training distribution).

2. Methodology

Experimental Setup

Models: BART-Base, Flan-T5-Base, Llama 3.1-8B, and Gemma-2-2B.
Tasks:
- Translation: English-Afrikaans (NLLB dataset, FLORES test set).
- Question Answering (QA): SQUAD dataset.
- Math: GSM8K dataset.
Fine-Tuning: Models were fine-tuned with varying numbers of epochs (up to 10) and sample sizes (100 to 2000). Early stopping was employed based on validation metrics.
Evaluation: The primary metric for success was the Spearman correlation ( $\rho$ ) between the confidence score and the output quality (measured via ChrF+, F1, or Exact Match).

Confidence Metrics Analyzed

The study focused on two categories of metrics that do not require additional training:

Probability-Based: Rely on token-level probabilities.
- Average Token Log Probability (Avg Tok Prob).
- Average Token Entropy (Avg Tok Ent).
- Dropout Entropy (DO Ent).
- Beam Search metrics (Weighted Average, Ratios, Sums).
Self-Consistency-Based: Rely on the variance of multiple sampled outputs (via dropout).
- Variance in BLEU, KL Divergence, and METEOR scores across dropout samples.
- CoCoA: A hybrid metric combining probability and consistency.

Analytical Framework

To understand why correlations change, the authors introduced a sample-level analysis framework:

Concordance: Confidence and Quality move in the same direction.
Relative Overconfidence: Confidence increases while Quality decreases.
Relative Underconfidence: Confidence decreases while Quality increases.
Relative Miscorrelation: Defined as a pair of samples $(A, B)$ where the relative ranking of quality and confidence flips after an epoch of training (e.g., $A$ was better than $B$ , but after training, the model is more confident in $B$ ).

3. Key Contributions & Results

RQ1: Sensitivity of Correlation to SFT Parameters

The study found that confidence metrics are highly sensitive to SFT parameters (epochs and sample size).

Degradation: In 33.3% (48/144) of task-model-metric configurations, the correlation between confidence and quality degraded after SFT.
Volatility: Correlation can fluctuate significantly even within a single training epoch (changes up to 0.391 Spearman points observed).
Sample Size Impact: The number of training samples significantly alters the correlation, even when controlling for the number of training steps.
Conclusion: Metrics cannot be used "off-the-shelf" after fine-tuning; they require re-validation for specific SFT configurations.

RQ2: Underlying Dynamics of Miscorrelation

The authors identified distinct behavioral patterns for different metric types:

Probability-Based Metrics (Overconfidence): SFT tends to make these metrics relatively overconfident. The average log probabilities shift upward across most outputs regardless of whether the output quality actually improved. This suggests the model becomes "confident" simply because the output resembles the training distribution, not because it is correct.
Consistency-Based Metrics (Underconfidence): These metrics tend to become relatively underconfident post-SFT.
Mechanism of Miscorrelation: The primary cause of degraded correlation is Case 1 Miscorrelation: The relative ranking of confidence scores flips between two samples, even though their relative quality ranking remains the same.
- Root Cause: Confidence scores are influenced by factors other than quality, specifically the similarity of the output to the training distribution. The paper demonstrates a statistically significant positive correlation between confidence scores and the cosine similarity of samples to the training data.

Downstream Impact (Case Study)

To demonstrate practical consequences, the authors applied confidence scores to a TruthfulQA task to filter correct answers.

Result: In 47% of cases, the ability of the confidence score to identify correct answers (AUROC) decreased after SFT.
Implication: Relying on uncalibrated confidence scores post-SFT can lead to filtering out correct answers or keeping incorrect ones, reducing the utility of the model in safety-critical or high-stakes applications.

4. Significance and Implications

Rejection of "Off-the-Shelf" Usage: The paper strongly argues that UQ metrics cannot be blindly applied to fine-tuned models. The correlation between confidence and quality is not an invariant property of the model architecture but is highly dependent on the fine-tuning process.
Diagnosis of Overconfidence: It provides a mechanistic explanation for why LLMs become overconfident after SFT: they learn to assign high probabilities to outputs that look like the training data, even if those outputs are factually incorrect for the specific query.
Future Directions:
- Robust Metrics: Future UQ metrics must be designed to be robust to SFT, potentially by decoupling confidence from training distribution similarity.
- Reporting Standards: Researchers should report the consistency of confidence metrics across different fine-tuning stages rather than just pre-training or single-point evaluations.
- Interventions: The findings motivate the development of calibration techniques that specifically target the "relative overconfidence" induced by SFT.

Summary

This paper serves as a critical warning to the NLP community: Fine-tuning fundamentally alters the relationship between a model's internal confidence and its actual correctness. Without rigorous testing and potential recalibration, confidence scores derived from fine-tuned models may be misleading, leading to poor downstream decision-making.