💬 NLP

Generalization of Fine-Tuned Uncertainty Communication and Metacognition in Large Language Models

This paper demonstrates that supervised fine-tuning can improve large language models' ability to communicate uncertainty and align confidence with accuracy, though these metacognitive gains generalize better across domains than across different confidence assessment formats, with multitask training offering the most robust generalization.

Original authors: Mark Steyvers, Catarina Belem, Padhraic Smyth

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mark Steyvers, Catarina Belem, Padhraic Smyth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot friend who can answer almost any question you ask. The problem is, this robot is sometimes wrong, but it never admits it. It answers with the same booming, confident voice whether it's talking about the capital of France or a made-up fact about space aliens. This is dangerous because if you trust its confident voice, you might make bad decisions.

This paper asks: Can we teach this robot to say, "I'm not sure," or "I'm pretty confident," in a way that actually matches how right it is?

The researchers tried to "train" the robot (specifically, two popular AI models) to be better at this self-awareness. Here is what they found, explained simply:

1. The Two Ways to Test "Self-Awareness"

To see if the robot was learning, the researchers used two different games:

The "Confidence Score" Game: The robot answers a question and gives a number from 0 to 100% saying how sure it is.
- The Goal: If the robot says "90% sure," it should be right 90% of the time.
The "Which One?" Game: The robot is shown two questions. It has to pick the one it thinks it can answer correctly.
- The Goal: It should pick the question it knows the answer to, and skip the one it's guessing on.

2. The Training Method: "Practice Makes Perfect"

The researchers didn't just tell the robot, "Be more honest." Instead, they gave it a massive amount of practice problems.

They asked the robot the same question 10 times.
If the robot gave the same answer 9 out of 10 times, they taught it: "You are consistent, so you should be highly confident."
If the robot gave 10 different answers, they taught it: "You are confused, so you should be low confidence."
They then asked the robot to predict its own confidence based on this pattern.

3. The Good News: It Learned to Be Honest (Mostly)

After this training, the robot got much better at matching its confidence to its actual accuracy.

Before training: The robot was like a overconfident student who always got an "A" on a test but actually only knew half the material. It said "95% sure" even when it was wrong.
After training: The robot became more like a cautious expert. When it knew the answer, it said "90% sure." When it was guessing, it said "40% sure."
The Magic: This skill didn't just stay in the subjects it practiced on (like math or trivia). It also helped the robot be more honest about new, difficult topics it hadn't seen before, like medical advice and legal questions. This is a big deal because it means the robot can be trusted more in serious situations, even if it hasn't been specifically trained on those specific cases.

4. The Bad News: Skills Don't Always Transfer

Here is where it gets tricky. The researchers found that being good at one game didn't automatically make the robot good at the other.

The "Score" vs. The "Choice": If they trained the robot to give a number (e.g., "75%"), it got really good at giving numbers. But when they asked it to play the "Which One?" game, it didn't get much better.
The Reverse: If they trained it to play the "Which One?" game, it got good at picking the right question, but it still couldn't give a good number when asked.

The Analogy: Imagine a basketball player who is great at shooting free throws (giving a number). You train them specifically on free throws. They get amazing at free throws. But if you ask them to play defense (picking the right opponent to guard), they aren't any better at it. The skills are related, but they are different muscles.

5. The Solution: The "All-Rounder" Training

The researchers tried one more thing: they trained the robot on both games at the same time.

The Result: This worked much better. The robot learned a kind of "general self-awareness" that helped it in both the number game and the choice game. It became a more flexible, reliable thinker across the board.

The Bottom Line

Can we teach AI to know when it's guessing? Yes.
Does it work on new topics? Yes, it helps the AI be more honest about medical and legal questions, even if it was mostly trained on trivia and math.
Is it perfect? No. Teaching an AI to give a confidence score doesn't automatically teach it to compare two questions, and vice versa.
The Fix: To get the best results, you have to train the AI on multiple types of "self-checking" tasks at once.

In short, the paper shows that AI can learn to be a better judge of its own knowledge, but it needs a varied diet of training to do so effectively. It's not just about memorizing facts; it's about learning how to say, "I know this," or "I'm not sure," in a way that humans can actually trust.

Technical Summary: Generalization of Fine-Tuned Uncertainty Communication and Metacognition in Large Language Models

Problem Statement

Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as law, medicine, and education. A critical limitation in these settings is that models often generate factually incorrect information with high confidence, misleading users who lack clear signals of uncertainty. While LLMs may maintain internal uncertainty signals (e.g., token likelihoods or sampling consistency), they frequently fail to express these signals explicitly in a calibrated, interpretable manner. Current explicit confidence reporting is often poorly aligned with answer correctness, and implicit measures (like sampling-based consistency) require access to internal probabilities or incur high computational costs. The central problem addressed is whether supervised fine-tuning can improve an LLM's ability to communicate uncertainty and whether these metacognitive improvements generalize across different knowledge domains and distinct metacognitive task formats.

Methodology

Models and Datasets

The study utilized two representative models: GPT-4.1-mini and Llama3.1-70B-Instruct.

Training Domains: Three datasets were used for fine-tuning: MMLU-PRO (expert-level multiple-choice), GSM8K (grade-school math), and TriviaQA (short-answer general knowledge). These were used individually and in combination (M+G+T).
Out-of-Domain Evaluation: Three datasets reserved exclusively for testing generalization: TruthfulQA (misconceptions), MetaMedQA (medical reasoning), and LegalBench (legal reasoning).

Metacognitive Tasks

Two complementary tasks were evaluated:

Single-Question Confidence Estimation: The model provides a numeric confidence score (0–1) alongside its answer. Performance is measured by Calibration (Expected Calibration Error, ECE) and Discrimination (Area Under the Curve, AUC, and answer-level meta-information, $meta-I^2_{r,ans}$ ).
Pairwise Confidence Comparison: Given two questions, the model selects the one it is more likely to answer correctly. Performance is measured by AUCc (alignment with reference consistency scores) and AUCa (alignment with actual answer correctness).

Fine-Tuning Procedure

The authors employed supervised fine-tuning using consistency-based uncertainty signals as training targets.

Target Generation: For each training question, $N=10$ independent samples were generated. A consistency score was calculated based on the proportion of samples matching the modal answer.
Calibration: For single-question tasks, consistency scores were mapped to empirical accuracy (factual calibration) to create target confidence values. Small amounts of noise were added to these targets to prevent the model from memorizing discrete bins and to encourage smoother reporting.
Training Variants: Models were fine-tuned on:
- Single-question confidence (S).
- Pairwise comparison (C).
- Both tasks jointly (C+S).

Evaluation Metrics

Calibration (ECE): Measures the gap between stated confidence and observed accuracy.
Discrimination (AUC): Measures the probability that a correct answer receives higher confidence than an incorrect one.
Meta-Information: Quantifies the reduction in uncertainty about answer correctness given the model's confidence bin.

Key Contributions and Results

1. Improvement Within Training Domains

Supervised fine-tuning significantly improved both calibration and discrimination within the domains used for training.

Calibration: Fine-tuning reduced ECE substantially (e.g., $\Delta ECE = -0.57$ for GPT-4.1-mini on within-domain tests). Models shifted from overconfident, clustered predictions to a wider range of confidence values aligned with empirical accuracy.
Discrimination: AUC increased significantly (e.g., $\Delta AUC = 0.17$ ), indicating the models became better at assigning higher confidence to correct answers and lower confidence to incorrect ones.
Accuracy: Crucially, overall answer accuracy remained stable ( $\Delta Accuracy \approx 0$ ), confirming that gains in discrimination were due to better uncertainty estimation rather than improved factual correctness.

2. Generalization Across Domains

Models fine-tuned on the combined dataset (M+G+T) showed improved calibration and discrimination when evaluated on out-of-domain benchmarks (TruthfulQA, MetaMedQA, LegalBench).

While gains were smaller than within-domain improvements, they were statistically significant ( $\Delta AUC = 0.05$ , $\Delta ECE = -0.03$ ).
This suggests that metacognitive skills regarding uncertainty communication can transfer to unfamiliar content and answer formats, including complex medical and legal reasoning tasks.

3. Limited Transfer Across Metacognitive Tasks

A key finding is the lack of bidirectional transfer between the two metacognitive tasks when trained in isolation.

Models fine-tuned only on single-question confidence did not improve in pairwise comparison tasks.
Models fine-tuned only on pairwise comparison did not improve in single-question confidence estimation.
This indicates that absolute estimation and relative ranking are learned as distinct metacognitive routines rather than different expressions of a single underlying uncertainty signal.

4. Benefits of Multitask Fine-Tuning

Multitask fine-tuning (C+S) produced broader generalization than single-task training.

Models trained on both tasks simultaneously showed superior performance in cross-domain evaluations compared to single-task models.
For example, on MetaMedQA, the multitask model achieved higher AUC and lower ECE than the single-task models.
This suggests that joint training encourages the development of partially shared internal representations of uncertainty, bridging the gap between isolated metacognitive routines.

5. Model Architecture Differences

While both GPT-4.1-mini and Llama3.1-70B showed similar trends, there were nuances. Llama-3.1-70B benefited from multitask training for single-question tasks but showed a slight decline in pairwise comparison performance under multitask fine-tuning compared to single-task comparison training. This implies that the ability to form shared uncertainty representations via multitask training may vary across model architectures.

6. Ablation Studies

Factual Calibration: Including the step to map consistency scores to empirical accuracy significantly improved calibration (reducing overconfidence) but did not significantly change discrimination (AUC).
Noise Injection: Adding noise to target confidence values did not change aggregate performance metrics but significantly increased the diversity of confidence values reported by the model, preventing the output of only discrete, round numbers.

Significance and Claims

The paper claims that uncertainty communication in LLMs is trainable but that transfer across metacognitive tasks is limited without specific intervention.

Trainability: Supervised fine-tuning using consistency-based signals effectively aligns stated confidence with observed accuracy, addressing the risk of overconfident errors in high-stakes applications.
Domain Generalization: These improvements transfer to unseen domains (medical, legal), suggesting that uncertainty communication is not strictly bound to specific knowledge bases.
Task Specificity: The study challenges the assumption that metacognitive skills are monolithic. The failure of single-task training to transfer to other metacognitive formats suggests that LLMs learn distinct "routines" for different types of uncertainty assessment.
Multitask Solution: The paper posits that multitask fine-tuning is a viable strategy to overcome these limitations, fostering shared uncertainty representations that support broader generalization.

The authors conclude that while their findings are promising, they are limited to two model families and specific tasks. They call for further testing across diverse model families and a broader set of metacognitive tasks (e.g., abstention, error detection) to fully understand the generalizability of these shared uncertainty representations.