💬 NLP

Human-Alignment, Calibration, and Activation Patterns in Large Language Model Uncertainty

This paper investigates the extent to which large language models exhibit human-like uncertainty signals in both their overt behavior and internal activations, examining the relationship between this "uncertainty alignment" and traditional calibration metrics across various datasets and the impact of instruction fine-tuning.

Original authors: Kyle Moore, Jesse Roberts, Daryl Watson, William Ward, Grayson Heyboer

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kyle Moore, Jesse Roberts, Daryl Watson, William Ward, Grayson Heyboer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game of "Guess the Answer" with a very smart, but sometimes overconfident, robot friend. You ask it a question, and it gives you an answer. Sometimes, it's right. Sometimes, it's wrong. But here's the tricky part: How does the robot know when it's unsure?

This paper is like a detective story investigating that exact question. The researchers wanted to know: Do Large Language Models (LLMs) feel "uncertainty" the same way humans do? And if they do, can we see it in their "brain" (internal code) or just in their "mouth" (the words they type)?

Here is the breakdown of their investigation, using simple analogies:

1. The Big Question: Are Robots Like Us?

When humans are unsure about an answer, we might hesitate, say "I think it's this, but I'm not 100% sure," or take longer to answer. We have a natural "uncertainty signal."

The researchers asked: Do AI models have this same signal?

The Analogy: Imagine a human and a robot taking a quiz together. When the human doesn't know the answer, they might scratch their head. The researchers wanted to see if the robot "scratches its head" (shows internal confusion) or if it just confidently guesses wrong (hallucinates).

2. The Two Main Tests

The researchers looked at two specific things:

Calibration (The "Honesty" Test): This asks, "When the robot says it's 80% sure, is it actually right 80% of the time?"
- Analogy: If a weather app says "80% chance of rain," does it actually rain 8 out of 10 times? If yes, it's well-calibrated.
Alignment (The "Human-Like" Test): This asks, "Does the robot get confused about the same questions that humans get confused about?"
- Analogy: If a human finds a riddle hard and takes a long time to solve it, does the robot also struggle with that same riddle? If they both struggle with the same things, they are "aligned."

3. The Experiment: 30 Robots, Many Questions

The team tested 30 different AI models (like LLaMa, Mistral, and Gemma) of various sizes. They asked them questions from four different "quiz books":

Multiple Choice: Standard trivia.
Open-Ended: "Tell me a story" or "What is the capital?" without options.
Human Data: Crucially, they had data on how real humans answered these same questions. They knew which questions humans found hard (based on how many people got it wrong or how long they took to answer).

4. The Big Discoveries

A. The "Instruct" Trap

Many AI models come in two versions:

Base Model: The raw, untrained version.
Instruct Model: The version trained to follow human instructions and chat nicely.

The Finding: The researchers found that training the robot to be a "good student" (Instruct) actually made it worse at knowing when it was unsure.

The Metaphor: Imagine a student who is naturally good at math but gets nervous when a teacher tells them to "act like a teacher." The "Instruct" training made the robots more confident, but less honest about their confusion. They became "overconfident" and less like humans when they were unsure.

B. The "Brain" vs. The "Mouth"

This is the most surprising part. The researchers looked inside the AI's "brain" (its internal electrical signals, called activations) and compared it to what the AI actually said (its output).

The Finding: The AI's "brain" showed much stronger signs of human-like uncertainty than its "mouth" did.

The Metaphor: Imagine a person who is terrified of a spider but tries to act cool and says, "I'm fine."
- The Mouth (Output): Says "I'm fine." (Looks confident).
- The Brain (Activations): Heart is racing, pupils are dilated. (Actually scared).
- The researchers found that the AI's "heart was racing" (internal signals) much more clearly showed when it was confused, even if its "voice" sounded confident. The human-like uncertainty was hidden deep inside the code, not in the final answer.

C. Group vs. Individual

The researchers also found that the AI was better at mimicking groups of people than individual people.

The Metaphor: The AI is good at knowing, "On average, people find this question hard." But it's not as good at knowing, "This specific person is confused right now."

5. What Does This Mean? (Strictly from the paper)

The paper concludes that:

AI uncertainty is real but weak: It exists, but it's not a perfect mirror of human uncertainty.
Training hurts honesty: Making AI models follow instructions (Instruct Fine-Tuning) seems to break their ability to signal uncertainty, both in how they act and how they are calibrated.
Look inside the box: If you want to know if an AI is truly unsure, you can't just listen to what it says. You have to look at its internal "brain waves" (activations), where the human-like signals are much stronger.

In short: The paper suggests that while AI models aren't perfect mirrors of human doubt, they do have a "hidden nervous system" that reacts to confusion similarly to humans. However, the more we train them to be helpful chatbots, the more they seem to hide that nervousness behind a mask of confidence.

Technical Summary: Human-Alignment, Calibration, and Activation Patterns in Large Language Model Uncertainty

Problem Statement

Uncertainty Quantification (UQ) in Large Language Models (LLMs) has primarily focused on calibration—ensuring that a model's confidence scores accurately reflect its task efficacy to mitigate hallucinations. However, a critical, underexplored question remains: how similar is LLM uncertainty behavior to human uncertainty?

Understanding this "human alignment" is vital for human-AI collaboration. If LLMs exhibit uncertainty patterns similar to humans, users may more safely transfer intuitive expectations from human-human interactions to human-AI interactions. Conversely, significant dissimilarities could allow humans and AI to compensate for each other's weaknesses. This work investigates the presence and strength of human-similar uncertainty signals (alignment) in both overt model behavior (logits) and internal activation patterns, while also examining the simultaneous occurrence of alignment and calibration.

Methodology

Models and Datasets

The study evaluated 30 open-weight text-only LLMs across four families (LLaMa, Mistral, Gemma, Falcon) and sizes ranging from 1B to 13B parameters. Both base and instruct-finetuned versions were tested.
Four datasets were utilized:

MMLU: A standard multiple-choice QA (MCQA) dataset used as a baseline for Expected Calibration Error (ECE) but lacking human uncertainty data.
ProtoQA: Crowd-sourced preference-based questions (8,724 items) providing human response distributions.
CamChoice: Reading comprehension MCQA (504 items) with human response distributions.
Coane: A dataset of 421 general knowledge questions presented to older adults. Crucially, it provides:
- Human response distributions (for group uncertainty/entropy).
- Average response times (for individual uncertainty, based on cognitive science interpretations).
- Both MCQA and Open-Ended QA (OEQA) variants.

Uncertainty Measures

The authors employed a broad suite of inference-time UQ measures calculated from logits and activations:

MCQA Measures: Full distribution metrics (top-1 probability, total entropy), cloze-based metrics, top-k/p entropies, and nucleus sizes.
Free Response (FR) Measures: Sequence perplexity (PPL) and averages of entropy/size metrics over generated tokens.
Bracketed FR (BFR) & 1st Token FR (1TFR): Measures restricted to the segment before a closing brace or the first generated token, respectively, to isolate the answer from potential over-generation.

Experimental Design

Alignment: Measured via Spearman correlation between model uncertainty signals and human uncertainty (derived from response entropy or response time).
Calibration: Measured using ECESweep, a dynamic binning method that minimizes bias in Expected Calibration Error estimation.
Activation Probing: Linear regression models were trained on layer-wise last-token activations to predict human uncertainty. This allowed the authors to map where and how alignment signals manifest internally compared to output logits.

Key Contributions and Results

1. Weak-to-Moderate Positive Alignment

The study found that LLM uncertainty is generally weakly-to-moderately positively correlated with human uncertainty. This relationship is mediated by the specific task and model architecture.

Group vs. Individual: Alignment is more strongly detectable when comparing model uncertainty to group-level human uncertainty (response distribution entropy) rather than individual-level uncertainty (response time).
Task Dependency: Alignment signals were sporadic in the CamChoice dataset and nearly non-existent in OEQA tasks, with the exception of the Falcon 3 10B model (which showed correlation with response time but not in BFR/1TFR contexts).

2. Degradation by Instruct Finetuning

A primary finding is that instruct finetuning significantly degrades both uncertainty alignment and calibration.

Across nearly every measure and dataset, base models demonstrated stronger alignment with human uncertainty than their instruct-finetuned counterparts.
Similarly, calibration (ECE) was generally worse in instruct models, with notable exceptions where finetuning improved calibration in limited-option contexts but degraded it in others.

3. Internal Activations vs. Overt Behavior

The study reveals a critical discrepancy between internal states and output behavior:

Stronger Internal Signals: Human-aligned uncertainty is more strongly detectable in internal activations than in output logits.
Quantitative Gap: For example, the Gemma 2 9B model showed a maximum correlation of $r \approx 0.6$ with human entropy in internal probes, compared to only $r = 0.32$ for logit-based measures.
Layer Dynamics: Group-level uncertainty (entropy) alignment tends to increase with depth in the network, whereas individual-level (response time) alignment remains relatively flat across layers.

4. Calibration Superiority

Despite alignment challenges, LLMs tend to be more well-calibrated than human subjects across most UQ measures. Human calibration errors on the Coane dataset ranged from 0.21 to 0.45, whereas many models achieved ECE values below 0.1.

Significance and Claims

The paper claims to provide the first extensive investigation into uncertainty alignment across a diverse set of models and tasks, moving beyond the traditional focus on calibration alone.

Behavioral Insight: The work suggests that while overt model behavior (logits) may not fully reflect human-like uncertainty, the internal representations of LLMs contain stronger signals of human alignment. This implies that the mechanisms for human-like uncertainty exist within the model but are not always expressed in the final output, potentially due to the constraints of instruction tuning.
Impact of Tuning: The findings highlight a trade-off: while instruction tuning improves task performance in many domains, it appears to disrupt the natural emergence of human-similar uncertainty behaviors and calibration.
Future Directions: The authors modestly suggest that identifying these alignment signals in latent vectors could eventually allow for the direct manipulation or enhancement of alignment behaviors without significant retraining. However, they acknowledge limitations, noting that the results are currently restricted to short-form factual recall and may not generalize to complex reasoning or planning tasks.

In summary, the paper establishes that LLMs possess a latent capacity for human-aligned uncertainty, but this capacity is often suppressed by standard instruction tuning and is more visible in internal activations than in final predictions.