💬 NLP

Why Don't You Know? Evaluating the Impact of Uncertainty Sources on Uncertainty Quantification in LLMs

This paper introduces a new dataset to systematically evaluate how different sources of uncertainty—such as knowledge gaps, output variability, and input ambiguity—affect the performance of existing uncertainty quantification methods in Large Language Models, revealing that many methods fail or become misleading when uncertainty arises from factors beyond mere knowledge limitations.

Original authors: Maiya Goloburda, Roman Vashurin, Fedor Chernogorsky, Nurkhan Laiyk, Daniil Orel, Preslav Nakov, Maxim Panov

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maiya Goloburda, Roman Vashurin, Fedor Chernogorsky, Nurkhan Laiyk, Daniil Orel, Preslav Nakov, Maxim Panov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, well-read but slightly overconfident robot assistant to help you with important tasks. You ask it questions, and it answers instantly. But sometimes, the robot is wrong. The big question for developers is: How do we know when the robot is guessing versus when it's actually sure?

This paper is like a detective story investigating how well we can tell the difference between the robot's different types of "doubt."

The Problem: One Size Doesn't Fit All

Currently, most systems try to give the robot a single "confidence score" (like a percentage from 0% to 100%) to say how sure it is. The paper argues this is like trying to describe a storm, a traffic jam, and a flat tire all with the same word: "Problem."

The authors realized that a robot's uncertainty comes from three very different sources, and treating them the same way causes confusion:

The "I Don't Know" Gap (Model Knowledge):
- The Analogy: You ask the robot, "Who was the 45th President of the US?" but the robot was trained on data that stops before that happened. It genuinely doesn't know the answer.
- The Right Move: The robot should say, "I don't know," or ask you to check a book.
The "Pick Your Favorite" Scenario (Output Variability):
- The Analogy: You ask, "Name a fruit that is red." The robot could say "Apple," "Strawberry," or "Cherry." All are correct. If the robot says "Apple" with 100% confidence, it's not lying; it just picked one valid option from a list.
- The Right Move: The robot should feel confident, even though there are many right answers. It shouldn't panic just because there are options.
The "Vague Question" Trap (Input Ambiguity):
- The Analogy: You ask, "Who won the game?" but you never told the robot which game. The robot has to guess if you mean soccer, chess, or a video game.
- The Right Move: The robot should stop and ask, "Which game do you mean?" instead of confidently guessing "The Lakers" and being wrong.

The Experiment: Building a Special Test

To figure out which "confidence meter" works best for which situation, the authors built a special test dataset. Think of it like a driving test with three specific tracks:

Track A: A road with a missing bridge (Knowledge Gap).
Track B: A road with three valid exits (Output Variability).
Track C: A road with no signs (Ambiguity).

They tested many different "confidence meters" (mathematical methods) on these tracks to see which one could correctly identify the problem.

The Big Discovery

The results were surprising and a bit worrying:

The "Knowledge Gap" Meter works great: When the robot genuinely doesn't know something, most confidence meters correctly say, "Hey, I'm unsure here."
The "Vague Question" Meter fails: When the question is unclear, the robot often confidently guesses an answer anyway. The confidence meters think the robot is sure, even though it's actually guessing in the dark.
The "Pick Your Favorite" Meter gets confused: When there are multiple right answers, some meters think the robot is "uncertain" just because it didn't pick the exact same word every time, even though the answer was correct.

The Metaphor: It's like having a smoke detector that works perfectly when there's a fire, but starts screaming when you just open a window (variability) or when you walk into a foggy room (ambiguity). You end up ignoring the alarm because it's always going off for the wrong reasons.

Why This Matters

If we use these flawed confidence meters in real life, bad things could happen:

In Medicine: A robot might confidently give a wrong diagnosis because the patient's symptoms were vague, and the robot didn't ask for clarification.
In Law: A robot might refuse to answer a simple question because it thinks there are "too many possible answers," slowing down the work.

The Conclusion

The paper concludes that we need to stop using a single "confidence score." Instead, we need smart, multi-tool systems that can ask:

"Do you not know the answer?" (Check knowledge)
"Is the question unclear?" (Check ambiguity)
"Are there many right answers?" (Check variability)

Only by understanding why the robot is uncertain can we teach it to behave safely: admitting ignorance when it doesn't know, asking for clarity when the question is vague, and staying confident when it's just picking from a list of correct options.

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in critical domains (medicine, law, etc.), making Uncertainty Quantification (UQ) essential for safe deployment. Current UQ methods typically output a single scalar confidence score (e.g., "80% likely to be correct"). However, this approach is flawed because uncertainty in natural language arises from distinct sources that require different system behaviors:

Model Knowledge Uncertainty: The model lacks the necessary facts (Action: Abstain or retrieve external knowledge).
Output Variability: Multiple distinct, valid answers exist for a single question (Action: Present multiple options or diversify).
Input Ambiguity: The user's query is underspecified or ambiguous (Action: Request clarification).

Existing datasets and evaluation frameworks fail to isolate these sources, often conflating them. Consequently, it is unclear whether current UQ methods can distinguish between a model "not knowing" an answer versus a situation where "many answers are correct" or "the question is unclear." This leads to misleading confidence scores and inappropriate downstream actions.

2. Methodology

A. Dataset Construction

The authors introduced a new, human-validated dataset designed to isolate the three specific uncertainty sources.

Source: Derived from the MAQA dataset (multi-answer questions) and Natural Questions.
Structure: Questions are organized into triplets. Each triplet contains three variations of the same core topic, each targeting a specific uncertainty source:
1. Type 1 (Model Knowledge): A well-specified question with a single unique correct answer. (Uncertainty stems from the model's lack of knowledge).
2. Type 2 (Output Variability): A well-specified question where multiple distinct answers are valid (e.g., "Name one desert animal"). (Uncertainty stems from the spread of probability mass across valid candidates).
3. Type 3 (Input Ambiguity): A question derived from Type 1 but with critical context removed to make it underspecified (e.g., replacing specific entities with vague references). (Uncertainty stems from the inability to determine the user's intent).
Scale: 539 questions per type (1,617 total), ensuring balanced domains and comparable lengths to control for confounding factors.

B. Experimental Setup

Models: Three open-weight instruct models representing different architectures: Llama-3.1-8B, Gemma-3-12B, and Qwen-2.5-14B.
UQ Methods Evaluated: A comprehensive suite of methods categorized into:
- Token-Probability: Sequence Probability (SP), Perplexity (PPL), Mean Token Entropy (MTE).
- Consistency-Based: Semantic Entropy, Number of Semantic Sets, Graph Laplacian metrics.
- Hybrid: Semantic Density, CoCoA, SAR.
- Verbalized: P(True), Verbalized-2S (asking the model to self-report confidence).
Evaluation Metrics:
- Prediction-Rejection Ratio (PRR): Measures how well a method ranks predictions by quality when rejecting the lowest-confidence ones.
- Spearman Correlation: Correlation between the uncertainty score and the actual quality of the generation.
Quality Assessment: Used an LLM-as-a-Judge (GPT-5) to verify correctness, with human validation confirming 98–100% agreement.

3. Key Contributions

New Benchmark Dataset: The first dataset explicitly categorizing and isolating uncertainty sources (Knowledge, Variability, Ambiguity) into controlled triplets, enabling fair comparison of UQ methods.
Systematic Evaluation: The first large-scale study evaluating how existing UQ methods perform under these distinct, controlled uncertainty conditions across multiple model families.
Failure Mode Analysis: An in-depth analysis revealing that current methods are highly sensitive to the source of uncertainty, often failing to distinguish between "unknown facts" and "multiple valid answers."

4. Key Results

No "One-Size-Fits-All" Method: No single UQ method consistently outperforms others across all uncertainty sources. Performance is highly context-dependent.
Knowledge Uncertainty (Type 1):
- Best Performers: Hybrid methods (combining token probabilities and consistency) and Semantic Density.
- Insight: When a single correct answer exists, both semantic agreement and probabilistic confidence are strong signals.
Output Variability (Type 2):
- Best Performers: Token-probability methods (e.g., MTE, MCSE).
- Failure of Consistency: Consistency-based methods (which assume high agreement = high confidence) fail here. They interpret the natural diversity of valid answers as "high uncertainty" or "low quality," leading to false negatives.
Input Ambiguity (Type 3):
- Best Performers: Verbalized methods (specifically Verbalized-2S), which improve with model size.
- Failure of Probabilities: Token-probability methods often assign high confidence to generic, safe answers that fit multiple interpretations, underestimating the true ambiguity.
- Model Behavior: Models overwhelmingly commit to an arbitrary interpretation of ambiguous inputs rather than asking for clarification, even when their internal uncertainty scores might suggest otherwise.
Correlation Decay: All methods show strong Spearman correlation with quality in single-answer settings but degrade significantly when output variability or input ambiguity is introduced.

5. Significance and Future Directions

Rethinking UQ: The paper argues that a single scalar confidence score is insufficient for real-world deployment. Systems need structured uncertainty representations that explicitly disentangle what is uncertain (knowledge vs. variability vs. ambiguity).
Action-Oriented UQ: The source of uncertainty should dictate the system's action:
- Knowledge Gap $\rightarrow$ Retrieval or Abstention.
- Variability $\rightarrow$ Present multiple options.
- Ambiguity $\rightarrow$ Ask for clarification.
Limitations: The study focuses on QA tasks and white-box models. It notes that black-box proprietary models may behave differently due to built-in safety filters or tool use.
Conclusion: Current UQ methods are effective proxies for "model ignorance" but are often misleading for "valid diversity" or "user ambiguity." Future work must focus on methods that can identify the source of uncertainty to enable safer, more context-aware AI systems.