📊 statistics

LLMs as Implicit Imputers: Uncertainty Should Scale with Missing Information

This paper argues that large language models acting as implicit imputers should exhibit uncertainty scaling with missing information, finding that while sampling-based confidence fails to reflect increasing context degradation, response entropy effectively tracks missingness and better predicts accuracy.

Original authors: Stef van Buuren

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Stef van Buuren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. Usually, you have a full file of evidence (the "context") to help you find the culprit. But sometimes, the file gets shredded, and you only have a few scraps of paper left.

This paper asks a simple but crucial question: When your detective (an AI language model) is working with missing evidence, does it know how unsure it should be?

The authors argue that the AI should act like a "statistical detective" who admits, "I'm guessing here because I don't have enough clues." Instead, they found that current AIs often act like overconfident detectives who guess wildly but insist they are 100% sure.

Here is the breakdown of their findings using simple analogies:

1. The Core Idea: The "Missing Clue" Test

The researchers treated the AI like a Multiple Imputer. In statistics, when data is missing, experts don't just guess one answer; they generate many possible scenarios to see how much the answer might vary.

The Test: They took questions and slowly shredded the background information (the context) until the AI had nothing to read but the question itself.
The Rule: As the clues disappear, the AI's "uncertainty" should go up. If it has no clues, it should be very unsure. If it has all the clues, it should be very sure.

2. The Two Ways to Measure "Uncertainty"

The researchers looked at two ways to measure how unsure the AI was when they asked it the same question 10 times in a row:

The "Loud Voice" (Confidence): This measures how often the AI gives the same answer every time. If it says "Denver Broncos" 10 times out of 10, it sounds very confident.
The "Crowd Chatter" (Entropy): This measures how different the answers are. If the AI says "Denver Broncos" once, "New England" once, "The Patriots" once, and "I don't know" seven times, there is a lot of "chatter" or variety. This is high entropy.

3. The Big Discovery: The "Loud Voice" is a Liar

The study found a major problem with the "Loud Voice" (Confidence):

What happened: Even when the AI had zero context (no clues), it often gave the same wrong answer 10 times in a row. It sounded incredibly confident, even though it was completely guessing based on what it memorized from its training.
The Analogy: Imagine a student taking a test who has forgotten the textbook. They guess "C" for every single question. They are 100% consistent (high confidence), but they are 100% wrong. The AI was doing exactly this.

4. The Winner: "Crowd Chatter" (Entropy)

The "Crowd Chatter" (Entropy) behaved much better:

What happened: As the researchers removed the context, the AI's answers started to scatter. Instead of saying "Denver Broncos" 10 times, it started saying "Denver," "Broncos," "The Team," "I don't know," etc.
The Analogy: This is like a group of detectives in a room. When they have all the evidence, they all point to the same suspect. When you take the evidence away, they start pointing in different directions. The more confused they get, the more they point in different directions.
The Result: The "Crowd Chatter" measure went up exactly as the AI's accuracy went down. It was a true signal of confusion.

5. The "Resolution Ratio" (How much did the clues help?)

The authors created a simple score called the Resolution Ratio.

Think of it like a dimmer switch for a lightbulb.
No Context (Off): The room is dark (high uncertainty).
Full Context (On): The room is bright (low uncertainty).
The Score: This measures how much the "light" (the context) turned the "darkness" (uncertainty) off. They found that having the full context resolved about 80% of the uncertainty, which makes sense.

Summary of the Paper's Claims

The Problem: Current AI models are often "improper imputers." When they lack information, they suppress their uncertainty and give a single, confident, but often wrong answer.
The Solution: We should look at how much the AI's answers vary (Entropy) rather than how often it repeats the same answer (Confidence).
The Proof: When context was removed, the AI's accuracy dropped, but its "Confidence" stayed high (lying to us). Its "Entropy" (variety of answers) went up, correctly signaling that it was lost.
The Takeaway: If you want to know if an AI is guessing because it's missing information, don't ask how sure it sounds. Ask: "If I asked you 10 times, would you give me 10 different answers?" If yes, it knows it's unsure. If it gives you the same wrong answer 10 times, it's overconfident.

The paper concludes that Entropy is a much better "black box" tool for telling us when an AI is flying blind than the standard "Confidence" scores we usually see.

Technical Summary: LLMs as Implicit Imputers

Problem Statement

Large Language Models (LLMs) are increasingly deployed in scenarios where input context is incomplete, truncated, or degraded (e.g., partial retrieval in RAG systems). Despite operating under such information deficits, LLMs typically generate single answers with high apparent certainty. This raises a critical question: Do LLM uncertainty estimates appropriately reflect the amount of missing information?

Current evaluations often focus on static calibration (matching confidence to accuracy under fixed inputs). However, there is a lack of understanding regarding whether uncertainty estimates dynamically scale as inputs become less informative. The paper argues that a proper predictive system should exhibit uncertainty that increases as the fraction of missing information increases, a principle grounded in Bayesian deep learning and missing data theory.

Methodology

Theoretical Framework: Implicit Multiple Imputation

The authors operationalize the evaluation of LLM uncertainty by drawing an analogy to Multiple Imputation (MI) [Rubin, 1987]. In MI, uncertainty arises from unobserved variables and is quantified by the variability across multiple completed datasets.

Analogy: Repeated sampling from an LLM under incomplete context is viewed as generating "implicit completions" of missing information.
Criterion: A proper imputer (and thus a proper LLM) must exhibit uncertainty that scales with the amount of missing information. If context is removed, the variability (uncertainty) of the generated answers should increase.

Experimental Setup

Dataset: SQuAD (Reading Comprehension), utilizing short-answer spans extracted from context paragraphs.
Evidence Conditions: A controlled framework where context availability is varied via a parameter $\alpha \in [0, 1]$ $α \in [0, 1]$ . Context $C(\alpha)$ $C (α)$ is constructed by retaining the first $100 \cdot \alpha$ $100 \cdot α$ percent of the original context characters.
- $\alpha = 1$ : Full context.
- $\alpha = 0$ : No context.
Sampling: Using gpt-4o-mini with temperature $T=0.7$ , $m=10$ stochastic responses are generated per input $(q, C)$ . To minimize correlation, each sample is obtained via a separate API call.
Normalization: Responses are normalized to handle minor lexical variations (e.g., article omission) but not semantic generalization.

Uncertainty Measures

Two black-box measures are estimated from the empirical distribution of $m$ sampled answers:

Sampling-based Confidence ( $c$ ): The empirical frequency of the most frequent answer ( $c = \max_k p_k$ ). This measures modal concentration.
Response Entropy ( $H$ ): Shannon entropy of the normalized answer distribution ( $H = -\sum p_k \log p_k$ ). This measures dispersion across competing answers.
Resolution Ratio ( $\rho_R(\alpha)$ ): A diagnostic metric estimating the proportion of baseline uncertainty resolved by context level $\alpha$ .
$\rho_R(\alpha) = \frac{[H_0 - H(\alpha)]_+}{H_0}$
Where $H_0$ is the entropy under no context. $\rho_R = 0$ implies context provides no uncertainty reduction; $\rho_R = 1$ implies uncertainty is fully resolved.

Information Dependence Index (IDI)

Defined as $\Delta = Acc_1 - Acc_0$ , this metric quantifies the extent to which predictive accuracy depends on the presence of context.

Key Contributions

Evaluation Framework: A principled framework for assessing LLM uncertainty under partial observability, grounded in the requirement that uncertainty must increase as information decreases.
MI Analogy: Explicitly linking repeated LLM sampling to Multiple Imputation, motivating entropy as a black-box proxy for predictive uncertainty induced by missing information.
Diagnostic Tool: Introduction of the Resolution Ratio $\rho_R(\alpha)$ , a heuristic that measures the fraction of baseline uncertainty resolved by context, estimable solely from black-box sampling.
Empirical Validation: Quantitative evidence demonstrating that entropy tracks missing-information uncertainty more faithfully than confidence, with a significant advantage in explained variance (quadratic $R^2$ gap up to 0.057).

Results

Taxonomy of Model Behavior

Analysis of 400 contexts revealed three distinct regimes under context removal:

Memorized: High accuracy with and without context; near-zero entropy ( $H_0 \approx 0$ ). The model relies on parametric knowledge.
Biased: High accuracy with context, but high accuracy with incorrect answers without context; near-zero entropy ( $H_0 \approx 0$ ). The model is deterministically wrong when context is missing.
Uncertain: High accuracy with context, but low accuracy and high entropy without context. This regime reflects genuine uncertainty induced by missing information.

Distribution: The "Uncertain" regime was the largest (43.8%), indicating that most questions require context and admit diverse responses when it is missing.

Performance of Uncertainty Measures

Entropy vs. Confidence: As context is removed ( $\alpha$ $α$ decreases), accuracy drops monotonically.
- Entropy ( $H$ ): Increases monotonically as context is removed, closely tracking the loss in accuracy.
- Confidence ( $c$ ): Declines much more slowly and remains high even as accuracy collapses. In the context-sensitive subset, accuracy fell to 0.05 while confidence remained at 0.52.
Predictive Power: Regression analysis showed that entropy consistently explains more variance in accuracy than confidence across all evidence levels.
- At $\alpha = 0.5$ (Full sample), entropy explained 24.3% of variance vs. 18.6% for confidence ( $\Delta R^2 = 0.057$ ).
- Entropy explained approximately 31% more variance than confidence at this level.
Calibration: The calibration plot (Accuracy vs. Confidence) revealed substantial overconfidence that worsened as context was removed. Entropy, conversely, behaved consistently with the MI criterion.

Resolution Ratio

The Resolution Ratio $\rho_R(\alpha)$ showed that full context resolves approximately 80% of baseline uncertainty in the full sample. In the context-sensitive subset, $\rho_R(\alpha)$ predicted accuracy monotonically, whereas confidence exhibited non-monotone patterns driven by the "biased" regime.

Significance and Claims

The paper claims that response entropy is a more responsive black-box uncertainty measure than sampling-based confidence under incomplete context.

Theoretical Alignment: Entropy behaves in accordance with the Bayesian requirement that uncertainty should scale with missing information. Confidence systematically fails this requirement, often remaining high even when the underlying distribution is diffuse (a condition where a proper imputer should signal high uncertainty).
Practical Utility: The proposed metrics (Entropy and $\rho_R$ ) require no access to model internals (weights, logits) or entailment classifiers. They are applicable to any LLM accessible via an API.
Implications for LLM Design: The authors suggest that the "overconfidence" observed in the biased regime may stem from the softmax operation, which concentrates probability mass on the modal response. They propose that Predictive Mean Matching (PMM)—sampling from an implicit donor pool rather than passing through softmax—could yield better-calibrated uncertainty, aligning LLM behavior with proper imputation theory.

The paper concludes that importing Multiple Imputation criteria into the LLM setting provides a theory-inspired standard for evaluating uncertainty, independent of specific tasks or models, and opens a channel between statistical missing-data literature and LLM uncertainty research.