💬 NLP

Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer

This paper challenges the view that hallucinations stem solely from missing knowledge, revealing that larger instruction-tuned LLMs often hallucinate because they fail to commit to a correct concept already present in their probability distribution, instead dispersing probability mass across alternatives due to a scale-dependent sharpening mechanism that drives both helpfulness and confident errors.

Original authors: Jewon Yeom, Jaewon Sok, Heejun Kim, Seonghyeon Park, Jeongjae Park, Taesup Kim

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jewon Yeom, Jaewon Sok, Heejun Kim, Seonghyeon Park, Jeongjae Park, Taesup Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not About "Not Knowing"

Most people think AI hallucinations (when an AI makes things up) happen because the AI simply doesn't know the answer. The idea is: If the AI knows the fact, it tells the truth. If it doesn't, it guesses.

This paper argues that is wrong.

The researchers found that often, the AI does know the answer. It has the correct information "in its head" at the very moment it starts speaking. However, it still chooses to say the wrong thing. They call this a "Commitment Failure."

The Analogy: The Nervous Chef

Imagine a very talented chef (the AI) who is asked, "What is the capital of France?"

The Old Theory: If the chef doesn't know the answer, they guess "London." If they know it, they say "Paris."
The New Theory (This Paper): The chef knows the answer is Paris. In fact, they are thinking about "Paris" so hard that their brain is buzzing with the word. But, because they are thinking about "Paris" in so many different ways at once (e.g., "Paris," "paris," "the city of lights," "France's capital"), their brain gets confused.
- They have a strong feeling for "Paris," but it's spread out across all those different phrases.
- At the same time, they have a very sharp, focused thought about "London" (maybe because they saw a picture of Big Ben earlier).
- Even though the "Paris" feeling is stronger overall, the "London" thought is so sharp and concentrated that it wins the race to their mouth. They blurt out "London" confidently, even though they knew it was Paris.

How They Discovered This

The researchers looked at how AI models "think" before they speak. They focused on the very first word the AI generates (the "commitment step").

They introduced a new way of measuring what the AI knows, called Semantic Probability Mass.

Old Way: Did the AI pick the exact right word? (e.g., Did it say "Paris"?)
New Way: Did the AI have a strong feeling about the concept of Paris, even if it was split between "Paris," "paris," and "the city of lights"?

The Findings:

They tested many AI models (from small to huge).
They found that 16% to 47% of the time the AI made a mistake, it actually had a strong "feeling" for the correct answer.
The Twist: The bigger the AI model, the more often this happened. Bigger models didn't just know more; they made more of these specific types of mistakes where they knew the answer but still got it wrong.

Why Does This Happen? (The "Sharpening" Effect)

The paper suggests that Instruction Tuning (training the AI to be helpful and follow rules) changes how the AI thinks.

Imagine the AI's brain is a landscape of hills and valleys.

Before Training: The hills are flat and fuzzy. The AI is unsure.
After Training: The AI becomes very decisive. It turns its thoughts into sharp, tall spikes.
- When the AI is right, the spike for the correct answer is so tall and sharp that it wins easily.
- But, sometimes the AI gets confused. The "correct" answer is split into many small, flat hills (because it's thinking of "Paris," "paris," "France," etc.). Meanwhile, the "wrong" answer is one single, incredibly sharp, tall spike.
- The AI's decision-making process is like a ball rolling down a hill. It always rolls toward the sharpest point. Even if the "correct" side has more total "hill area," the "wrong" side has the sharpest peak, so the ball rolls there.

The Two Types of Mistakes

The researchers found two ways this "Commitment Failure" happens:

The Wrong First Word: The AI starts the sentence with the wrong word immediately (e.g., saying "Moscow" instead of "Paris"), even though it had a strong feeling for Paris.
The Wrong Path: The AI starts with the right word (e.g., "Paris") but then immediately goes off track in the next few words, turning "Paris" into a story about a different city.

The "Alignment Tax"

The paper concludes that this is a side effect of making AI models "helpful."

To make an AI confident and fast, we train it to be very decisive (sharp spikes).
This works great when the AI is right.
But when the AI is slightly confused, that same "decisiveness" makes it confidently wrong. It doesn't hesitate to pick the wrong answer because its brain is so good at picking the "sharpest" option, even if that option is incorrect.

Summary

Hallucinations aren't always ignorance. Sometimes the AI knows the answer but fails to "commit" to it.
Bigger models aren't perfect. As models get bigger, they get better at being decisive, which ironically makes them more prone to being confidently wrong.
The problem is distribution. The AI spreads its "knowledge" of the right answer too thin across many variations, while the wrong answer gets all the focus in one sharp spot. The AI picks the sharp spot, not the right answer.

The paper suggests that to fix this, we might need to teach AI to look at the "whole concept" (all the ways to say "Paris") rather than just picking the single sharpest word.

Technical Summary: Hallucination as Commitment Failure

Problem Statement

Large Language Models (LLMs) frequently produce fluent but factually incorrect outputs, known as hallucinations. A prevailing assumption in the field is that hallucination is a direct consequence of missing knowledge: the model hallucinates only when the correct answer is absent from its generation-time distribution. Recent work has identified "critical tokens" or "forking tokens" where entropy spikes, suggesting that hallucinations are decided at specific steps in the generation trajectory. However, existing uncertainty-based detection methods (e.g., semantic entropy) often fail to identify "confident hallucinations"—cases where the model emits a wrong answer with high confidence despite possessing the correct knowledge.

This paper challenges the assumption that hallucination implies a lack of knowledge. It asks: At the moment a model commits to an answer, does it possess the correct concept in its probability distribution, even if it emits a wrong token?

Methodology

Concept and Semantic Probability Mass ( $P_{mass}$ )

The authors introduce a semantic notion of answer availability. Instead of analyzing individual tokens, they define a concept $c$ as an equivalence class of token completions denoting the same answer (e.g., "Saint Basil's Cathedral," "St. Basil's," "Cathedral of...").
They define the per-step semantic probability mass as:
$P_{mass}(t; c) = \sum_{v \in S_c} P_\theta(v \mid Q, y_{<t})$
where $S_c$ is the set of first-token IDs corresponding to the surface forms of concept $c$ . This metric aggregates the probability mass assigned to all valid aliases of the correct answer at a specific generation step.

Experimental Setup

Models: The study covers 18 models across the Qwen (0.8B to 72B) and Llama (1B to 70B) families, including both Instruct (instruction-tuned) and Base variants.
Datasets: Short-form QA (TriviaQA, NQ-Open) and Multiple-Choice QA (MMLU, ARC-Challenge).
Commitment Step ( $t_c$ ): In instruction-tuned short-form QA, the commitment step is fixed at $t=1$ (the first generated token). For long-form generation, $t_c$ is identified as the step where the answer entity is emitted.
Definition of Commitment Failure: A hallucinated sample is classified as a commitment failure if $P_{mass}(t_c; c^*) \geq 0.2$ , where $c^*$ is the ground-truth concept. This indicates the model assigned substantial probability mass to the correct concept but still emitted a wrong final answer.

Analysis Dimensions

Commitment Failure Rate (CF%): The proportion of hallucinations that are commitment failures.
Decomposition: Separating failures into First-Token Selection Failures (greedy token $\notin S_{c^*}$ ) and Multi-Token Divergences (greedy token $\in S_{c^*}$ but subsequent tokens diverge).
Distributional Analysis: Comparing the probability distribution of commitment failures against matched correct samples (samples with similar $P_{mass}$ but correct output).
Instruction Tuning vs. Scale: Isolating the effects of model size versus instruction tuning by comparing Instruct and Base models.

Key Results

1. Commitment Failures are Prevalent and Scale-Dependent

Across all instruction-tuned models, 16% to 47% of hallucinations are commitment failures. Crucially, this rate rises monotonically with model scale.

Qwen Instruct: CF% increases from 16% (0.8B) to 41% (72B).
Llama Instruct: CF% increases from 16% (1B) to 47% (70B).
This finding contradicts the intuition that larger models simply "know more" and hallucinate less; rather, larger models hallucinate more often despite having the correct concept in their distribution.

2. The Mechanism: Instruction-Induced Sharpening

The paper identifies instruction tuning as the primary driver of this phenomenon, not scale alone.

Base Models: Show flat wrong-token probabilities (~0.30) across scales and lower commitment failure rates.
Instruct Models: Exhibit monotonic sharpening. The probability of the wrong greedy token in selection failures increases with scale (e.g., Qwen Instruct: 0.31 at 0.8B $\to$ 0.57 at 72B).
Distributional Shift: In commitment failures, the correct concept's mass is often fragmented across multiple alias tokens (e.g., "Saint," "St," "C"), preventing any single alias from winning the argmax against a competing concept's dominant token. In larger Instruct models, this fragmentation collapses (mass concentrates on one alias), but the wrong concept's token becomes even sharper, leading to "wrong-attractor" failures.

3. Multi-Token and Hidden State Evidence

Multi-Token Divergences: In large Instruct models (70B+), when the first token aligns with a correct alias, the second token's entropy ( $H_{t=2}$ ) drops to near-zero (0.05–0.10). The model commits to a specific multi-token phrase immediately, even if that phrase leads to a wrong entity (e.g., "George Washington Carver" instead of "George Washington").
Hidden States: Pre-generation hidden states at $t=1$ contain predictive information about correctness. Instruct models show higher AUROC in hidden-state probes compared to Base models, and this signal is "front-loaded" (stronger at $t=1$ ).
Output Mapping: The gap between Instruct and Base models is larger in output-level detection (AUROC +0.29) than in hidden-state probes (+0.08), suggesting instruction tuning amplifies the signal during the projection to the output vocabulary.

4. Two Faces of Commitment Failure

The paper distinguishes two structural failure modes:

Fragmentation-driven: (Common in smaller/Base models) Correct mass is split across aliases; no single token wins.
Wrong-Attractor-driven: (Common in large Instruct models) Correct mass has collapsed onto a single alias, but a wrong concept's token is even sharper ( $P_{greedy} > P_{mass}(c^*)$ ).

Significance and Claims

The paper reframes confident hallucination not as a momentary slip or a lack of knowledge, but as a structural consequence of how probability mass is shaped at the commitment step.

Unified Mechanism: The same instruction-induced sharpening that produces confident correctness (when the committed concept is right) also produces confident misselection (when the committed concept is wrong).
Alignment Tax: This phenomenon is situated as one face of the broader "alignment tax," where alignment (instruction tuning) improves helpfulness and fluency but degrades calibration and causes mode collapse, leading to overconfident errors.
Limitations of Uncertainty Detection: The results explain why standard uncertainty metrics (like token entropy or semantic entropy) fail to flag confident hallucinations: the model does have the correct concept (high $P_{mass}$ ), but the distribution is shaped such that the wrong token wins the argmax.
Analytical Probe: The authors propose $P_{mass}$ as an analytical probe to understand model behavior, noting it is not a deployable detector (as it requires ground-truth aliases) but reveals that "knowing the answer" and "emitting the answer" are distinct events in the generation process.

The paper concludes that larger models do not just know more; they misfire more often on what they know, driven by a sharpening disposition that makes the model decisive in both correct and incorrect commitments.