💬 NLP

Lost in Diffusion: Uncovering Hallucination Patterns and Failure Modes in Diffusion Large Language Models

This paper presents the first controlled study revealing that Diffusion Large Language Models (dLLMs) exhibit higher hallucination rates and distinct failure modes compared to autoregressive counterparts, highlighting critical reliability challenges despite their performance on general tasks.

Original authors: Zhengnan Guo, Fei Tan

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhengnan Guo, Fei Tan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a story.

The Old Way (Auto-Regressive Models):
Think of the current popular AI models (like the ones powering most chatbots today) as a strict line of dominoes. To write a story, they must place the first domino, then the second, then the third, one by one. If they make a mistake on the second domino, the third one might fall the wrong way, and by the end, the whole story is a mess. This is called the "snowball effect"—one small error grows into a big lie.

The New Way (Diffusion Models):
Now, imagine a new type of AI called a Diffusion Large Language Model (dLLM). Instead of placing dominoes one by one, imagine this AI starts with a cloud of static noise (like a TV screen full of snow) and tries to turn that noise into a clear picture of your story all at once. It looks at the whole page, guesses what words should go where, and then "denoises" (cleans up) the text in many small steps until the story appears.

Theoretically, this sounds amazing. Because the AI sees the whole story at once, it should be able to fix mistakes easily, right? If it writes a wrong word in the middle, it can just look back and fix it without ruining the rest.

The Big Discovery: "Lost in Diffusion"

The researchers in this paper asked a simple question: "Is this new 'cloud-to-text' method actually better at telling the truth, or does it just lie in different, weirder ways?"

They set up a controlled experiment, pitting the new Diffusion models against the old "domino" models using the exact same brain size and training data. Here is what they found, explained simply:

1. The New Models Lie More (Surprisingly!)

Even though the Diffusion models have a "bird's-eye view" of the whole story, they are actually more prone to hallucinating (making things up) than the old domino models.

Analogy: Imagine a painter trying to fix a mistake on a canvas. The old model paints a stroke, realizes it's wrong, and tries to paint over it, but the paint smears. The new model tries to fix the whole painting at once, but because it's working with "noise," it sometimes invents a completely new, fake tree in the middle of the forest just to make the picture look balanced.

2. Two Different Ways of Failing

The paper found that these new models fail in two very different ways, depending on how they are built:

The "Rigid" Model (LLaDA): This model tries to be like the old dominoes but uses the new noise-removal technique. It gets stuck early. It starts writing the story left-to-right, gets confused, and then just stops or repeats itself. It's like a car that starts driving, hits a bump, and then the engine just stalls.
The "Chaotic" Model (Dream): This model is truly non-sequential. It jumps around the page, fixing the end of the sentence before the beginning. While this allows it to improve the story if you give it more time (more "compute"), it often gets distracted.
- Analogy: Imagine you are writing a letter to your grandma. Suddenly, the AI gets a random thought about a math equation or a coding tutorial and starts writing that in the middle of your letter, ignoring your original request. This is called "Context Intrusion."

3. The Three Weird Glitches

The researchers identified three specific "glitches" that only happen with this new Diffusion method:

Premature Termination: The AI gets confused about how two parts of the sentence fit together and just gives up, cutting the sentence off abruptly or inserting weird symbols.
- Metaphor: Like a chef who starts cooking a soup, realizes the ingredients don't match, and just dumps the pot on the floor.
Incomplete Denoising: The AI leaves behind "ghosts" of words. It might write a sentence that looks like English but is actually just gibberish or repeated letters (like "erserserserser").
- Metaphor: Like a photo that is still blurry in the corners, or a radio signal that is half-static.
Context Intrusion: As mentioned above, the AI gets hijacked by a random topic.
- Metaphor: You ask for a recipe for cake, and halfway through, the AI starts explaining how to build a bridge.

The Bottom Line

The paper concludes that while these new Diffusion models are getting very good at writing general text, they are currently less reliable than the old models when it comes to facts.

They haven't solved the problem of "hallucination"; they've just changed the type of hallucination. Instead of making a small mistake that gets bigger, they make weird, structural mistakes that are harder to predict.

The Takeaway:
Think of the new Diffusion models as a brilliant but chaotic artist. They can see the whole picture, which is great, but they are currently too noisy and prone to daydreaming about unrelated topics. Before we can trust them with important facts (like medical advice or news), we need to teach them how to stop daydreaming and stick to the story.

1. Problem Statement

While Diffusion Large Language Models (dLLMs) have emerged as a promising non-autoregressive (NAR) alternative to traditional Auto-regressive (AR) models, offering advantages in global planning and bidirectional visibility, their faithfulness remains underexplored. Specifically, it is unclear whether the diffusion mechanism mitigates or exacerbates hallucinations (inconsistencies between model output and source content/training data).

Current AR models suffer from known failure modes like the "snowballing effect" (error propagation) and the "reversal curse." Theoretically, dLLMs should mitigate these via iterative refinement and bidirectional context. However, the stochastic nature of diffusion introduces intrinsic noise. The paper addresses the critical question: Does the diffusion mechanism improve or degrade reliability compared to AR baselines?

2. Methodology

To isolate the impact of the generation paradigm from confounding factors (like model scale or training data), the authors designed a controlled pairwise comparison framework:

Control Groups:
- Architectural Alignment: Comparing LLaDA-8B (dLLM) against LLaMA-3-8B (AR). Both share similar backbone architectures and parameter counts.
- Parametric Alignment: Comparing Dream-7B (dLLM) against Qwen2.5-7B (AR). Crucially, Dream is initialized directly from Qwen weights, ensuring identical parametric knowledge; any performance gap is attributed solely to the diffusion process.
Inference Settings:
- The study uses a canonical diffusion setting (bypassing acceleration techniques like semi-autoregressive decoding) where the number of denoising steps ( $T$ ) equals the sequence length ( $L$ ).
- Temperature is set to 0 to eliminate sampling randomness.
- Decoding strategies: High-confidence decoding for LLaDA; Minimum Entropy decoding for Dream.
Evaluation Benchmarks:
- Adapted HalluLens to evaluate Extrinsic Hallucination across three tasks:
  1. PreciseWikiQA: Fact recall from parametric memory.
  2. LongWiki: Factual consistency in long-form generation (1,024 tokens).
  3. NonExistentRefusal: Ability to refuse queries about fabricated entities (knowledge boundaries).
- Evaluation is performed via an automated LLM-as-a-Judge (Gemini-2.5-Flash) with human validation on a stratified subset.

3. Key Contributions

First Controlled Benchmark: The study provides the first systematic comparison revealing that current dLLMs exhibit a significantly higher propensity for hallucination than comparable AR models.
Dynamics of Inference Compute: The authors identify a dichotomy in how inference steps affect performance:
- Quasi-autoregressive models (e.g., LLaDA) suffer from early saturation; increasing steps yields no improvement.
- Non-sequential models (e.g., Dream) show positive scaling, where increased compute allows for continuous refinement and reduced hallucination.
Novel Failure Modes: The paper identifies and categorizes three distinct failure modes unique to the diffusion process:
- Premature Termination: Abrupt EOS insertion due to syntactic misalignment between independently decoded segments.
- Incomplete Denoising: Residual noise or hollow symbols where the model fails to resolve rare entities, leading to structural collapse.
- Context Intrusion: The model hijacks the query trajectory to unrelated topics (e.g., math or code) triggered by high-frequency tokens denoised in future positions.

4. Key Results

Performance Comparison (Table 1)

Across all tasks, dLLMs underperformed AR baselines:

PreciseWikiQA: Dream-7B had a 92.54% Hallucination Rate vs. Qwen2.5-7B's 89.06%. LLaDA-8B had a Correct Rate of only 3.92% vs. LLaMA-3-8B's 10.30%.
LongWiki: While LLaDA achieved competitive Recall, both dLLMs suffered from lower Precision, resulting in inferior F1 scores.
NonExistentRefusal: dLLMs struggled significantly to refuse invalid queries. Dream-7B failed to recognize non-existent entities in 98.50% of cases (vs. 94.05% for Qwen).

Inference Dynamics (Figure 1)

LLaDA-8B: Showed early saturation. Performance metrics (F1@32) remained stagnant (~0.27) even as denoising steps increased from 128 to 1024. This is attributed to its linear noise scheduler and high-confidence decoding, which enforce a left-to-right flow, inheriting AR rigidity while adding diffusion noise.
Dream-7B: Demonstrated monotonic improvement with increased steps. Its minimum entropy decoding allows for genuine non-sequential refinement, effectively reducing hallucinations as compute budget increases. However, this flexibility correlates with a higher incidence of Context Intrusion.

Failure Mode Analysis (Table 2)

Human annotation of 200 hallucinated outputs revealed:

Incomplete Denoising was the most common failure in LLaDA (60%).
Context Intrusion was most frequent in Dream (58%).
Premature Termination occurred in both but was more prevalent in LLaDA.

5. Significance and Implications

Reliability Challenge: Despite narrowing the performance gap on general tasks, dLLMs currently pose a critical challenge to reliability due to distinct and severe hallucination mechanisms.
Structural Deficiencies: The "snowballing effect" is not eliminated in dLLMs but transformed. Early denoising errors become entrenched because current architectures lack mechanisms for retrospective correction (re-masking) once tokens are fixed.
Future Directions: The findings suggest that achieving reliable non-autoregressive generation requires more than simple denoising. Future dLLMs need dynamic sequence editing capabilities (insertion, deletion, re-masking) akin to human writing to resolve global consistency conflicts.
Caveats: The study focuses on pre-trained models to isolate the generation paradigm. Instruction-tuned dLLMs showed high variance and "over-refusal" behaviors, suggesting that current post-training alignment techniques for diffusion models are immature and introduce significant noise.

In conclusion, while diffusion offers theoretical advantages for global planning, current implementations are less faithful than autoregressive models, suffering from unique failure modes that require new architectural solutions to overcome.