Technical Summary: Uncertainty Quantification for Large Language Diffusion Models

Problem Statement

Large Language Diffusion Models (LLDMs) have emerged as a competitive alternative to autoregressive Large Language Models (LLMs), offering faster inference through semi-autoregressive decoding and parallel token generation. However, like their autoregressive counterparts, LLDMs are prone to hallucinations, necessitating reliable Uncertainty Quantification (UQ) for safe deployment.

Existing UQ methods are fundamentally misaligned with the diffusion paradigm:

1. Factorization Mismatch: Standard white-box methods rely on left-to-right autoregressive factorization ($p(y_l | y_{<l}, x)$), which is not natively available in LLDMs.
2. Efficiency Mismatch: Black-box methods typically rely on repeated sampling to estimate consistency (e.g., Semantic Entropy, CoCoA). Since LLDMs are designed for efficiency, multiplying inference cost by repeated sampling negates their primary advantage.

There is currently no systematic framework for UQ in LLDMs that leverages their unique iterative denoising process without incurring prohibitive computational overhead.

Methodology

The authors propose D-CoCoA, a family of diffusion-aware UQ methods that derive uncertainty signals directly from the iterative denoising trajectory, avoiding the need for repeated sampling. The approach integrates three specific categories of diffusion-specific signals:

1. Diffusion-Specific Uncertainty Signals

The paper introduces several zero-shot signals derived from the generation process:

• Masked Diffusion Likelihood: The authors utilize a Monte Carlo estimator (MCNLL) to approximate the negative log-likelihood based on the masked diffusion objective, serving as a principled likelihood proxy.
• Trajectory Statistics: Signals include the Number of Function Evaluations (NFE) as a proxy for denoising complexity, remasking behavior (frequency of tokens being re-masked), and token-level prediction instability (frequency of token flips during the trajectory).
• Trajectory Semantic Instability (AD): A novel signal measuring the semantic dissimilarity between intermediate denoising states ($\tilde{y}_t$) and the final output ($y$). This captures semantic drift before convergence.

2. Theoretical Grounding

The paper provides a theoretical justification for using trajectory dissimilarity as an uncertainty score. Theorem 1 proves that the expected trajectory dissimilarity ($u_{AD}$) is a lower bound on the masked diffusion training objective (loss). Consequently, a high value of $u_{AD}$ indicates a high expected prediction error, motivating its use as a proxy for hallucination risk.

3. The D-CoCoA Framework

Building on the state-of-the-art CoCoA method (which combines information-based scores with consistency-based scores), the authors adapt it for LLDMs:

• Information Component: Replaced with diffusion-aware likelihood estimates (e.g., MCNLL or CommitNLL).
• Consistency Component: Replaced with trajectory-based semantic dissimilarity (AD), utilizing intermediate states from the single denoising trajectory instead of multiple sampled generations.

Two specific configurations are proposed:

• D-CoCoA-L (Local): Combines CommitNLL (confidence at token commitment) with block-wise trajectory dissimilarity.
• D-CoCoA-G (Global): Combines normalized MCNLL with full-trajectory dissimilarity and an NFE complexity penalty.

Key Contributions

1. First Systematic Study: This work presents the first comprehensive investigation of UQ specifically tailored for LLDMs.
2. Novel Signals: Introduction of diffusion-specific uncertainty signals, including trajectory semantic instability, remasking dynamics, and denoising complexity.
3. Theoretical Proof: Formal demonstration that expected trajectory dissimilarity lower-bounds the masked diffusion training objective.
4. D-CoCoA Method: A unified, efficient UQ method that adapts CoCoA to the diffusion setting, replacing expensive sampling with single-pass trajectory analysis.
5. Empirical Validation: A large-scale study across eight datasets and two models (LLaDA-1.5 and Dream) demonstrating the efficacy of the proposed approach.

Experimental Results

The evaluation was conducted on eight datasets covering Question Answering (MMLU, TriviaQA, CoQA, GSM8k), Summarization (SamSum, XSum), and Machine Translation (WMT14, WMT19). Performance was measured using the Prediction Rejection Ratio (PRR) and computational overhead.

• Performance vs. Efficiency: D-CoCoA methods achieve a superior performance-efficiency trade-off. They approach the performance of the strongest sampling-based baselines (e.g., CoCoA-MCNLL, KLE) while incurring up to 100x lower computational overhead.
• Baseline Comparison:
  • Standard autoregressive UQ methods (e.g., MSP, Perplexity, SAR) performed poorly or marginally better than random when applied to LLDMs due to the factorization mismatch.
  • Purely diffusion-specific signals (e.g., NFE, Remask) provided useful information but were generally insufficient for high-quality UQ on their own.
  • The hybrid D-CoCoA approach significantly outperformed both standard baselines and standalone diffusion signals.
• Ablation Studies: The results confirmed that using the full generation trajectory and MCNLL likelihood yields the most consistent performance. The inclusion of NFE as a complexity penalty further improved the global variant (D-CoCoA-G).

Significance and Claims

The paper claims that LLDMs can deliver both fast inference and reliable hallucination detection simultaneously, without the need for the costly sampling procedures required by autoregressive UQ methods.

By leveraging the distinctive properties of the iterative denoising process—specifically the trajectory of intermediate states and masking dynamics—the authors demonstrate that effective uncertainty estimation can be achieved at almost no additional computational cost. This work addresses a critical gap in the deployment of diffusion-based language models, enabling their use in safety-critical applications where reliability is paramount. The authors emphasize that their approach does not require task-specific labels or repeated sampling, making it a practical, unsupervised solution for the emerging class of diffusion language models.