Technical Summary: GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

1. Problem Statement

Diffusion Large Language Models (dLLMs) offer significant inference efficiency gains over autoregressive models (ARMs) by generating multiple tokens simultaneously and breaking the strict left-to-right generation order. However, dLLMs currently lag behind state-of-the-art ARMs in generation quality, necessitating effective fine-tuning methods like Reinforcement Learning (RL).

The central obstacle to applying RL to dLLMs is the intractability of the policy likelihood. Existing solutions generally fall into two categories:

1. Trajectory Likelihood Methods: Estimate exact likelihoods by accumulating transition probabilities. While unbiased, they are computationally expensive and misaligned with pre-training objectives.
2. Sequence Likelihood (ELBO) Methods: Use the Evidence Lower Bound (ELBO) estimated from randomly masked sequences as a surrogate for the true likelihood. These are computationally efficient and aligned with pre-training.

The Core Issue: The paper argues that ELBO-based RL methods suffer from a Training-Inference Mismatch (TIM) bias. This bias arises from two sources:

• Likelihood Gap: The ELBO is a lower bound with a non-negligible gap from the true likelihood, introducing bias and variance in the importance sampling ratios used in Policy Gradient (PG) or PPO objectives.
• Sampling Distribution Mismatch: dLLM inference uses iterative re-masking and block-wise decoding, inducing a sampling distribution ($\pi^{rm}_{old}$) that differs from the training policy distribution ($\hat{\pi}_{old}$) estimated via ELBO.

This mismatch causes the importance sampling correction to be ineffective, leading to degraded performance and, in severe cases, catastrophic training collapse.

2. Methodology: Guided Denoiser Self-Distillation (GDSD)

The authors propose Guided Denoiser Self-Distillation (GDSD), a framework that reframes RL for dLLMs as an off-policy self-distillation problem, thereby bypassing the need for likelihood estimation and avoiding TIM bias.

2.1 Theoretical Foundation

Under reverse-KL regularized RL, the optimal policy $\pi^*$ has a closed-form solution:
$$\pi^*(x_0) \propto \pi_{old}(x_0)^{1-\beta} \cdot \pi_{ref}(x_0)^\beta \cdot \exp(\psi A(x_0))$$
where $A(x_0)$ is the advantage function, $\beta$ is a regularization coefficient, and $\psi$ is a guidance coefficient.

The authors show that this optimal policy induces a guided denoising distribution (a "teacher" denoiser $p^*$) for the masked diffusion model:
$$p^*(x_0|x_t) = p^{ref}_{old}(x_0|x_t) \cdot \exp(\psi A(x_0) - A_t(x_t))$$
where $p^{ref}_{old}$ is a geometric mixture of the old and reference policies, and $A_t(x_t)$ is a normalization constant.

2.2 The GDSD Objective

Instead of optimizing a likelihood-based objective, GDSD directly distills the current denoiser $p_\theta$ from the teacher $p^*$ by matching their logits. The objective minimizes the squared $L_2$ distance between the student and teacher logits:
$$\mathcal{L}_{GDSD} = \mathbb{E} \left[ \left( \log p_\theta(x_0|x_t) - \log p^*(x_0|x_t) \right)^2 \right]$$

2.3 Normalization-Free Reformulation

A major challenge in implementing this is the intractable normalization constant $Z_t$ (or $A_t$) in the teacher distribution. The authors propose two solutions to eliminate this:

1. Direct Matching: Leveraging the translation invariance of the Softmax operator, the constant can be ignored if the student logits match the unnormalized teacher logits up to an additive constant.
2. Token-Level Logit Centralization (TLC): To prevent uncontrolled logit scale shifts during iterative updates, the authors propose centralizing the logits. By subtracting the mean logit over the vocabulary for each token position, the normalization constant is entirely eliminated from the objective.

The final practical objective (with TLC) is:
$$\mathcal{L}_{GDSD} = \mathbb{E} \left[ \left( \log \bar{p}_\theta(x_0|x_t) - \log \bar{p}_{old}(x_0|x_t) - \psi A(x_0) \right)^2 + \beta \left( \log \bar{p}_\theta(x_0|x_t) - \log \bar{p}_{ref}(x_0|x_t) \right)^2 \right]$$
where $\bar{p}$ denotes the logit-centralized model.

3. Key Contributions

1. Reduction to Self-Distillation: The paper establishes that reverse-KL regularized RL for dLLMs is equivalent to distilling a guided denoiser from a self-teacher derived from the closed-form optimal policy.
2. GDSD Framework: A novel, likelihood-free self-distillation method that eliminates TIM bias by design, as it does not rely on importance sampling ratios or ELBO surrogates.
3. Normalization-Free Optimization: The introduction of Token-Level Logit Centralization (TLC) allows for efficient online RL without computing intractable partition functions.
4. Theoretical Unification: The paper demonstrates that existing ELBO-based methods (e.g., wd1, DMPO, SPG, UniGRPO) can be viewed as specific instances of distillation using different divergence measures (Forward-KL or Reverse-KL), but they suffer from data inefficiency or TIM bias that GDSD avoids.

4. Experimental Results

The authors evaluated GDSD on LLaDA-8B and Dream-7B across six benchmarks:

• Mathematical Reasoning: GSM8K, MATH500
• Planning: Countdown, Sudoku
• Coding: HumanEval, MBPP

Key Findings:

• Performance Gains: GDSD consistently outperforms state-of-the-art ELBO-based methods. On the Dream-7B model, GDSD achieved test accuracy improvements of up to +19.6% on planning tasks (Sudoku and Countdown). On LLaDA-8B, improvements ranged from +0.6% to +5% across planning, math, and coding benchmarks.
• Training Stability: GDSD demonstrated significantly more stable training reward dynamics compared to ELBO-based baselines (e.g., SPG, ESPO), which exhibited oscillations or collapse.
• Ablation Studies: The Token-Level Logit Centralization (TLC) component was shown to further stabilize training and, in some cases (like Sudoku), help the model escape local minima to achieve higher final rewards.

5. Significance and Claims

The paper claims that direct denoiser self-distillation offers a more stable and effective alternative to ELBO-based RL for dLLMs. By removing the reliance on likelihood surrogates, GDSD avoids the structural biases inherent in importance-sampling approaches. The results suggest that the performance gains observed are not merely due to hyperparameter tuning but stem from a more principled optimization route that aligns the training objective with the actual inference distribution of dLLMs. The authors conclude that ELBO is not necessary for RL with dLLMs and that its removal leads to more robust alignment and reasoning capabilities.