Technical Summary: LESS: Mutual-Stability Sampling for Diffusion Language Models

Problem Statement

Diffusion Large Language Models (dLLMs) offer a promising alternative to autoregressive (AR) decoding by enabling parallel token updates and bidirectional conditioning through iterative denoising of masked sequences. However, their practical efficiency is currently bottlenecked by sampling procedures that rely on fixed reverse denoising schedules. Standard samplers execute a predetermined number of steps, often wasting computation on positions where predictions have already stabilized while risking premature commitment to unstable tokens. Once a token is unmasked in a diffusion process, it is typically treated as fixed; if an incorrect token is committed early due to a rigid schedule, subsequent reverse steps cannot revise it, leading to quality degradation. Existing training-free adaptive samplers address this partially by using confidence or margin-based criteria, but they often fail to capture the full temporal and distributional stability of a token's predictive trajectory.

Methodology

The authors propose LESS (Less is More), a training-free, model-agnostic adaptive sampler that frames token commitment as a per-position online stopping problem. LESS implements mutual-stability sampling, where a masked position is eligible for unmasking only when three specific stability signals jointly indicate that the prediction has settled.

The Joint Stability Rule

For each masked position $i$ at reverse step $t$, LESS evaluates three signals derived from the model's predictive distribution $p_{t,i}$:

1. Top-1 Predictive Confidence: The probability assigned to the current top-1 token ($w_{t,i}$). A position must exceed a confidence threshold $c$.
2. Top-1 Token Persistence: The current top-1 token must remain unchanged over a recent window of $P$ reverse steps. This prevents committing to a token that has just flipped, even if its current confidence is high.
3. Top-K Inter-Step Jensen–Shannon Divergence (JSD): This is the novel contribution of LESS. It measures the stability of the predictive distribution itself between consecutive reverse steps.
   • Instead of using asymmetric Kullback-Leibler (KL) divergence, LESS uses the symmetric and bounded JSD.
   • To ensure computational efficiency and focus on relevant candidates, the JSD is computed on a coarsened vocabulary consisting of the union of the top-$K$ tokens from the current and previous steps, with the remaining mass collapsed into a residual symbol.
   • The JSD must fall below a threshold $d$.

A position is accepted for unmasking if and only if:
$$\text{accept}_{t,i} = (\text{conf}_{t,i} \ge c) \land \text{persist}_{t,i} \land (\text{JSD}_{t,i} \le d)$$

Unmasking Policy

LESS employs a frontier-first policy for full-sequence diffusion models. It attempts to unmask the leftmost masked position if it is eligible. If not, it scans a look-ahead window to find the first eligible position. If no positions are eligible, the step is skipped. To guarantee termination, a forced-unmasking fallback is triggered after $\Pi$ consecutive skipped steps, though this fallback is not counted as a "stable" commitment.

Key Contributions

1. Formulation of Token Commitment: The paper frames adaptive unmasking in dLLMs as a per-position online stopping problem induced by the irreversibility of unmasking.
2. Top-K Inter-Step JSD: The authors introduce the top-K inter-step Jensen–Shannon divergence as a bounded, symmetric stability signal. This addresses limitations in prior work where confidence alone cannot detect ongoing distributional motion, and KL divergence is directional and unbounded.
3. LESS Sampler: A training-free sampler that combines confidence, token persistence, and inter-step JSD into a joint stability rule. It requires no model retraining and is compatible with existing dLLM checkpoints.
4. Empirical Validation: Comprehensive evaluation across three dLLM families (Dream-7B, LLaDA-8B, LLaDA-1.5-8B) and seven benchmarks covering math, code, and general knowledge.

Results

The evaluation demonstrates that LESS significantly improves the accuracy-efficiency trade-off compared to both fixed-budget decoding and other strong training-free adaptive samplers (Prophet and KLASS).

• Efficiency: Across three model families, LESS uses 72.1% fewer reverse steps on average compared to fixed-budget decoding (Base).
• Accuracy: LESS achieves an average accuracy gain of +0.91 points over the fixed-budget baseline while using fewer steps. It also outperforms Prophet and KLASS in mean accuracy across all model families.
• Task Performance: The gains are most pronounced in math and code generation tasks. For example, on Dream-7B, LESS achieves 65.18% mean accuracy (vs. 63.30% for Base) with only 64.3 steps (vs. 256 for Base).
• Compute Savings: Since each reverse step requires a Transformer forward pass, the reduction in steps translates directly to lower inference compute. On Dream-7B GSM8K, LESS reduces wall-clock latency from 19.45s to 5.16s and reduces the number of forward evaluations by approximately 75% at high step budgets.
• Ablation Studies:
  • Signal Complementarity: No single signal (confidence, persistence, or JSD) performs as well as the joint rule. Confidence is the primary driver, but persistence and JSD provide essential safeguards against premature commitment and distributional drift.
  • Robustness: The method performs consistently across different models and decoding regimes (full-sequence vs. semi-autoregressive blockwise) using a single global configuration $(c=0.75, d=0.040)$.

Significance and Claims

The paper claims that LESS provides a practical, training-free mechanism to accelerate diffusion language model decoding by committing tokens only when their predictions are confident, temporally persistent, and distributionally stable.

The authors emphasize that stability is more than just confidence. A high-confidence prediction can still be unreliable if the top-1 token has recently changed or if probability mass is still shifting among alternatives. By treating stability as a joint property of the current prediction and its recent trajectory, LESS avoids the pitfalls of premature commitment.

The work positions LESS as orthogonal to and complementary with other acceleration methods:

• It complements training-time methods (e.g., distillation) that shorten the reverse process.
• It complements systems methods (e.g., caching, speculative decoding) that reduce per-step latency.
• It acts as a drop-in inference-time policy that can be combined with these methods without modifying the underlying model.

The paper concludes that mutual stability is a viable criterion for robust early token commitment, enabling dLLMs to achieve higher efficiency without sacrificing, and often improving, generation quality. The authors acknowledge that the method exposes a quality-efficiency trade-off controlled by the stability thresholds, which may require tuning for specific models or tasks, though a fixed global configuration proved effective across their diverse evaluation suite.