CORA-Diff: Confidence-Oriented Residual Acceptance for Efficient Diffusion Language Model Inference

Technical Summary: CORA-Diff

Problem Statement
Diffusion Language Models (DLMs) offer the potential for parallel token generation, contrasting with the sequential nature of autoregressive models. However, practical DLM decoders often rely on a fixed denoising horizon, executing dense Transformer passes for every step until the horizon is reached. This approach leads to significant computational redundancy, as many token predictions stabilize early in the trajectory, yet the decoder continues to process them in dense passes until the block is fully resolved. Existing acceleration methods attempt to mitigate this but frequently rely on learned acceptance filters, modified decision scores (logits), explicit dependency models, or specialized system mechanisms (e.g., cache reuse). These approaches often require training auxiliary components or altering the backbone, limiting their generality and ease of deployment.

Methodology: CORA-Diff
The authors propose CORA-Diff (Confidence-Oriented Residual Acceptance for Diffusion Language Models), a training-free method designed to identify and accept "residual" positions—those left unresolved by the original transfer rule—based solely on native trajectory signals.

• Core Mechanism: CORA-Diff preserves the original backbone and the original transfer rule. It introduces a lightweight gating mechanism applied only to positions that the original rule leaves unresolved.
• Dual-Signal Gating: A token at an unresolved position is accepted early if it satisfies two conditions derived from the current decoding trajectory:
  1. Confidence: The top-1 prediction probability ($c^{(s)}_i$) exceeds a threshold $\delta_p$.
  2. Persistence: The same top-1 prediction has persisted for a required number of consecutive executed steps ($m^{(s)}_i \ge m$).
• Residual Acceptance: Accepted tokens are fixed to their current prediction and remain visible as context for subsequent steps, but they are excluded from further denoising. The block terminates early once all positions are resolved (either by the original rule or the CORA gate), skipping the remaining dense forward passes.
• Theoretical Basis: The method is grounded in the hypothesis that high-confidence, persistent predictions are statistically likely to match the fixed-horizon dense endpoint. The authors derive a conditional drift-tail bound, showing that under a "persistence-drift separation" assumption, the probability of disagreement with the dense endpoint decreases exponentially with the persistence window $m$.
• Calibration: Thresholds $(\delta_p, m)$ are selected on a separate calibration subset (e.g., 1,000 GSM8K prompts) to minimize the step ratio while constraining task score degradation and output disagreement. Once selected, these thresholds are frozen for all subsequent evaluations and cross-backbone transfers.

Key Contributions

1. Training-Free Residual Acceptance: The introduction of a method that accelerates DLM inference without requiring learned acceptance models, logit modifications, or backbone changes. It relies exclusively on native confidence and persistence signals.
2. Theoretical Formalization: The formalization of early acceptance through dense-endpoint disagreement, providing a margin-based sufficient condition for agreement and deriving a conditional drift-tail bound that explains the reliability of persistent predictions.
3. Efficient Block-Level Termination: Demonstrating that this lightweight rule enables the early termination of decoding blocks, significantly reducing the number of executed dense steps.

Experimental Results
The method was evaluated on the LLaDA-8B-Instruct model across four benchmarks (GSM8K, MATH, HumanEval, MBPP) and tested for transferability to the Dream model.

• Runtime Performance: CORA-Diff achieved the lowest measured runtime in all eight task–length settings (combinations of 256/256 and 1024/1024 denoising horizons).
  • Under the fixed-horizon mechanism-isolation protocol, it achieved speedups of 4.34×–9.49× (256/256) and 10.68×–13.14× (1024/1024).
  • Compared to the closest runtime baseline (Learn2PD), CORA-Diff was 9.1%–30.9% faster.
  • In EOS-aware deployment (where decoding stops only after a committed end-of-sequence token), it provided incremental speedups of 2.70× (GSM8K) and 3.32× (HumanEval) over EOS-aware dense decoding.
• Task Quality: Task scores matched or exceeded dense decoding in five of the eight settings. The largest observed drop in performance was 1.22 points.
• Transferability: Without retuning, CORA-Diff transferred to the Dream backbone, achieving speedups of 3.18×–3.53× with negligible task score changes (at most 0.006).
• Ablation: Experiments confirmed that combining confidence and persistence yields better trade-offs than using either signal alone.

Significance and Claims
The paper claims that native confidence and persistence signals are sufficient in practice to identify residual positions that will match the deterministic dense endpoint. By leveraging these signals, CORA-Diff removes substantial repeated denoising computation through a simple, training-free gate. The work demonstrates that significant inference acceleration can be achieved without the overhead of auxiliary models, system-specific optimizations, or modifications to the pre-trained model, while preserving task quality. The authors position this as a complementary approach to existing accelerators, focusing on block completion rather than just token acceptance or per-step system optimizations.