💬 NLP

Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

This paper introduces Efficient-DLM, a framework that converts pretrained autoregressive models into highly efficient diffusion language models by employing a block-wise attention pattern and position-dependent token masking, thereby achieving superior accuracy and throughput compared to state-of-the-art models.

Original authors: Yonggan Fu, Lexington Whalen, Zhifan Ye, Xin Dong, Shizhe Diao, Jingyu Liu, Chengyue Wu, Hao Zhang, Enze Xie, Song Han, Maksim Khadkevich, Jan Kautz, Yingyan Celine Lin, Pavlo Molchanov

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yonggan Fu, Lexington Whalen, Zhifan Ye, Xin Dong, Shizhe Diao, Jingyu Liu, Chengyue Wu, Hao Zhang, Enze Xie, Song Han, Maksim Khadkevich, Jan Kautz, Yingyan Celine Lin, Pavlo Molchanov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Traffic Jam vs. The Highway

Imagine two ways to write a story:

The Old Way (Autoregressive/AR): This is like a single-lane road. You write one word, then the next, then the next. You can't write the second word until the first one is finished. It's very accurate, but it's slow because you have to wait in line for every single word.
The New Way (Diffusion/DLM): This is like a multi-lane highway. You can write many words at the same time (in parallel). It should be much faster. However, in the past, these "highways" were bumpy, confusing, and often produced lower-quality stories than the single-lane road. They were also very expensive to build (train).

The Problem: Scientists wanted the speed of the highway with the quality of the single-lane road, but they couldn't figure out how to build it without starting from scratch (which costs a fortune).

The Solution: The authors of this paper figured out how to take an existing, high-quality "single-lane" model and transform it into a "highway" model that is both fast and accurate. They call this new family of models Efficient-DLM.

How They Did It: Three Key Tricks

The paper identifies three main "rules" they had to follow to make this transformation work.

1. The "Clean Context" Rule (The Block-Way)

The Old Mistake: Previous attempts tried to let the model look at the entire sentence at once, even the parts that were still messy or missing. Imagine trying to solve a puzzle where half the pieces are missing, and you are allowed to look at the empty spaces as if they were filled. This confused the model and made it forget what it learned before.
The Fix: The authors decided to break the sentence into small blocks (like chapters in a book).
- The model looks at the previous chapters, which are already finished and clean.
- It only tries to fix the current chapter, which is messy.
- Analogy: Think of it like a construction crew. They don't try to fix the whole building at once. They finish the foundation and the first floor (clean context) before moving up to fix the second floor (the messy block). This keeps the structure stable and allows them to use a "shortcut" (called KV caching) to remember what they've already built, making the process much faster.

2. The "No Token Shift" Rule (Stop Guessing the Next Step)

The Old Mistake: When converting the old model, researchers used to make the model guess the next word, just like the old single-lane model did.
The Fix: The authors found that the model doesn't need to guess the "next" word. It just needs to fix the missing words right in front of it.
Analogy: Imagine you are editing a document.
- Old Way: You read a sentence, then try to predict the very next sentence before you even finish editing the current one.
- New Way: You just focus on filling in the blank spaces in the current paragraph. It turns out, fixing the blanks is easier and more accurate than trying to predict the future.

3. The "Position-Dependent Masking" Rule (The Left-to-Right Bias)

The Problem: When the model is being trained, it used to hide words randomly (like picking lottery numbers). But when the model is actually used (inference), it naturally tends to finish words from left to right, just like humans read. This created a mismatch: the training didn't look like the real test.
The Fix: The authors changed the training so that the model is more likely to hide words at the end of a block, rather than the beginning.
Analogy: Imagine a teacher grading a test.
- Old Training: The teacher covers up random questions on the page.
- New Training: The teacher realizes that students usually get stuck on the last few questions of a section. So, the teacher specifically practices covering up the end of the section. This prepares the model for the real world, where it has to finish the sentence from left to right.

The Results: Speed and Smarts

By following these rules, the authors created the Efficient-DLM family (sizes 1.5B, 4B, and 8B).

Speed: They are significantly faster than the best existing models. For example, their 8B model is 4.5 times faster than a competitor called "Dream" and 2.7 times faster than "Qwen3 4B."
Accuracy: Despite being faster, they are actually more accurate on math, coding, and reasoning tasks than the models they beat.
Flexibility: Because they can generate multiple words at once, you can tune them. If you need speed, you generate many words at once. If you need extreme precision, you generate fewer words at once. It's like having a car that can switch between "Racing Mode" and "Cruising Mode" instantly.

Summary

The paper says: "We took a slow, accurate model and taught it to run on a highway. We did this by organizing the work into clean blocks, stopping it from guessing the future, and training it to focus on the end of sentences. The result is a model family that is both faster and smarter than the current state-of-the-art."

They also noted that because these models can look at text from both directions (bidirectional), they are surprisingly good at understanding the "meaning" of text for search and embedding tasks, outperforming the older single-direction models.

1. Problem Statement

While Autoregressive (AR) Large Language Models (LLMs) achieve state-of-the-art accuracy, they suffer from a sequential decoding bottleneck (generating one token at a time), which limits throughput and hardware utilization, especially in memory-constrained scenarios.

Diffusion Language Models (dLMs) offer a promising alternative by enabling parallel, non-autoregressive generation through iterative denoising. However, existing dLMs face two critical challenges:

Training Efficiency: Training dLMs from scratch is prohibitively expensive because they must learn all possible token permutations, whereas AR models only learn left-to-right dependencies.
Performance Gap: When converting pretrained AR models to dLMs, existing methods (often using fully bidirectional attention) suffer from significant accuracy drops and fail to fully leverage Key-Value (KV) caching, resulting in slower inference speeds than AR models in practice.

The core question addressed by this work is: How can we efficiently convert pretrained AR models into dLMs that preserve high task accuracy while achieving significantly faster parallel generation speeds?

2. Methodology: The Efficient-DLM Framework

The authors propose a systematic approach to convert pretrained AR models into efficient dLMs through Continuous Pretraining. Their methodology focuses on three key pillars:

A. Attention Pattern Optimization

The paper identifies that the attention pattern is the most critical factor in AR-to-dLM conversion.

Limitation of Fully Bidirectional Attention: Existing methods corrupt the entire sequence and allow all tokens to attend to each other. This causes large weight drifts from the original AR model, breaks causality, and complicates KV caching.
Proposed Solution: Block-wise Attention with Clean Context:
- Block-wise Causality: The sequence is divided into blocks. Attention is causal across blocks (a block only attends to previous blocks) but bidirectional within a block.
- Clean Context Conditioning: During training, a noisy block is conditioned on the clean (fully decoded) context of all preceding blocks, rather than corrupted context.
- Benefit: This design mimics the test-time inference process (where previous blocks are fully decoded), minimizes weight drift from the pretrained AR model, and enables native KV caching.

B. Position-Dependent Token Masking

The authors identify a training-test gap in token masking strategies.

The Gap: Existing dLMs use uniform masking (randomly masking tokens regardless of position). However, during inference with confidence-based sampling, denoising naturally exhibits a left-to-right tendency (tokens are decoded sequentially from left to right as confidence builds).
Proposed Solution: A position-dependent masking strategy where the probability of masking a token increases as the token position moves toward the end of the block (especially at low noise levels).
- This is controlled by a hyperparameter $\beta$ (or half-life ratio $\lambda$ ).
- Goal: To force the model to learn the "harder" cases (later tokens in a block) more effectively during training, aligning the training distribution with the inference behavior.

C. Training Dynamics and Token Shift

Token Shift: The paper finds that the "token shift" mechanism (predicting the next token rather than the masked token itself, common in some AR-to-dLM conversions) is unnecessary and potentially harmful. Removing it improves accuracy.
Training Budget: While low-cost training (~~10B tokens) recovers most AR accuracy, extended training (~~100B+ tokens) significantly improves likelihood estimation, enabling more aggressive parallel decoding (higher throughput) without sacrificing accuracy.

3. Key Contributions

Efficient-DLM Model Family: The release of a new family of models (1.5B, 4B, 8B) converted from Qwen2.5 and Qwen3 AR models, which outperform both SOTA AR and dLM baselines.
Block-wise Attention with Clean Context: A novel training scheme that preserves AR weight distributions and enables KV caching, solving the accuracy-efficiency trade-off.
Position-Dependent Masking: A strategy to bridge the training-inference gap, significantly boosting performance under aggressive parallel decoding settings.
Comprehensive Analysis: Systematic studies on block sizes, attention patterns, and training dynamics, providing actionable guidelines for the community.

4. Experimental Results

The Efficient-DLM models were benchmarked against SOTA AR models (Qwen3, Llama3.2) and dLMs (Dream, LLaDA) across 12 tasks (Math, Coding, Reasoning, etc.).

Accuracy vs. Throughput:
- Efficient-DLM 8B achieves +5.4% higher accuracy than Dream 7B with 4.5× higher throughput.
- Compared to Qwen3 4B, Efficient-DLM 8B achieves +2.7% higher accuracy with 2.7× higher throughput.
- It outperforms Qwen3 8B in accuracy while offering significantly better throughput.
Flexibility: A single Efficient-DLM model can dynamically balance accuracy and speed by adjusting the confidence threshold during inference, offering a "one-for-all" solution for different deployment scenarios.
Text Embeddings: Due to their bidirectional nature, Efficient-DLMs significantly outperform AR models of the same size on text embedding tasks (MTEB benchmark), showing a ~7-10% average improvement.
Ablation Studies:
- Switching from bidirectional to block-wise attention with clean context improved average accuracy by ~19%.
- Adding position-dependent masking improved accuracy by ~2.6% under aggressive parallel decoding.

5. Significance and Impact

Paradigm Shift: This work demonstrates that dLMs can be faster and more accurate than AR models if the conversion process is optimized correctly. It challenges the notion that dLMs are inherently slower or less accurate.
Practical Deployment: By enabling KV caching and parallel decoding, Efficient-DLMs make high-throughput LLM inference feasible on limited hardware, addressing a major bottleneck in real-world applications.
Scalability: The findings provide a scalable recipe for converting existing massive AR models (like Qwen, Llama) into efficient dLMs without the need for training from scratch, drastically reducing the cost of adopting diffusion-based generation.
Future Directions: The paper suggests that dLMs are not just alternatives for generation but superior candidates for tasks requiring bidirectional context (like embeddings) and planning.

In summary, Efficient-DLM establishes a new state-of-the-art for diffusion language models by solving the efficiency and accuracy bottlenecks through careful architectural design (block-wise attention) and training strategy alignment (position-dependent masking).