📊 statistics

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

This paper introduces RePlaid, a likelihood-based continuous diffusion language model that, through architectural alignment with modern discrete models and optimized training, achieves competitive scaling laws, superior generation quality, and a new state-of-the-art perplexity bound, challenging the notion that continuous diffusion is less scalable than discrete approaches.

Original authors: Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Two Ways to Write with AI

Imagine you want to teach a robot to write a story. There are two main ways to do this:

The "Autoregressive" Way (The Serial Writer): This is how most current AI (like the one you are talking to right now) works. It writes one word at a time, like a person typing a sentence. It knows the first word, then guesses the second, then the third. It's very fast and accurate, but it has to do this word-by-word, which can be slow for long texts.
The "Diffusion" Way (The Sculptor): This is a newer, trendier method. Imagine a block of marble that is initially just a messy pile of dust. The AI's job is to slowly chip away the dust, refining the shape until a statue (a perfect sentence) emerges.
- Discrete Diffusion: The sculptor chips away at specific, distinct blocks (like removing a whole word at a time).
- Continuous Diffusion: The sculptor smoothes the surface continuously, like sanding down a rough stone until it's perfect. This method is theoretically smoother and allows for more creative editing, but until now, it was thought to be much slower and less efficient than the "Serial Writer."

The Problem: The "Speed Bump"

For a long time, researchers believed that the Continuous Diffusion method (the smooth sculptor) was fundamentally broken for large-scale language tasks. They thought it required 64 times more computing power than the standard "Serial Writer" just to get the same quality of writing. Because of this "speed bump," everyone assumed continuous diffusion would never be able to scale up to build massive, powerful AI models.

The Solution: RePlaid (The "Re-tuned" Sculptor)

The authors of this paper decided to test this belief. They took an existing continuous diffusion model called Plaid and gave it a major makeover, renaming it RePlaid.

Think of Plaid as an old, slightly rusty sculpture tool. It had the right idea, but it wasn't built with the modern tools that the "Serial Writers" were using. The authors didn't invent a new tool; they just re-aligned the old one. They:

Swapped out the internal gears to match the modern "Serial Writer" architecture (using the same "Transformer" engine).
Fixed the way it handles the "noise" (the dust) it adds to the text.
Made sure the math used to train it was perfectly optimized.

The Results: Closing the Gap

When they tested RePlaid against the best "Serial Writers" and "Discrete Diffusion" models using the exact same rules and budget:

The Gap Shrunk: The computing power gap dropped from a massive 64x down to just 20x. While it's still not as fast as the "Serial Writer," it is no longer "impossible." It's now competitive.
Better Quality: RePlaid achieved a new record for the best writing quality (measured by "perplexity," which is like a score for how confused the model is) among all continuous diffusion models. It actually wrote better than some of the "Discrete Diffusion" models.
Efficiency: It managed to do this while using fewer parameters (the "brain size" of the model) than its competitors.

Why Did It Work? (The Secret Sauce)

The paper explains two main reasons why this "re-tuned" sculptor works so well:

1. The "Evenly Distributed Difficulty" (The Noise Schedule)
Imagine you are trying to clean a dirty window. If you try to scrub the whole window at once, you might get tired or miss spots.

Old Way: Some models tried to scrub the window in a weird, uneven pattern, making some parts very hard to clean and others too easy.
RePlaid's Way: By using a specific mathematical trick (optimizing the "noise schedule"), RePlaid naturally figured out how to spread the cleaning effort evenly across the whole window. It didn't need a human to tell it how to do this; the math of "likelihood" (a measure of how probable the text is) forced it to find the most efficient path automatically.

2. The "Organized Clay" (Embedding Geometry)
Imagine the AI is trying to mold clay into shapes.

Old Way: Some models treated the clay as a chaotic mess, where every piece of clay was scattered randomly. This made it hard to find the right shape.
RePlaid's Way: RePlaid learned to organize the clay into neat, structured piles (low-rank geometry). It learned that certain "clay pieces" (words) belong together in specific patterns. This structure made it much easier for the model to predict the next word, leading to much better writing.

The Bottom Line

This paper challenges the idea that "smooth" continuous diffusion is too slow for big language models. By simply re-aligning an existing model with modern standards, the authors showed that continuous diffusion can scale up and compete with the best models we have today.

They didn't just make a slightly better model; they proved that the method itself (continuous diffusion) is viable and powerful, provided you tune it correctly. It's like discovering that an old car engine wasn't broken, it just needed the right fuel and a tune-up to run as fast as a Ferrari.

Technical Summary: Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

Problem Statement

While diffusion models have gained significant traction in language modeling, continuous diffusion has historically been viewed as less scalable than discrete diffusion and autoregressive (AR) models. Previous studies, specifically the original Plaid work, reported a substantial compute gap (64×) for continuous diffusion to match the likelihood of AR models. This performance disparity has led to a prevailing narrative that continuous diffusion is inherently unscalable for large-scale language tasks. The core challenge addressed by this paper is determining whether this gap is an intrinsic limitation of continuous diffusion or a consequence of incompatible experimental setups and scaling analyses.

Methodology

The authors introduce RePlaid, a revised version of the Plaid likelihood-based continuous diffusion language model (DLM), designed to align strictly with modern discrete DLM practices.

1. Architectural Alignment

To ensure a fair comparison, RePlaid adopts the exact architecture and training protocols used by state-of-the-art discrete models (MDLM and Duo) as defined by Sahoo et al. [51]. Key alignment features include:

Transformer Backbone: Utilization of a Diffusion Transformer (DiT) with bidirectional attention, RoPE (Rotary Position Embeddings), and AdaLN-Zero modulation.
Optimization: Identical hyperparameters (AdamW, cosine learning rate schedule, weight decay) and dataset (SlimPajama) as the discrete baselines.
Numerical Precision: Removal of FP32-only logit heads used in the original Plaid to eliminate numerical confounders.
Embeddings: Retention of Plaid's core innovation: learnable, low-dimensional ( $d_e=16$ ), length-normalized embeddings, which are significantly cheaper to compute than high-dimensional one-hot embeddings used in other continuous models.

2. Unified Scaling Analysis

The paper performs the first IsoFLOP analysis comparing continuous and discrete DLMs. Models of varying sizes are trained up to fixed compute budgets ( $C \in \{6 \times 10^{18}, \dots, 1 \times 10^{20}\}$ FLOPs) to identify compute-optimal model sizes and validation losses. This allows for a direct comparison of scaling laws between RePlaid, MDLM, Duo, and AR baselines.

3. Theoretical Investigation

The authors analyze the Evidence Lower Bound (ELBO) objective to understand why likelihood-based training succeeds where heuristic methods (like cross-entropy or flow matching) might struggle. They investigate:

The relationship between the learned noise schedule and the distribution of denoising difficulty.
The geometric properties of the learned token embeddings under likelihood optimization versus cross-entropy optimization.

Key Contributions

1. RePlaid and Benchmarking

The authors introduce RePlaid, which achieves a state-of-the-art perplexity (PPL) of 22.1 on OpenWebText among continuous DLMs. This outperforms the original Plaid (24.4), concurrent continuous models like LangFlow (32.2), and even surpasses the discrete model Duo (25.2), while approaching the performance of MDLM (23.1).

2. Unified Scaling Laws

In a unified setting, RePlaid demonstrates that continuous diffusion scales competitively with discrete diffusion:

Compute Gap: RePlaid requires only a 20× compute overhead compared to AR models to match their validation loss (down from the previously cited 64×).
Parameter Efficiency: RePlaid achieves its compute-optimal frontier with 3.4× fewer parameters than AR baselines and 1.8× fewer parameters than MDLM and Duo, while outperforming Duo's loss.
Over-training Regime: In the over-trained regime (training beyond compute-optimal budgets), RePlaid consistently outperforms MDLM, suggesting superior efficiency for inference-heavy scenarios.

3. Theoretical Insights

The paper provides theoretical explanations for RePlaid's success:

Automatic Noise Scheduling: Optimizing the noise schedule to minimize the ELBO's variance naturally yields a linear cross-entropy (information loss) schedule over time. This evenly distributes denoising difficulty without requiring heuristic time reparameterizations used in other works.
Embedding Geometry: Optimizing embeddings via the likelihood bound creates structured, low-rank geometries in the latent space. In contrast, adding cross-entropy terms or using fixed embeddings disrupts this structure, leading to token dispersion and higher perplexity. The paper demonstrates that learning the embedding geometry is the primary driver of performance gains.

Results

Likelihood: On OpenWebText, RePlaid (with self-conditioning) achieves a PPL of 22.1, the best among continuous DLMs and superior to Duo (25.2) and LangFlow (32.2).
Generation Quality: Using standard DDPM sampling, RePlaid generates text with higher quality (lower GenPPL) than Duo and other continuous models across all sampling steps.
Diversity and Quality Trade-off: RePlaid maintains a favorable GenPPL-entropy frontier. While self-conditioning can lead to lower entropy at very high sampling steps, RePlaid generally outperforms discrete models (Duo, MDLM) and other continuous models (FLM, LangFlow) in the MAUVE metric, which captures both quality and diversity.
Sampling Efficiency: Higher-order ODE solvers (e.g., DPM-Solver++) improve low-step sampling but converge to the performance of the first-order DDIM solver at high step counts, where the ancestral DDPM sampler performs best.

Significance and Claims

The paper claims that continuous diffusion, when trained via a principled likelihood-based objective (ELBO) and aligned with modern architectural standards, is a highly competitive and scalable alternative to discrete DLMs.

The authors challenge the narrative of inherent unscalability by showing that the previously reported 64× compute gap was largely an artifact of incompatible scaling analyses and suboptimal setups. By narrowing the gap to 20× and demonstrating superior parameter efficiency, the work suggests that continuous diffusion offers a viable path toward efficient text generation, particularly leveraging its potential for controllability and acceleration via ODE solvers.

The paper modestly acknowledges that a 20× compute gap remains compared to AR models and identifies learnable token embeddings as the critical factor for RePlaid's success. Future work is suggested to explore richer contextual encoders and the potential for latent-space inference-time scaling, but the current contribution is strictly limited to establishing the competitive scaling laws and theoretical underpinnings of likelihood-based continuous diffusion.