💬 NLP

Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner

This paper proposes Coevolutionary Continuous Discrete Diffusion (CCDD), a novel framework that unifies continuous and discrete diffusion processes within a single model to overcome the trainability and decoding challenges of continuous diffusion while leveraging its superior theoretical expressivity for enhanced language modeling performance.

Original authors: Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, Dinghuai Zhang

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, Dinghuai Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Two Schools of Thought

Imagine you are trying to teach a robot to write a story. There are currently two main ways to do this, and both have a major flaw:

The "Step-by-Step" Writer (Autoregressive Models): This robot writes one word at a time, from left to right. It's like a person writing a sentence where they can't go back and change the first word once they've written the tenth. It's fast, but it struggles with complex puzzles (like Sudoku) because it can't "think ahead" or "backtrack" easily.
The "Scratch-and-Rewrite" Artist (Diffusion Models): This robot starts with a page full of static noise (or blank spaces) and slowly refines it into a story. It can look at the whole page at once, fix mistakes, and rearrange things.
- The Discrete Artist: Works with actual words. It's good at fixing specific words but sometimes gets stuck in a loop or loses the "big picture" meaning.
- The Continuous Artist: Works with "vibes" or "feelings" (mathematical numbers) instead of specific words. It's incredibly powerful at understanding complex logic and planning, but it's very hard to train because turning those "vibes" back into actual readable words is a messy, difficult process.

The Dilemma: The "Continuous Artist" has a bigger brain (more theoretical power) but is clumsy at the actual job of writing words. The "Discrete Artist" is good at writing words but has a smaller brain for complex reasoning.

The Solution: The "Coevolutionary" Team

The authors of this paper propose a new method called CCDD (Coevolutionary Continuous Discrete Diffusion).

Think of CCDD as a two-person writing team working on the same document at the same time:

Person A (The Discrete Token): This person holds a list of actual words. They are good at knowing the grammar and the specific spelling.
Person B (The Continuous Latent): This person holds a "feeling" or a "concept map" of the story. They understand the deep logic, the plot twists, and the complex reasoning (like solving a math problem).

How they work together:
Instead of Person A and Person B working separately, they are linked.

Training: They are trained together. Person B helps Person A understand the meaning behind the words, while Person A helps Person B ground those abstract "feelings" into real, readable words.
The Magic: Because they are linked, the model gets the best of both worlds. It has the brainpower of the continuous "feeling" space (great for reasoning) and the precision of the discrete word space (great for generating text).

The Analogy: The Architect and the Builder

Imagine building a house:

The Discrete Model is like a Builder who is great at laying bricks but doesn't understand the blueprints. They might build a wall in the wrong place because they can't see the whole house.
The Continuous Model is like an Architect who has a perfect 3D mental model of the house but can't physically lay the bricks. They know exactly where the walls should go, but translating that 3D vision into a pile of bricks is hard.
CCDD is the Architect and Builder working side-by-side. The Architect (Continuous) guides the Builder (Discrete) with a clear vision of the structure. The Builder gives the Architect immediate feedback on what is physically possible. The result? A house that is both structurally sound (good reasoning) and built correctly (good text).

What the Paper Actually Found

The authors tested this new "team" approach on real-world tasks:

Writing Stories: On standard language datasets, their model made significantly fewer mistakes (lower "perplexity") than previous models of the same size. It learned faster, too.
Solving Puzzles: They tested the model on hard logic puzzles like Sudoku, 3-SAT (a complex logic problem), and Countdown (a math game).
- Previous models (like standard "step-by-step" writers) struggled or failed.
- The "Continuous" models were theoretically good but failed in practice.
- CCDD crushed it. It solved these puzzles with high accuracy, often in fewer steps than the competition.
Speed: One of the biggest wins is efficiency. Usually, to get a good result from a diffusion model, you need to take hundreds of tiny steps (like slowly refining a sketch). CCDD can produce high-quality results in just 8 steps, whereas other models needed 256 steps to get a similar quality.

The Takeaway

The paper argues that we don't have to choose between a model that is "smart at reasoning" and a model that is "good at writing words." By creating a system where the "abstract reasoning" and the "concrete words" evolve together (co-evolve), we get a model that is both a brilliant thinker and a skilled writer.

In short: They built a robot that can "think in feelings" to solve hard problems, but instantly translates those feelings into perfect words, making it faster and smarter than anything we've had before.

Technical Summary: Coevolutionary Continuous Discrete Diffusion (CCDD)

1. Problem Statement

Recent advancements in Large Language Models (LLMs) have highlighted limitations in standard autoregressive (AR) architectures, particularly regarding complex reasoning tasks like Sudoku, graph connectivity, and formal language recognition. Theoretical analysis suggests that standard transformers are restricted to the complexity class $TC^0$ even with logarithmic Chain-of-Thought (CoT) steps, whereas looped transformers (LT) and continuous reasoning methods can theoretically reach $TC^1$ or higher.

However, a significant gap exists between theoretical expressivity and empirical performance:

Looped Transformers (LT): While theoretically more expressive, they suffer from poor trainability due to a lack of intermediate supervision, leading to out-of-distribution (OOD) issues during inference when the loop depth differs from training.
Continuous Diffusion Models (CDMs): These models theoretically dominate discrete counterparts in expressivity by operating in continuous latent spaces and generalizing looped transformers. Yet, they empirically underperform discrete diffusion models (DDMs) and AR models. The authors attribute this to practical trainability challenges: large decision spaces, low-quality embeddings, and the combinatorial complexity of decoding continuous representations back into discrete tokens.
Discrete Diffusion Models (DDMs): While highly trainable and effective, they operate in explicit token spaces, potentially losing information during the "logits $\to$ sample $\to$ embed" quantization process at each step.

The core problem is the expressivity-trainability dilemma: models with high theoretical expressivity (CDMs, LTs) are difficult to train, while highly trainable models (DDMs, AR) may lack the necessary latent reasoning capabilities for complex tasks.

2. Methodology: Coevolutionary Continuous Discrete Diffusion (CCDD)

The paper proposes Coevolutionary Continuous Discrete Diffusion (CCDD), a novel language modeling paradigm that unifies continuous and discrete diffusion processes within a single framework.

2.1 Joint Diffusion Process

CCDD defines a joint diffusion process on the union of a discrete token space ( $X = \Omega^L$ ) and a continuous representation space ( $Z = \mathbb{R}^{L \times d}$ ).

Forward Process: The corruption process is factored but applied simultaneously to both modalities.
- Discrete Component: Evolves via a Continuous-Time Markov Chain (CTMC), typically using masked noise (absorbing state) or uniform noise.
- Continuous Component: Evolves via a Stochastic Differential Equation (SDE) or Probability Flow ODE (PF-ODE) in a contextualized embedding space (e.g., derived from pretrained models like Qwen3-Embedding).
- The joint forward kernel is the product of the individual kernels: $q_t(x_t, z_t | x_0, z_0) = q_t^{disc}(x_t | x_0) q_t^{cont}(z_t | z_0)$ .
Reverse Process: A single time-conditioned network $f_\theta(\cdot, t)$ takes the noisy pair $(x_t, z_t)$ as input and outputs modality-specific heads.
- The reverse update is factored: $p_\theta(x_s, z_s | x_t, z_t) = p_\theta^{disc}(x_s | x_t, z_t) p_\theta^{cont}(z_s | x_t, z_t)$ .
- Crucially, while the update rules are factored, the estimation for each modality conditions on both states. The discrete decoder sees the continuous latent $z_t$ , and the continuous decoder sees the discrete tokens $x_t$ . This allows the continuous component to provide rich semantic guidance (implicit reasoning) to the discrete component, while the discrete component provides explicit structure to the continuous component.

2.2 Training Objective

The training loss is a weighted sum of the continuous and discrete losses:
$\mathcal{L}_{CCDD} = \gamma_{cont} \cdot \mathcal{L}_{cont} + \gamma_{disc} \cdot \mathcal{L}_{disc}$

$\mathcal{L}_{cont}$ : Standard MSE loss for predicting noise or the original continuous representation (e.g., $\epsilon$ -prediction).
$\mathcal{L}_{disc}$ : Cross-entropy loss for predicting the original discrete tokens (typically on masked positions).

2.3 Architectural Designs

The paper introduces several architectures for the joint denoising network based on Diffusion Transformers (DiT):

MDiT (Multimodal DiT): Mixes continuous and discrete embeddings before the transformer blocks. It adds negligible parameters and FLOPs compared to discrete-only baselines.
MMDiT: Uses staggered cross-attention and self-attention blocks for multimodal interaction, doubling parameters and FLOPs.
MoEDiT (Mixture of Experts DiT): Maintains separate MLPs for discrete and continuous tokens within a shared attention mechanism, using MoE routing to dynamically process tokens based on their state. This balances parameter efficiency with expressivity.

2.4 Inference Techniques

Classifier-Free Guidance (CFG): The continuous representations act as self-generated guidance for discrete token generation. The model is trained with a probability $p_{drop}$ of zeroing out continuous embeddings (unconditional), allowing for CFG scaling during sampling to improve quality.
Flexible Sampling: CCDD supports both few-step deterministic sampling (via ODE) for efficiency and multi-step stochastic sampling (via SDE) for high-quality generation.

3. Key Contributions

3.1 Theoretical Analysis

Expressivity Hierarchy: The authors prove that Continuous Diffusion Models (CDMs) strictly dominate Discrete Diffusion Models (DDMs) in terms of trajectory-level expressivity. CDMs produce absolutely continuous marginals, whereas DDMs are restricted to finite atomic supports, leading to information loss at every discrete sampling step.
Generalization of Looped Transformers: The paper demonstrates that a continuous diffusion sampler can simulate any looped transformer rollout with the same architecture and parameter count. Conversely, deterministic looped transformers cannot simulate non-degenerate stochastic paths produced by continuous diffusion.
Trainability Analysis: The paper attributes the empirical failure of CDMs to the lack of intermediate supervision in LTs and the decoding complexity/representation deficiency in CDMs. CCDD is proposed as the solution that retains the expressivity of CDMs while gaining the trainability of DDMs through explicit token supervision.

3.2 Empirical Results

Language Modeling: On the LM1B and OpenWebText (OWT) datasets, CCDD significantly outperforms discrete baselines (MDLM, SEDD) and autoregressive models.
- On LM1B, CCDD-MDiT reduced validation perplexity by >25% compared to a discrete baseline with the same parameters (92.1M).
- On OWT, CCDD reduced perplexity by >35% compared to discrete baselines.
- CCDD achieved these results with 25x faster training convergence (40k iterations vs. 1000k for the baseline).
Few-Step Generation: CCDD exhibits superior few-step generation capabilities. A model with only 8 denoising steps outperformed a discrete baseline (MDLM) using 256 steps, demonstrating a 16x acceleration in generation speed without quality loss.
Complex Reasoning: On tasks requiring strong planning and backtracking (Sudoku, 3-SAT, Countdown), CCDD surpassed all previous methods, including autoregressive models, masked discrete diffusion, and looped transformers. For instance, CCDD achieved 100% accuracy on Sudoku with just 2 steps, matching the best looped transformer results.

4. Significance and Claims

The paper claims that CCDD successfully bridges the gap between theoretical expressivity and practical trainability in language modeling. By defining a joint diffusion process, CCDD leverages the strong expressive power of continuous latent spaces (enabling implicit reasoning, planning, and global context retention) while utilizing discrete tokens to ensure trainability and decoding stability.

The authors emphasize that CCDD is not merely a hybrid but a principled unification that:

Generalizes existing paradigms (CDM, DDM, LT) into a single framework.
Retains the ability to perform "latent reasoning" in the continuous space, which is crucial for complex tasks like Sudoku and 3-SAT.
Improves efficiency by allowing flexible inference-time scaling (few-step ODE vs. multi-step SDE) and achieving faster training convergence than looped transformers.
Demonstrates that combining modalities can yield performance gains that exceed the sum of their parts, particularly in downstream benchmarks and complex reasoning scenarios.

The work suggests that future language modeling paradigms should move beyond purely discrete token spaces to leverage continuous latent representations for reasoning, provided that explicit discrete supervision is maintained to ensure effective training.