🤖 machine learning

Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokens

By instrumenting DiffusionGemma 26B, this study reveals that its token commitment process is neither purely parallel nor strictly sequential but rather a regime-dependent, partial left-to-right bias that forms large simultaneous batches, demonstrating that perceived decoding order is often an artifact of measurement granularity rather than a fixed architectural property.

Original authors: Ali Asaria, Tony Salomone, Deep Gandhi

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Asaria, Tony Salomone, Deep Gandhi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant, chaotic art studio where a robot is trying to paint a picture (or write a story) on a large canvas.

In the old way of doing things (called "Autoregressive" models), the robot paints strictly from left to right, like reading a book. It finishes one word, then the next, one by one.

In the new "Diffusion" way, the robot starts with a canvas full of blank, masked squares. It's supposed to look at the whole picture at once and fill in many squares simultaneously. The big question researchers asked was: Does this new robot actually paint the whole canvas at once, or does it secretly sneak back to the left and paint word-by-word anyway?

The authors of this paper decided to put a tiny camera inside the robot's brain to watch exactly how it paints. They studied a specific model called DiffusionGemma.

Here is what they found, explained simply:

1. It's Not "All at Once," But It's Not "One by One" Either

The robot doesn't paint the whole picture in a single magical flash, nor does it paint strictly left-to-right like a human reading a book.

Instead, it paints in large, messy batches.

The Analogy: Imagine a teacher grading a stack of 20 essays. A strict left-to-right robot would grade Essay 1, then Essay 2, then Essay 3.
What DiffusionGemma does: It grabs a handful of essays (say, 15 of them), grades them all at the same time, puts them down, grabs another handful, and grades those.
The Result: Within that handful, the robot doesn't care which essay it graded first. It's a "batch" of work. Because it does these big batches, the order looks a little bit messy, but there is still a slight tendency to move from left to right as it goes through the batches.

2. The "16-Block" Myth

Earlier, people thought the robot worked in perfect blocks of 16 words (like a train with 16-carriage cars).

The Finding: The researchers tested this by looking at the robot's work in groups of 4, 8, 16, 32, and 64 words.
The Truth: The robot didn't suddenly snap into a perfect pattern at 16. The pattern just got smoother and more "left-to-right" as the groups got bigger. The number 16 wasn't a special rule for the robot; it was just a number the researchers picked to look at the data. The robot is more fluid than that.

3. The "JSON" Exception

The robot behaves differently depending on what it's asked to do.

For Stories, Code, and Math: It follows that "messy batch" left-to-right pattern.
For Structured Data (like JSON): It acts almost like a random scatter. If you ask it to fill out a form with specific keys and values, it doesn't care about the order at all. It fills them in wherever it feels like it, with no left-to-right bias.

4. The "Confidence" Check

The robot has a "confidence meter" (it measures how sure it is about a word before locking it in).

On Math Problems: If the robot is very confident (low "entropy"), it is usually right. If it's unsure, it's more likely to be wrong. The confidence meter works well here.
On Factual Questions: The confidence meter is useless. The robot can be super confident and still get the fact wrong. It's like a student who is 100% sure they know the capital of France is "London" because they are confident, not because they are right.

5. The "Early Finish" Surprise

The robot was given a budget to take up to 48 "steps" (rounds of thinking) to finish a task.

The Reality: It almost never uses the full budget. It usually finishes in a tiny burst of just 3 to 17 steps. It locks in its answers very quickly, often when it is already extremely confident, long before it runs out of time.

6. Is it as Good as the Old Robot?

When the researchers compared this new Diffusion robot to the old "left-to-right" robot (Gemma-4), they found they are equally good at getting the answers right on math, facts, and JSON tasks. The new robot just gets there in a different, slightly messier way.

Summary

The paper concludes that DiffusionGemma is neither purely parallel nor purely sequential. It's a hybrid:

It works in large, simultaneous batches.
It has a weak tendency to move left-to-right, but only when you look at it from a distance.
It finishes early and stops thinking once it feels confident.
Its "confidence" is a good predictor of success for math, but a bad predictor for facts.

The authors emphasize that we need to stop assuming these models work like perfect blocks or perfect lines. They are more like a group of painters working in teams, sometimes overlapping, sometimes finishing early, and sometimes ignoring the order entirely depending on the job.

Technical Summary: Neither Parallel Nor Sequential

Problem Statement
Open diffusion language models (DLMs) are theoretically characterized as parallel, non-autoregressive decoders that commit tokens iteratively by denoising a masked canvas. However, the actual order in which a deployed checkpoint finalizes tokens is rarely measured empirically. It remains unclear whether these models exhibit strict parallelism, clean block-autoregression, or a hybrid behavior. This ambiguity obscures the true latency advantages and decoding dynamics of the paradigm.

Methodology
The authors conducted an inference-only interpretability study of the public checkpoint google/diffusiongemma-26B-A4B-it, a 25.2B parameter (3.8B active) masked discrete-diffusion mixture-of-experts model built on Gemma 4.

Instrumentation: The study hooked the model's EntropyBoundSampler.accept_canvas method within the Hugging Face transformers library. This allowed the authors to record, for every generation, exactly which canvas positions were committed at each accept-call and the per-position entropy of the committing logits.
Experimental Design: The evaluation utilized a 686-prompt probe suite across six regimes: Math (GSM8K), Code Completion (HumanEval), Code Synthesis (MBPP), Factual Recall, Open-ended Instructions, and Constrained JSON.
Strengthening Run: To ensure robustness, the authors performed a "strengthening run" using 5 random seeds and 20 prompts per regime (100 traces per regime, 600 total generations).
Metrics:
- Order: The primary metric was the tie-aware Kendall rank correlation ( $\tau_b$ ) between a token's commit-call index and its left-to-right position.
- Granularity: A block- $\tau_b$ sweep was performed over bin sizes $B \in \{4, 8, 16, 32, 64\}$ to test for privileged architectural blocks.
- Confidence-Correctness: The predictive power of commit confidence (negative entropy) on correctness was measured via AUROC and reliability curves, strictly within-regime to avoid Simpson's paradox.
- Controls: A synthetic block-sequential control (commit time = position // 16) was used to bound the expected $\tau_b$ of a clean block-autoregressive process.

Key Findings

Partial, Granularity-Dependent Left-to-Right Bias:
DiffusionGemma does not exhibit clean block-autoregression. Instead, it shows a partial left-to-right bias that is moderate at the token level and rises smoothly toward sequential behavior as analysis bin size increases.
- Token-level $\tau_b$ : Ranges from 0.43 (Code) to 0.60 (Code Synthesis), significantly below the +1 of strict autoregression and well below the 0.94–0.96 of the block-sequential control.
- Bin Size Sensitivity: Block- $\tau_b$ rises monotonically with bin size (e.g., for Math: 0.59 at $B=4$ to 0.91 at $B=64$ ). Crucially, there is no jump at $B=16$ , indicating that 16 is an analyst-chosen bin, not a privileged architectural block size.
- Sub-Block Disorder: The gap between real token order and the block-sequential control confirms genuine disorder within blocks.
Large Commit Batches and Unresolved Order:
Commitment occurs in large batches. On average, 13–26 content tokens commit per accept-call. A significant fraction of token pairs (up to 0.72 in JSON) commit in the same call, meaning their relative order is unresolved. The measured $\tau_b$ reflects an accept-call-level tendency rather than a strict within-step ordering.
Regime Dependence:
- Structured JSON: The constrained JSON regime is approximately order-independent ( $\tau_b \approx -0.044$ ), distinct from the moderate positive bias of prose, code, and math regimes.
- Confidence-Correctness Discrimination: Commit confidence predicts correctness on Math (GSM8K AUROC 0.749, CI excluding chance) but is null on Factual Recall (AUROC 0.471, CI includes 0.5). Pooling these regimes creates a Simpson's paradox artifact, reversing the trend.
Aggressive Early Stopping:
Despite a 48-step denoising budget, generations finish in a small burst of accept-calls (3.3–17.1 calls). The entropy at commit is far below the 0.1 bound (0.002–0.012 nats), indicating the model commits content well before exhausting its step budget.
Accuracy:
On scorable regimes (Math, Factual, JSON), DiffusionGemma's accuracy is comparable to its matched autoregressive sibling, Gemma-4 26B-A4B, though the study was not powered for an equivalence test.

Significance and Contributions
The paper claims to revise the understanding of diffusion decoding from "secretly left-to-right" or "structured out-of-order" to a more nuanced reality: a partial, granularity-dependent left-to-right bias that is regime-specific and characterized by large, unresolved commit batches.

Methodologically, the paper contributes a framework for measuring decoding order that addresses specific pitfalls:

EOS Padding Artifacts: Naive metrics are dominated by early commitment of padding tokens; restricting analysis to content positions is essential.
Simpson's Paradox: Pooling confidence-correctness data across regimes with different base accuracies and entropy scales yields invalid conclusions.
Commit Non-Monotonicity: The study accounts for "un-accept" events (re-masking) by defining order as first-acceptance order, verifying robustness on the strictly monotone subset.
Block-Size Sensitivity: It demonstrates that perceived block sizes are often artifacts of the analyst's binning choice rather than model architecture.

The authors conclude that the instrumentation lens—observing a shipped sampler's per-step commitments rather than proposing a new strategy—provides a necessary tool for auditing the actual behavior of diffusion language models.