🤖 machine learning

Path-Dependent Denoising: A Non-Conservative Field Perspective on Order Collapse in Diffusion Language Models

This paper introduces a theoretical framework based on order-induced pseudo-joints and local denoising circulation to diagnose and quantify the path-dependence in Diffusion Language Models, revealing that their tendency to collapse into autoregressive-like trajectories stems from the incompatibility of local denoising conditionals rather than estimation errors.

Original authors: Jeonseong Kim

Published 2026-05-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeonseong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Jigsaw Puzzle" Problem

Imagine you have a jigsaw puzzle that is completely scrambled. Your goal is to put the pieces back together to form a picture.

Old Way (Autoregressive Models): You must start at the top-left corner and fill in the puzzle piece by piece, strictly moving from left to right, row by row. You can't place piece #50 until piece #49 is done. This is slow but reliable.
New Way (Diffusion Language Models): You are allowed to grab any piece you want and place it anywhere, all at the same time. You could fill in the sky, then the grass, then the middle of the road, all in parallel. This sounds much faster.

The Problem: Even though the new method allows you to work in any order, in practice, it often gets confused. If you try to fill in too many pieces at once, the picture comes out wrong. The model seems to "drift" back to doing things one by one, like the old way, because it's afraid to work in parallel.

This paper asks: Why does the model get confused when we let it work in any order?

The Core Concept: "Local Rules" vs. "The Big Picture"

The authors argue that the confusion happens because the model is good at giving local instructions but bad at ensuring those instructions fit together into a consistent big picture.

The Analogy: The Committee of Experts

Imagine you are trying to guess the ending of a movie. You have a team of experts (the model) who can tell you what happens next.

Scenario A (Order 1): You ask Expert 1, "What happens after the hero opens the door?" They say, "He sees a dragon." Then you ask Expert 2, "What happens after he sees the dragon?" They say, "He fights the dragon."
- Result: A consistent story.
Scenario B (Order 2): You ask Expert 2 first, "What happens after the hero opens the door?" They say, "He sees a dragon." Then you ask Expert 1, "What happens after he sees the dragon?" They say, "He runs away."
- Result: A contradictory story.

In a perfect world, the experts should agree on the story regardless of who you ask first. But in these AI models, the "experts" (the local predictions) often disagree depending on the order you ask them.

The "Curl": Measuring the Confusion

The paper introduces a mathematical tool called "Curl" (borrowed from physics, where it measures how much a fluid swirls).

Zero Curl: The experts agree perfectly. If you ask them in Order A or Order B, they tell the same story. The puzzle pieces fit together no matter how you place them.
High Curl: The experts are inconsistent. Asking them in a different order changes the story. This "swirl" or "confusion" is what causes the model to fail when trying to work in parallel.

The authors prove that if you have a "swirl" (non-zero curl) between any two steps, the whole process becomes path-dependent. This means the final result depends entirely on the specific order you chose to fill in the blanks, rather than just the content of the blanks themselves.

Why Does This Happen? (Three Reasons)

The paper breaks down why parallel generation fails into three distinct causes, like a recipe for a bad cake:

The "Incompatibility" (The Curl): The model's local rules are just mathematically inconsistent. It's like a map where the roads don't connect. If you go North then East, you end up in a different place than if you go East then North.
- Key Finding: Even if the data (the language) is directional (like sentences), the model should be able to learn to be consistent. The fact that it isn't is a flaw in the model's learning, not a flaw in language itself.
The "Total Correlation" (The Teamwork Issue): Even if the model's rules are perfectly consistent (Zero Curl), it can still fail if the pieces are too dependent on each other.
- Analogy: Imagine trying to guess the next word in a sentence where the next word depends on three other words you haven't guessed yet. If you try to guess all three at once without talking to each other, you will likely get it wrong. The pieces are "too connected" to be guessed independently.
The "Estimation Error" (The Comfort Zone): The model might be better at guessing words when it has a clear "prefix" (the start of the sentence) than when it has a messy mix of masked words.
- Result: The model naturally prefers to work left-to-right because that path is "easier" for it to calculate, even if it could theoretically work in any order. It's like a student who knows how to solve a math problem if you show them step 1, but panics if you ask them to solve step 3 first.

The Takeaway: How to Fix It

The paper doesn't just point out the problem; it offers a way to diagnose it.

Don't just blame "Language is sequential": The authors show that the problem isn't that language must be sequential. It's that the model hasn't learned to make its local rules consistent with each other.
The "Curl" Test: We can now measure exactly how "swirly" or inconsistent a model is. If the curl is high, the model will fail at parallel generation.
Better Scheduling: Instead of just guessing words based on confidence, we should pick an order that avoids "swirly" spots and keeps the model in a "comfort zone" where it makes fewer mistakes.

Summary in One Sentence

This paper explains that Diffusion Language Models struggle to work in parallel not because language is inherently sequential, but because the model's local instructions are mathematically inconsistent (high "curl") and it gets confused when trying to guess multiple connected words at once; we can now measure this confusion to build better, faster models.

Technical Summary: Path-Dependent Denoising in Diffusion Language Models

Problem Statement

Diffusion Language Models (DLMs) offer a structural alternative to autoregressive (AR) models by enabling token updates in arbitrary orders or in parallel, theoretically bypassing the serial bottleneck of left-to-right generation. However, a central contradiction exists in practice: while the DLM interface permits arbitrary ordering, learned behaviors often drift toward AR-like trajectories, and high parallelism frequently degrades generation quality. Existing explanations attribute this to the inherent sequential nature of language or chain-of-thought supervision. This paper argues that these explanations leave the key structural question under-specified: for a fixed reverse-time step, does the family of local conditional distributions provided by the denoiser compose into a single, order-invariant joint distribution, or do they define only local rules that are incompatible across different update orders?

Methodology and Framework

The paper formalizes the problem through the lens of conditional compatibility, drawing on literature regarding conditionally specified distributions and Markov Random Fields. The core methodology involves defining and analyzing order-induced pseudo-joints and a metric called local circulation (curl).

Order-Induced Pseudo-Joints: For a block of unresolved positions $B$ and a permutation $\pi$ , the denoiser induces a pseudo-joint distribution $Q^\pi_{\theta,t}$ defined as the sequential product of local conditionals along that specific factorization path. Order-free generation requires these pseudo-joints to be invariant across permutations ( $Q^\pi \approx Q^{\pi'}$ ).
Local Circulation (Curl): The paper defines the local order curl $C^{a,b}_{ij}$ as the log-ratio between two pseudo-joints formed by swapping the update order of two adjacent positions $i$ and $j$ .
$C^{a,b}_{ij} = \log \frac{Q^{i \to j}(a,b)}{Q^{j \to i}(a,b)}$
This metric is invariant to additive logit shifts and serves as a local witness to incompatibility.
Path Dependence Decomposition: The authors establish that the global log-ratio between any two pseudo-joints (path dependence) decomposes exactly into a sum of local circulations along a sequence of adjacent swaps.
Failure Mode Separation: The framework distinguishes three distinct sources of parallel decoding failure:
- Incompatibility (Curl): Non-zero local circulation causing order sensitivity.
- Conditional Dependence: Even with zero curl, independent parallel updates fail if unresolved coordinates have high conditional total correlation (TC).
- Order-Specific Estimation Error: The model may perform better on specific factorization paths (e.g., left-to-right) due to lower conditional estimation errors in those contexts, even if the underlying data distribution is compatible.

Key Contributions

The paper makes four primary theoretical contributions:

Compatibility Formulation: It reframes arbitrary-order denoising as a pseudo-joint invariance problem, separating model-level conditional specifications from the decoding-level operator choices.
Local Circulation Diagnostic: It proves that elementary curl is exactly the log-ratio of swapped pseudo-joints and that its expectation under one order equals the KL divergence between the two order-induced distributions.
Path-Dependence Characterization: It provides an exact structural fact: a DLM is genuinely arbitrary-order on a block if and only if every reachable "elementary square" (pairwise swap) is curl-free. Global order gaps are shown to be the accumulation of local circulations.
Failure-Mode Separation: It clarifies that Bayes-optimal uniform masking implies zero curl (data directionality alone does not force incompatibility). It further demonstrates that parallel degradation can occur even at zero curl due to conditional total correlation, and that order preference is governed by order-specific conditional estimation errors.

Results and Theoretical Findings

The paper derives several theorems that clarify the mechanics of DLM behavior:

Theorem 3 (Bayes Limits): Under Bayes-optimal uniform masking, local curl vanishes. This implies that non-zero curl in practical models arises from finite capacity, imperfect optimization, or incomplete coverage of masking contexts, not merely from the sequential nature of language data.
Theorem 4 (Parallel Independence Error): Even with zero compatibility curl, one-shot independent parallel updates incur a penalty equal to the conditional total correlation (TC) of the block. This explains why parallel decoding can fail even when the model is perfectly compatible.
Theorem 5 (Order Preference): The preferred decoding order is the one that minimizes the cumulative expected local conditional estimation error. If prefix-like contexts are easier for the model to estimate than arbitrary mixed-mask contexts, the model will naturally drift toward left-to-right schedules.

Significance and Claims

The paper positions itself as a mechanism-oriented framework rather than an empirical study of specific model performance. Its significance lies in providing:

Exact Structural Diagnostics: A way to test when DLM decoding is genuinely order-free by measuring pseudo-joint order gaps and local curl, rather than relying on heuristic metrics like entropy or confidence.
Unified Explanation: A framework that unifies observed phenomena (AR-like collapse, parallel failure) by separating incompatibility-driven path dependence from conditional dependence and estimation error.
Validation Interface: A concrete set of operational tests (e.g., commutator diagnostics, order-specific error profiling) that can be applied to existing models without retraining to diagnose the sources of parallelism failure.

The authors modestly claim that this is a testable structural hypothesis. They acknowledge that the paper does not yet provide broad empirical evidence across all DLM families to prove that these metrics outperform simpler proxies (like token entropy) in predicting failure. The decomposition into incompatibility, conditional dependence, and estimation error is presented as conceptually clean but not yet operationally unique in practice. The work suggests that for DLMs to be truly parallel, they must satisfy three conditions simultaneously: stable pseudo-joints across orders, manageable block-level conditional dependence, and support for stable context distributions across different schedules.