📊 statistics

Sticky Jump Diffusions: A Unifying View of Masked, Continuous, and Hybrid Diffusion

This paper introduces Sticky Jump Diffusions (SJDs), a unifying continuous-time Markov framework that recovers masked, continuous, and hybrid diffusion models as limits and enables simulation-free training via Denoising Hazard Matching, while offering a flexible design space for corruption kernels that improves performance on tasks like CIFAR-10, Text8, and Sudoku.

Original authors: Pascal Jutras-Dubé, Patrick Pynadath, Jeremy Lu, Yuan Gao, Ruqi Zhang

Published 2026-07-14

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Pascal Jutras-Dubé, Patrick Pynadath, Jeremy Lu, Yuan Gao, Ruqi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to rebuild a shattered mosaic, but you don't just have the broken pieces; you have a magical, sticky floor that holds some pieces in place while others float around in a foggy cloud. This is the world of Sticky Jump Diffusions (SJDs), a new way of teaching computers to create things like images, text, and even Sudoku puzzles.

The Big Idea: A Sticky Floor and a Foggy Room

To understand this, let's look at how computers usually try to "un-break" things.

The Old Ways:

The Masked Approach: Imagine you have a sentence, and you cover some words with black boxes (masks). The computer guesses what's under the box. But here's the catch: once a word is covered, the computer treats it as a blank slate. It doesn't know how close the real word was to being guessed. It's like trying to guess a word in a game of "Hangman" where the computer forgets it ever saw the letters before.
The Continuous Approach: Imagine the computer turns every word into a floating point in a giant, smooth 3D space. It nudges these points around until they look right. But when it's done, the points are floating in mid-air, not on the actual words. The computer has to do a clumsy, extra step at the very end to snap them back onto the nearest word. It's like baking a cake and then having to manually glue the frosting back on after it falls off.
The Hybrid Approach: This tries to do both: keep some words masked and float others. But usually, the rules for when to snap a word back into place are just guessed or hand-picked by the programmers.

The New Solution (SJD):
The authors, a team from Purdue University, say: "Let's stop guessing the rules. Let's make the rules come from the physics of the process itself."

They created a system where data starts as solid "anchors" (like real words or pixel values). In the forward process (the "breaking" phase), these anchors let go of their mass at a specific rate and drift into a foggy, continuous space.

In the reverse process (the "fixing" phase), the magic happens. The computer doesn't just guess; it uses a mathematical law called flux balance. Think of it like a busy train station. If you know exactly how many people left the station and where they went, you can calculate exactly how many people need to arrive and where they need to go to keep the station balanced.

In SJD, the computer calculates the "hazard rate" (how likely a piece is to snap back) and the "destination" (which word it should become) automatically. It's not a hand-picked schedule; it's a natural consequence of how the data was broken.

The Secret Sauce: One Brain, Two Jobs

Usually, to fix this kind of problem, you might need one brain to guess the score (how to move the fog) and another brain to guess the jump (when to snap back).

The authors found a clever trick called Denoising Hazard Matching. They proved that a single neural network (one brain) can do both jobs. By training it with a standard "cross-entropy" game (a common way to teach computers to guess categories), the network learns the answers to both questions. It's like teaching a student to solve a math problem, and then realizing that the same student can also tell you exactly how long the answer will take to compute, just by looking at the problem again.

The "Sticky" Twist: Blending Neighbors

Here is where the paper gets really creative. In the old hybrid models, when a piece of data was corrupted, it was corrupted based only on itself. If you were fixing a pixel in an image, the computer only looked at that one pixel.

The authors introduced a blending matrix. Imagine you are fixing a sentence. Instead of looking at just the word you are fixing, you look at the words around it. If you are fixing a word in a Sudoku puzzle, you look at the numbers in the same row, column, and box.

The computer corrupts the data by blending it with its neighbors.

For Images (CIFAR-10): It blurs a pixel with its neighbors. This helps the computer understand that pixels next to each other usually belong together.
For Text (Text8): It blends a character with the characters around it, helping it understand that "q" is usually followed by "u".
For Sudoku: It blends a cell with the cells in its row, column, and 3x3 box, teaching the computer the rules of the game directly through the corruption process.

What the Numbers Say

The team tested this on three different types of puzzles:

Images (CIFAR-10): They measured the quality using a score called FID (lower is better). The new method got a score of 14.57, beating the previous best hybrid model (CADD) which scored 15.88, and the masked diffusion model (MDLM) which scored 18.11.
Text (Text8): They counted how many valid words the computer could generate. With a blending bandwidth of 1.5, the new method generated more valid words of length 5 or 6 than the previous best models, especially when given more time to think (higher NFE budgets).
Sudoku: This was the big test. The previous hybrid model (CADD) was unstable; in some training runs, it failed completely (accuracy dropped to near chance). The new method (SJD) never collapsed. It solved complete boards with an accuracy of 95.65%, compared to CADD's 47.12%. It also started solving boards much faster, taking off at 50,000 training steps instead of 203,000.

What They Explicitly Ruled Out

The authors were very clear about what doesn't work or isn't necessary:

No Hand-Tuned Schedules: They argue against the idea that you need to manually design a schedule for when to "commit" (snap back) to a token. In their system, the schedule is calculated automatically by the math.
No Second Network: They proved you don't need a separate part of the computer to calculate the "hazard" (the jump rate). One single network is enough.
Learning the Hazard Rate: They tried to teach the computer to learn the "hazard rate" (how fast things break) from scratch. They found that this actually made the results worse. The best results came when they kept the hazard rate fixed and simple, letting the "blending" (the neighbor interaction) do the heavy lifting.

How Sure Are They?

The authors are quite confident in the math. They proved that their method is the exact time-reversal of their forward process using rigorous theorems (Theorem 2.6 and Theorem 3.2). They didn't just suggest it might work; they showed the equations that make it work.

However, the performance numbers (like the FID scores and Sudoku accuracy) are based on simulations and experiments. They ran the models on specific datasets (CIFAR-10, Text8, Sudoku) and measured the results. They found that the new method consistently outperformed the old ones in these specific tests. They don't claim it works for everything in the universe, but for the tasks they tested, the evidence is strong.

The Takeaway

Sticky Jump Diffusions is like giving the computer a map of the terrain it's trying to rebuild. Instead of blindly guessing where to put the pieces or manually telling it when to stop, the computer uses the physics of the "breaking" process to figure out the perfect "fixing" process. And by letting the pieces "feel" their neighbors during the breaking phase, the computer learns to respect the structure of the world—whether that's the grid of a Sudoku board or the flow of a sentence—much better than before.

Technical Summary: Sticky Jump Diffusions (SJDs)

Problem Statement

Diffusion-based generative modeling has successfully extended from continuous data to discrete sequences (e.g., text), yet existing approaches face fundamental limitations in unifying discrete and continuous dynamics. Two dominant families exist:

Masked Diffusion: Operates on discrete token spaces, outputting valid tokens by construction. However, it treats each position as mapping to a single absorbing token, erasing graded information about the proximity of corruption to candidate tokens.
Continuous Diffusion: Operates on token embeddings using standard score-based learning. Its terminal state lies off the discrete support, requiring an external projection step (e.g., a decoder or quantization) to recover discrete tokens, which is not part of the learned dynamics.
Hybrid Diffusion: Pairs a masked chain with a continuous latent. While effective, these models often assemble reverse samplers by hand, treating commitment rates, destinations, and drifts as separate design choices rather than deriving them from a unified forward process.

The core challenge is to define a continuous-time Markov process that naturally bridges these regimes, where the reverse dynamics (commitment to discrete tokens and continuous diffusion) are mathematically derived from the forward process via flux balance, rather than being heuristically designed.

Methodology: Sticky Jump Diffusions (SJDs)

The authors introduce Sticky Jump Diffusions (SJDs), a class of continuous-time Markov processes on $\mathbb{R}^d$ with a finite or countable set of anchors $A$ (identified with token embeddings).

1. Forward Dynamics (Unsticking)

In forward time, data starts at an anchor $a \in A$ . The process evolves as follows:

Sticking Phase: The particle waits at the anchor $a$ for a random duration governed by a hazard rate $\vec{\lambda}_t$ .
Unsticking: At the hazard event, the particle "unsticks" to a new state $Y$ sampled from a kernel $R_t(\cdot | a)$ supported on the continuous region $X_A = \mathbb{R}^d \setminus A$ .
Diffusion: Once in $X_A$ , the trajectory diffuses according to a standard Stochastic Differential Equation (SDE): $d\vec{X}_t = \vec{b}_t(\vec{X}_t)dt + g_t d\vec{W}_t$ .
Mass Split: The marginal distribution $p_t$ splits into an absolutely continuous density on $X_A$ and atomic masses on the anchors $A$ .

Key Innovation (Cross-Position Blending): Unlike standard hybrids where the unsticking kernel depends only on the local anchor, SJDs allow the kernel to depend on the entire sequence via a blending matrix $W$ . The mean of the unsticking distribution at position $i$ is a weighted blend of neighbors' embeddings: $\mu_i(\vec{X}_0) = (W E(\vec{X}_0))_i$ . This encodes structural inductive biases (e.g., spatial locality or constraint graphs) directly into the corruption process.

2. Reverse Dynamics (Sticking)

The time reversal of an SJD yields a generative process with two coupled mechanisms:

Score-Driven SDE: On the continuous region $X_A$ , the process follows a reverse SDE driven by the score function $\nabla \log p_t(x)$ .
Sticky Jump Kernel: The process jumps from the continuous region back to anchors. The reverse hazard rate (when to jump) and the jump destination (where to jump) are uniquely determined by the flux balance equation (detailed balance between forward and reverse flows).
- Crucially, the commitment rate and destination are not hand-designed schedules but are fixed by the forward law.

3. Learning: Denoising Hazard Matching (DHM)

To make the reverse process trainable, the authors propose Denoising Hazard Matching (DHM), the hazard analogue of denoising score matching.

Single Classifier: A single denoising classifier $P_\theta(a | y, t)$ is trained via standard cross-entropy on SJD-corrupted pairs $(\vec{X}_0, \vec{X}_t)$ .
Analytic Reweighting: The reverse hazard and allocation probabilities are recovered analytically from $P_\theta$ $P_{θ}$ without a second network head.
- The reverse hazard $\vec{\lambda}^\star_t(y)$ is estimated as the conditional expectation of a forward-known target (the DHM target) under the SJD corruption.
- The allocation probability $\pi^\star_t(a|y)$ is derived by reweighting the classifier's logits with the DHM target.
Simulation-Free Training: Under standard conditions (closed-form survival, Gaussian closure of convolution), training requires no simulation of the SJD path. The mixture density required for the target is evaluated via a closed-form quadrature, making the training pipeline simulation-free and efficient.

Key Contributions

The SJD Process Class: A unified framework for jump-diffusions where the forward marginal splits into a continuous density and discrete atoms. This generalizes classical sticky-boundary diffusions to high-dimensional anchor sets.
Reverse Dynamics from a Single Classifier: The authors derive that both the score and the per-anchor reverse intensities can be estimated from a single denoising classifier via DHM. This eliminates the need for separate hazard heads and relies on a Rao–Blackwellized reweighting of the classifier output.
Unifying View of Diffusion Families: SJDs recover existing models as specific limits:
- Masked Diffusion: Arises when the unsticking kernel collapses to a single absorbing point. The "information void" in masked diffusion is explained structurally: the unsticking kernel is anchor-independent, so the corrupted state carries no evidence of the source token.
- Continuous Diffusion: Arises when the jump rate is zero. The necessity of a terminal projection is explained by the absence of atoms in the forward marginal, which yields no reverse jumps.
- Hybrid Diffusion: Arises when the unsticking kernel matches the forward perturbation kernel. The commit rate, destination, and drift are shown to follow naturally from flux balance rather than separate design.
The Jump Kernel as a Design Space: By varying the blending matrix $W$ , the authors introduce a new design space. Cross-position blending corrupts each position toward a blend of its neighbors' clean values, turning dependency structures (spatial, sequential, or constraint-based) into an inductive bias of the corruption itself.

Experimental Results

The authors empirically validate the cross-position blending axis on three datasets where structural dependencies are known:

CIFAR-10 (Image Generation): Using a value-space blur kernel to encode spatial locality, SJD with a non-local kernel ( $\sigma_B = 2.5$ ) achieves a lower FID (14.57) than the identity-kernel hybrid (CADD, 15.88) and other baselines (MDLM, Bit Diffusion) at 256 NFE.
Text8 (Text Generation): Using a 1D Gaussian convolution to encode local sequential correlation, SJD with $\sigma_W \ge 1.0$ surpasses CANDI and MDLM in valid-word generation counts, particularly at higher sequence lengths ( $\ge 6$ ) and across various NFE budgets.
Sudoku (Constraint Satisfaction): Using a constraint-graph kernel to encode row/column/box dependencies, SJD significantly outperforms the identity-kernel hybrid (CADD). While CADD exhibits bimodal performance (often collapsing to near-chance), SJD consistently solves boards, achieving 95.65% accuracy and converging 4x faster in training steps.

Significance and Claims

The paper claims that Sticky Jump Diffusions provide a process-level account that unifies masked, continuous, and hybrid diffusion.

Theoretical Unification: It demonstrates that the discrete and continuous coordinates in generative modeling are not separate entities to be coupled by hand, but rather a single reversed state where commitment rates and destinations are fixed by flux balance.
Design Space Expansion: By treating the unsticking kernel as an explicit design space, SJDs allow researchers to encode structural priors (like spatial locality or logical constraints) directly into the forward corruption dynamics. This improves generation quality and logical consistency without altering the reverse sampler's fundamental architecture.
Practical Efficiency: The framework trains and samples at the same cost as existing hybrids, as the SJD-specific structure enters only through analytic reweighting and closed-form quadratures, requiring no additional network parameters or simulation overhead.

The authors conclude that this unifying view makes it easier to extend diffusion models: new corruptions, schedules, and commitment rules can be specified as forward dynamics and obtained in reverse for free via the time-reversal theorem.