Technical Summary: A Tutorial on Diffusion Theory: From Differential Equations to Diffusion Models

Problem Statement

Diffusion models have emerged as a dominant paradigm in generative modeling, yet the field is characterized by a proliferation of formulations: discrete Markov chains (DDPM), continuous stochastic differential equations (SDEs), probability-flow ordinary differential equations (ODEs), score matching, and flow matching. While these approaches are known to be related, the precise mathematical correspondences between them—particularly regarding their training objectives, reverse-time dynamics, and sampling procedures—are often obscured by differing notations and discrete-versus-continuous perspectives. This tutorial aims to unify these perspectives by deriving diffusion models strictly from the viewpoint of differential equations, establishing a rigorous bridge between conditional Gaussian processes, marginalized forward dynamics, and their corresponding reverse-time formulations.

Methodology

The paper adopts a rigorous mathematical framework rooted in stochastic calculus and partial differential equations to reconstruct the theory of diffusion models. The methodology proceeds through the following logical stages:

1. Conditional Forward Process: The authors begin by defining a conditional Gaussian forward process $X_t | X_0 = x_0$ governed by parameters $\alpha_t$ and $\sigma_t$. They demonstrate that this specific path admits two equivalent representations:

   • A Conditional ODE driven by a velocity field derived from the conditional score.
   • A Conditional SDE driven by a drift term and a Brownian motion.
     The paper proves that the conditional density satisfies both the continuity equation (for the ODE) and the Fokker–Planck equation (for the SDE).

2. Marginalization: By averaging the conditional process over the data distribution $p_0$, the authors derive the marginalized forward process. They show that the marginal density $p_t(x)$ also satisfies a continuity equation and a Fokker–Planck equation. Crucially, they utilize Fisher's Identity to relate the marginal score $\nabla \log p_t(x)$ to the expectation of the conditional score, establishing that the marginal velocity field is the expectation of the conditional velocity field.

3. Reverse Dynamics: The paper introduces the reverse-time variable $\tau = 1-t$ and derives the reverse-time dynamics.

   • Reverse SDE: Using the time-reversal of the forward SDE, the authors derive the reverse SDE, which is governed by the marginal score $\nabla \log p_t(x)$.
   • Reverse ODE (Probability-Flow): They derive a deterministic ODE that shares the same marginal density path as the reverse SDE. This "probability-flow ODE" is shown to have a velocity field that is an affine transformation of the score.

4. Training Objectives: The tutorial connects the standard noise-prediction objective used in practice to the theoretical score-matching objective. Using an orthogonality identity, the authors prove that minimizing the expected squared error between a neural network's noise prediction and the true noise is equivalent (up to a constant) to minimizing the score-matching loss. This establishes that the standard training objective learns the marginal score required for both the reverse SDE and the reverse ODE.

5. Sampling and Unification: The paper details sampling algorithms for both the learned reverse SDE (Euler–Maruyama) and the reverse ODE (Euler, Heun, and DPM-Solver). Finally, it explicitly maps the discrete DDPM and DDIM algorithms onto this continuous framework:

   • DDPM is shown to correspond to a discrete approximation of the reverse SDE, where the stochastic term utilizes the exact posterior variance of the discrete chain.
   • DDIM is shown to correspond to a discrete approximation of the reverse ODE (deterministic), with a stochasticity parameter $\eta$ allowing interpolation between the two.

6. Guidance and Flow Matching: The framework is extended to include classifier guidance and classifier-free guidance, showing how these modify the reverse drift. Furthermore, the paper unifies Score Matching and Flow Matching, demonstrating that the standard diffusion loss can be interpreted as a weighted flow-matching objective for the induced probability-flow ODE.

Key Contributions

• Unified Derivation: The paper provides a single, coherent derivation showing that the conditional Gaussian forward path admits both ODE and SDE representations, and that marginalizing this path yields consistent marginal ODE and SDE formulations.
• Equivalence of Objectives: It rigorously proves that the standard noise-prediction loss is equivalent to score matching, and further demonstrates that this same loss induces a flow-matching objective for the deterministic reverse ODE.
• DDPM/DDIM Unification: The work explicitly connects the discrete DDPM and DDIM algorithms to the continuous reverse SDE and reverse ODE frameworks, respectively. It clarifies that DDPM sampling is a first-order discretization of the reverse SDE (with specific variance handling), while DDIM is a discretization of the reverse ODE.
• Clarification of Noise Terms: The appendix provides a detailed clarification of the three distinct noise symbols ($W_t$, $W^{rev}_\tau$, $\bar{W}_t$) used in forward and reverse time, resolving common notational confusion regarding Brownian motion in time-reversed settings.
• Generalization of Guidance: The tutorial formalizes both classifier and classifier-free guidance within the reverse SDE/ODE framework, showing how they modify the drift term via the conditional score.

Results

The paper does not present new empirical results or experimental benchmarks. Instead, its "results" are theoretical derivations and proofs:

• Theorem 1.2 & 1.4: Proof that the conditional Gaussian path is generated by both a specific ODE and SDE.
• Theorem 2.1 & 2.2: Proof that the marginalized process is generated by marginal ODE and SDE formulations.
• Theorem 3.3 & 3.5: Derivation of the reverse SDE and the reverse probability-flow ODE, proving they share the same density path.
• Theorem 4.2: Proof of the equivalence between the noise-prediction loss and the score-matching loss.
• Theorem 7.6 & 7.7: Demonstration that DDPM training and inference correspond to the reverse SDE framework on a discrete grid.
• Theorem 7.8: Demonstration that deterministic DDIM corresponds to the reverse ODE framework.
• Proposition 8.1: Proof that the diffusion loss is a weighted flow-matching objective for the reverse ODE velocity.

Significance and Claims

The paper positions itself as a tutorial intended to clarify the theoretical underpinnings of modern diffusion models. Its significance lies in:

1. Bridging Discrete and Continuous: It resolves the apparent disconnect between the discrete Markov chain formulations (DDPM) and the continuous SDE/ODE formulations (Song et al., 2020; Lipman et al., 2023), showing they are different discretizations of the same underlying continuous dynamics.
2. Unifying Perspectives: It unifies the "score-based" perspective (learning $\nabla \log p_t$) with the "flow-based" perspective (learning a velocity field $v_t$), showing that in the diffusion setting, these are linked by a simple affine transformation.
3. Pedagogical Rigor: By deriving every step from first principles (conditional Gaussian kernels, Fokker–Planck equations, Itô calculus), the paper provides a self-contained reference that avoids the notational inconsistencies often found across different sub-fields of generative modeling.

The authors claim that this framework offers a "single language" for understanding diffusion models, flow matching, and score matching, allowing researchers to see that the reverse SDE is the stochastic generative model, while the reverse ODE is the deterministic flow model, both driven by the same learned score function.