Denoising Diffusion Probabilistic Models

This paper presents high-quality image synthesis using denoising diffusion probabilistic models, achieving state-of-the-art results on CIFAR10 and LSUN datasets through a novel training objective connecting diffusion models to denoising score matching with Langevin dynamics.

The Gist

Imagine you are trying to teach a computer to draw a picture of a cat. Most AI models try to learn by looking at thousands of photos and trying to guess the next pixel, or by fighting against another AI in a game of "fake vs. real."

This paper introduces a different approach called Denoising Diffusion Probabilistic Models (or just "Diffusion Models"). Here is the simple, everyday explanation of how it works, using a few creative analogies.

1. The Two-Step Dance: The Messy Room and the Tidy Room

Think of the training process as a two-part dance between two states: Chaos and Order.

Step A: The Messy Room (The Forward Process)
Imagine you have a pristine, beautiful painting of a cat. Now, imagine a mischievous child (the "diffusion process") who slowly, step-by-step, throws dust, smudges, and random noise onto the painting.

• At first, the cat is still clear.
• After a few steps, it's a bit blurry.
• After 1,000 steps, the painting is completely covered in static noise. You can't see the cat at all; it looks like pure TV static.

The AI watches this happen. It doesn't need to learn how to make the mess; the rules of the mess are fixed. It just observes how the image turns into noise.

Step B: The Tidy Room (The Reverse Process)
Now, the AI has to learn the magic trick: How to un-mess the room.
The AI starts with a blank canvas full of static noise (pure chaos). It has to figure out how to remove the noise, step-by-step, to reveal the cat underneath.

• It looks at the noise and asks, "If I remove a little bit of this static, what does the underlying image look like?"
• It peels away the noise layer by layer.
• Eventually, after 1,000 steps of cleaning, the static is gone, and a perfect, high-quality cat appears.

The paper's big breakthrough is that the AI gets really, really good at this "cleaning" job.

2. The Secret Sauce: Predicting the "Noise"

How does the AI know what to clean?

In the past, people tried to teach the AI to predict the final image directly from the noise. But the authors found a smarter way. Instead of asking, "What is the cat?", they ask, "What is the noise?"

Imagine you are looking at a muddy window.

• Old way: Try to guess the exact shape of the tree outside.
• New way (This paper): Look at the mud on the glass and guess exactly where the mud is and how thick it is.

Once the AI knows exactly where the "mud" (noise) is, it can simply subtract it. If you know the noise, you can easily find the picture. The paper shows that training the AI to predict the noise is much easier and leads to much better pictures.

3. The "Progressive Reveal" (Like a Slow-Motion Movie)

One of the coolest things about this model is how it generates images. It's not like a printer that spits out a finished page instantly. It's more like a time-lapse video of a sculpture being carved.

• Start: A blob of random static.
• Middle: You start to see vague shapes. Maybe a round head, two ears. It's like looking at a cloud that vaguely looks like a face.
• End: The details sharpen. You see the whiskers, the fur texture, the eyes.

The paper calls this "progressive lossy decompression." It's like receiving a low-resolution JPEG that slowly gets higher and higher resolution as more data arrives. The AI builds the image from the "big picture" down to the tiny details.

4. Why is this a Big Deal?

Before this paper, other AI models (like GANs) were great at making pictures, but they were hard to train and sometimes unstable (they would crash or produce weird, blurry results).

This paper showed that:

1. The pictures are stunning: On standard tests, their AI produced images that were sharper and more realistic than almost anything else at the time (beating the previous champions).
2. It's stable: The training process is much more reliable. It doesn't "fight" like other models; it just learns to clean up noise.
3. It's versatile: Because the math is so clean, it can be used for many things, not just cats.

The Bottom Line

Think of this paper as teaching a computer to be a master restorer. Instead of trying to paint a masterpiece from scratch, the computer learns to look at a ruined, noisy canvas and perfectly reconstruct the original masterpiece by knowing exactly what the noise looks like and removing it.

The result? A computer that can generate incredibly realistic images, one tiny step of "cleaning" at a time.

Technical Summary

Here is a detailed technical summary of the paper "Denoising Diffusion Probabilistic Models" by Ho, Jain, and Abbeel.

1. Problem Statement

Deep generative models (GANs, VAEs, Flows, Autoregressive models) have achieved high-quality sample synthesis, but they often suffer from specific limitations:

• GANs: Suffer from training instability and mode collapse.
• VAEs/Flows: Often struggle to match the sample quality of GANs.
• Autoregressive Models: Can be slow to sample and require specific data orderings.
• Energy-Based Models (EBMs): Often difficult to train and evaluate.

While Diffusion Probabilistic Models (introduced by Sohl-Dickstein et al., 2015) were theoretically sound and easy to train, they had not previously demonstrated the ability to generate high-quality samples comparable to state-of-the-art GANs. The authors aim to bridge this gap by improving the training objective and parameterization of diffusion models to achieve state-of-the-art image synthesis.

2. Methodology

The paper proposes a refined approach to training diffusion models, connecting them to Denoising Score Matching and Langevin Dynamics.

A. The Diffusion Process

The model consists of two Markov chains:

1. Forward Process (Diffusion): A fixed process that gradually adds Gaussian noise to data $x_0$ over $T$ timesteps until the data becomes pure noise ($x_T \sim \mathcal{N}(0, I)$).
   $$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I)$$
   Crucially, this allows sampling $x_t$ at any timestep $t$ in closed form:
   $$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t}x_0, (1-\bar{\alpha}_t)I)$$
   where $\bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s)$.

2. Reverse Process (Generation): A learned process that attempts to reverse the diffusion, starting from pure noise and iteratively removing noise to recover the data.
   $$p_\theta(x_{0:T}) = p(x_T) \prod_{t=1}^T p_\theta(x_{t-1}|x_t)$$
   The transitions are modeled as Gaussians: $p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$.

B. Key Innovations & Parameterization

The authors introduce three critical modifications to the standard diffusion framework:

1. $\epsilon$-Prediction Parameterization:
   Instead of predicting the mean $\mu_\theta$ directly or the original data $x_0$, the neural network is trained to predict the noise $\epsilon$ added at step $t$.

   • The input to the network is the noisy image $x_t$.
   • The network $\epsilon_\theta(x_t, t)$ predicts the noise $\epsilon$ such that $x_t \approx \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon$.
   • This parameterization simplifies the variational bound and aligns the reverse process with Denoising Score Matching.

2. Simplified Training Objective ($L_{simple}$):
   The standard variational bound (ELBO) involves a weighted sum of KL divergences. The authors derive a simplified objective that ignores the complex weighting terms and focuses solely on the noise prediction error:
   $$L_{simple}(\theta) = \mathbb{E}_{t, x_0, \epsilon} \left[ \| \epsilon - \epsilon_\theta(\sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon, t) \|^2 \right]$$

   • Here, $t$ is sampled uniformly from $1$to$T$.
   • This objective is mathematically equivalent to training a denoising score matching model over multiple noise scales.
   • Empirically, this unweighted objective yields significantly better sample quality than the weighted ELBO.

3. Fixed Variance:
   The variance $\Sigma_\theta$ of the reverse process is set to fixed constants (either $\beta_t$ or $\tilde{\beta}_t$) rather than being learned. This stabilizes training and improves performance.

C. Architecture

• Backbone: A U-Net architecture similar to PixelCNN++ but with Group Normalization.
• Conditioning: Time step $t$ is injected into the network using Transformer-style sinusoidal position embeddings.
• Attention: Self-attention mechanisms are applied at the $16 \times 16$ feature map resolution to capture global dependencies.

3. Key Contributions

1. State-of-the-Art Sample Quality: The authors demonstrate that diffusion models can generate samples superior to many GANs and other generative models.
   • CIFAR-10: Achieved an FID of 3.17 and Inception Score of 9.46 (unconditional), outperforming most unconditional models and competing with conditional ones.
   • LSUN: Achieved FID scores comparable to ProgressiveGAN on 256x256 images.
2. Theoretical Connection: Established a rigorous link between diffusion models, denoising score matching, and annealed Langevin dynamics. This explains why the $\epsilon$-prediction objective works so well.
3. Progressive Lossy Decompression: The authors interpret the sampling process as a form of progressive decoding. They show that the model acts as a "lossy compressor" where the majority of bits (information) are used to describe imperceptible details, while the coarse structure is recovered early in the sampling process.
4. Generalized Autoregressive Decoding: They propose that the Gaussian diffusion process can be viewed as an autoregressive model with a generalized bit-ordering that is not restricted to spatial coordinates, potentially offering better inductive biases for images than standard masking.

4. Results

• Quantitative Metrics:
  • CIFAR-10: FID 3.17 (vs. 3.26 for StyleGAN2+ADA unconditional; 14.73 for BigGAN).
  • LSUN Bedroom: FID 4.90 (vs. 2.65 for StyleGAN2, but competitive with ProgressiveGAN).
  • Log-Likelihood: While the log-likelihood (bits/dim) is not competitive with autoregressive models (e.g., PixelCNN++), the authors argue this is because diffusion models prioritize perceptual quality over exact likelihood, effectively compressing imperceptible details.
• Qualitative Metrics:
  • Generated images show high fidelity, diverse modes, and coherent global structures.
  • Interpolation: Linear interpolation in the latent space (during the diffusion process) results in smooth, semantically meaningful transitions (e.g., changing pose, skin tone, or expression) without artifacts.
  • Progressive Generation: Visualizations show that large-scale features appear early in the reverse process (low $t$), while fine details appear later (high $t$).

5. Significance and Impact

• Paradigm Shift: This paper revitalized the field of diffusion models, shifting them from a theoretical curiosity to a dominant force in generative AI. It paved the way for subsequent breakthroughs like DALL-E 2, Stable Diffusion, and Imagen.
• Training Stability: Unlike GANs, which require delicate balancing between generator and discriminator, diffusion models are trained using a simple, stable mean-squared error objective.
• Flexibility: The framework is highly flexible, allowing for easy integration of conditioning (class labels, text, etc.) and application to various data modalities (audio, video, 3D).
• Compression Insight: The paper provides a novel perspective on generative modeling as a form of progressive lossy compression, offering new theoretical insights into how deep generative models represent data.

In summary, Ho et al. demonstrated that by re-parameterizing diffusion models to predict noise and optimizing a simplified, unweighted objective, one can achieve state-of-the-art image synthesis with stable training, effectively bridging the gap between variational inference, score matching, and high-quality generation.