Technical Summary: Diffusion Models, Denoiser Architecture and Creativity

Problem Statement

The "creativity" of diffusion models—their ability to generate highly realistic images that deviate from their training data—presents a theoretical paradox. It is established that if a diffusion model utilizes the Bayes optimal denoiser for a given dataset, the model will simply memorize and copy training samples, collapsing the generated distribution into a uniform distribution over the training set. While recent literature offers conflicting explanations for how models escape this memorization (attributing it either to local network properties or underlying data statistics), this paper argues that neither perspective alone is sufficient. The central problem addressed is the lack of a unified understanding of how the specific inductive bias of a denoiser architecture interacts with target data statistics to determine the final generated distribution.

Methodology

The authors employ a dual approach combining theoretical derivation and empirical validation to analyze the relationship between denoiser architecture and the resulting generated distribution ($P_{generated}$).

Theoretical Framework

The analysis is grounded in the Score Matching Theorem, which states that if a denoiser $S_\theta$ minimizes the expected squared error $E(\|S(y) - x\|^2)$, it converges to the score function of the data distribution, allowing the diffusion model to sample from the target distribution $P_{target}$. However, the authors investigate what occurs when the architecture is restricted and cannot perfectly match the true score function.

They derive explicit forms for $P_{generated}$ as a function of $P_{target}$ for three distinct architectural classes:

1. Linear Denoisers: Analyzed to show that the generated distribution is a Gaussian defined by the mean and covariance of the target data.
2. Polynomial Denoisers: Defined as linear layers with entry-wise power-of-$k$ activation functions. The authors derive that the generated distribution is a log-polynomial distribution dependent on the first $2d$ moments of the target (where $d$ is the polynomial degree).
3. Bottleneck Denoisers: Defined as networks with a restricted intermediate dimension $h$. The authors prove that samples from such models lie on an $h$-dimensional manifold, regardless of the input dimension.

Empirical Validation

To validate these theoretical claims, the authors trained diffusion models on two types of datasets:

• 2D Synthetic Data: Used to visualize the mathematical properties of linear, polynomial, and bottleneck architectures under controlled conditions.
• CelebA Subset: A small subset of 1,000 facial images was used to demonstrate architectural effects in a realistic setting. Crucially, all experiments used the exact same dataset and the exact same initial noise (random seed) to isolate the impact of architectural variations.

The authors systematically varied architectural hyperparameters, including:

• The number of pooling layers (altering the receptive field).
• The number of channels (altering capacity).
• Padding types (circular vs. zero padding).
• Bottleneck dimensions ($h$) in linear and MLP networks.
• Polynomial degrees ($k$).

Key Results

1. Architecture Dictates Distribution

The experiments demonstrate that minor architectural changes lead to drastically different generated distributions, even when trained on identical data.

• Receptive Field Variations: Changing the number of pooling layers in a U-Net architecture shifted the output from exact memorization (too large receptive field) to highly distorted outputs (too small receptive field).
• Linear Denoisers: When trained on datasets with identical means and covariances but different underlying structures (e.g., different Gaussian mixtures), linear denoisers produced identical generated distributions, confirming they only capture the first two moments.
• Bottleneck Constraints: In linear and MLP architectures, reducing the bottleneck dimension $h$ forced the generated data onto a lower-rank manifold.
  • $h=1000$ (matching dataset size) resulted in strict memorization.
  • $h=500$ and $h=100$ resulted in "creative" outputs that were novel but lost granular details, lying on lower-dimensional manifolds.
• Polynomial Degrees: Higher-degree polynomial denoisers could capture more complex modes (e.g., bimodal data) but still failed to perfectly reconstruct the target distribution if the degree was insufficient. Interestingly, high-degree polynomials exhibited a dual behavior, producing both memorized samples and novel images.

2. Locality is Not the Sole Driver

Contrary to the hypothesis that local processing (small receptive fields) is the primary driver of creativity, the authors show that architectures with the same receptive field can produce vastly different results (e.g., memorization vs. blurring) based on other structural choices like channel capacity or padding.

Key Contributions

1. Theoretical Characterization: The paper provides explicit mathematical forms for the generated distribution $P_{generated}$ for linear, polynomial, and bottleneck architectures, proving that the architecture inherently enforces a specific distribution shape.
2. Empirical Demonstration of Sensitivity: Through controlled experiments on CelebA, the authors show that small architectural variations (e.g., padding type, pooling count) can shift a model from perfect memorization to non-realistic generation, or to a specific type of "creativity."
3. Reframing Creativity: The work posits that creativity is not an emergent property of data statistics or locality alone, but a result of the alignment (or misalignment) between the inductive bias of the denoiser architecture and the true target distribution.

Significance and Claims

The paper claims that diffusion models will only be successful if the inductive bias of the denoiser architecture is in strong alignment with the true target distribution.

The authors argue that "creativity" in diffusion models is essentially the result of the architecture's inability to perfectly match the Bayes optimal denoiser for the specific target distribution, forcing the model to generate samples that lie on a manifold or distribution shape dictated by the architecture's constraints. This challenges the view that creativity is solely a function of local processing or data diversity, suggesting instead that the architectural design choices fundamentally determine the nature of the generated output.

The paper remains modest regarding its scope, acknowledging that its mathematical analysis is limited to simplified theoretical models and that empirical results are based on a small dataset (1,000 images), which restricts the demonstration of high-fidelity image generation but effectively isolates the specific phenomena of architectural influence.