An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models

1. Problem Statement

Diffusion models have achieved state-of-the-art results in generative tasks, yet two fundamental questions regarding their training dynamics remain unresolved:

1. Learning Order: Which parts of the data distribution (e.g., coarse structures vs. fine details) are learned first, and which remain under-trained (leading to artifacts if training is stopped early)?
2. Architectural Influence: How do specific architectural inductive biases (e.g., fully connected vs. convolutional) shape this learning trajectory?

Existing theories often focus on the sampling process after training or assume optimal scores. This paper aims to derive an analytical framework to track the evolution of the generated distribution during the training process itself, relating it directly to network parameters and data statistics.

2. Methodology

The authors develop a closed-form analytical framework based on the following core assumptions and techniques:

• Gaussian Equivalence Principle: They leverage the finding that for most diffusion times, the learned neural score is well-approximated by the linear score of a Gaussian distribution matching the data's mean and covariance. This allows them to treat arbitrary datasets as Gaussian with matched moments ($\mu, \Sigma$).
• Linear Denoiser Assumption: The analysis assumes the denoiser $D_\theta(x, \sigma)$ is linear (or affine) with respect to the input $x$. This reduces the complex non-linear training dynamics to a tractable system.
• Full-Batch Gradient Flow: Instead of stochastic gradient descent, the authors analyze the continuous-time limit of gradient flow on the full-batch Denoising Score Matching (DSM) loss. This yields deterministic Ordinary Differential Equations (ODEs) for weight evolution.
• Nested Dynamics: The framework solves two coupled processes:
  1. Weight Learning Dynamics: How the denoiser weights $W_\sigma$ evolve over training time $\tau$.
  2. Sampling Dynamics: How the generated distribution evolves by integrating the Probability Flow (PF) ODE backwards from noise to data, using the current (time-dependent) weights.
• Architectural Variations: The theory is extended to:
  • Deep linear networks (symmetric and general).
  • Linear convolutional networks (both full-width/circulant and local patch convolutions).

3. Key Contributions

A. Closed-Form Distribution Dynamics

The paper derives exact analytical expressions for the trajectories of weights and the generated covariance matrix for one-layer, two-layer, and convolutional denoisers. This provides a precise, mode-by-mode understanding of how the model learns the data spectrum.

B. The Inverse-Variance Spectral Law

The central theoretical finding is a universal inverse-variance spectral law:

• The time $\tau_k^*$ required for an eigenmode (or Fourier mode) $k$ to match its target variance scales inversely with its variance $\lambda_k$:
  $$\tau_k^* \propto \lambda_k^{-\alpha} \quad (\text{where } \alpha \approx 1)$$
• Implication: High-variance modes (coarse, global structures) are mastered orders of magnitude faster than low-variance modes (fine details). This explains why early stopping leads to "wrong detail" artifacts (blurred textures, missing fine features) while global shapes are correct.

C. Impact of Architecture on Bias

• Fully Connected (MLP) & Deep Linear Nets: The spectral bias persists. In deep symmetric networks, the bias follows a sigmoidal convergence curve, but the inverse-variance scaling law remains robust.
• Full-Width Convolution (Circulant): Weight sharing in full-width convolutions acts as a learning rate multiplier (speeding up convergence by a factor of $N$, the signal dimension) but does not eliminate the spectral bias. The bias simply shifts from the data eigenbasis to the Fourier basis.
• Local Convolution (Patch-based): This introduces a qualitatively different bias. Local filters couple neighboring pixels, binding many Fourier modes into a single learning unit. This coupling allows for the near-simultaneous emergence of many modes, effectively reshaping the learning dynamics and reducing the spectral bias observed in fully connected networks.

4. Experimental Results

The authors validated their theory on both synthetic and natural image datasets:

• Synthetic Gaussian Data: Experiments on high-dimensional Gaussians confirmed the inverse-variance power law ($\tau^* \propto \lambda^{-1}$) with high precision ($R^2 > 0.9$), even when using deep MLPs with non-linear activations (violating the linear assumption).
• Natural Images (MLP-UNet): On datasets like FFHQ and MNIST, MLP-based diffusion models exhibited the predicted spectral ordering: coarse facial contours (high-variance eigenmodes) emerged quickly, while fine details (low-variance modes) appeared much later. The power-law scaling held, though slightly attenuated compared to the linear theory.
• Natural Images (CNN-UNet): Convolutional U-Nets displayed a distinct behavior. Instead of sequential emergence, many modes emerged almost simultaneously.
  • Ablation Study: The authors found that the "absence" of spectral bias in standard CNNs is due to network width. Narrow networks (few channels) exhibit strong spectral bias (similar to patch theory), while wide networks (many channels, e.g., 128+) learn modes almost instantaneously. This suggests the ratio of network channels to input channels governs the deviation from theoretical predictions.

5. Significance and Implications

• Mechanistic Explanation: The paper provides the first analytical explanation for the "outline-first, details-later" phenomenon in diffusion models, linking it directly to the data covariance spectrum and the inverse-variance learning speed.
• Early Stopping Risks: It quantifies the risk of early stopping: models may capture the global structure but fail to reconstruct fine-grained textures or specific semantic details (e.g., finger joints, text strokes) because those low-variance modes have not yet converged.
• Architectural Design: The findings highlight that local convolution is a powerful inductive bias that can mitigate spectral bias, but its effectiveness is heavily dependent on network width. This challenges the assumption that standard CNNs behave like simple linear filters in the frequency domain.
• Future Directions: The work calls for deeper investigation into the unique inductive biases of local convolutions and suggests that high-channel architectures (common in modern latent diffusion models) may behave differently than the low-channel theoretical limits analyzed here.

In summary, this paper establishes a rigorous analytical foundation for understanding how diffusion models learn, revealing that data covariance dictates the order of learning and that architectural choices (specifically locality and width) can fundamentally alter this spectral bias.