📊 statistics

A theory of learning data statistics in diffusion models, from easy to hard

This paper establishes that diffusion models learn data statistics in a hierarchical manner, prioritizing simple pairwise correlations before tackling complex higher-order correlations, a process governed by a newly identified "diffusion information exponent" that determines the sample complexity required for learning different statistical orders.

Original authors: Lorenzo Bardone, Claudia Merger, Sebastian Goldt

Published 2026-03-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lorenzo Bardone, Claudia Merger, Sebastian Goldt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to draw pictures of cats. You don't just show it a finished photo; instead, you start with a picture of pure static (white noise) and ask the robot to slowly "clean" the noise until a cat appears. This is how Diffusion Models work. They are the AI behind many of the amazing images you see today.

But here's the mystery: How does the robot actually learn? Does it learn the shape of the ears first, then the fur texture, then the whiskers? Or does it learn everything at once?

This paper, "A theory of learning data statistics in diffusion models, from easy to hard," answers that question. It reveals that these AI models have a very human-like learning style: they learn the easy stuff first, and the hard stuff later.

Here is the breakdown using simple analogies:

1. The "Easy-to-Hard" Learning Curve

The authors discovered that diffusion models suffer from (or rather, benefit from) a "Distributional Simplicity Bias."

The Analogy: Imagine you are trying to learn a new language.
- Phase 1 (The Easy Stuff): At first, you only learn the most common words and basic sentence structures (like "Hello" or "The cat is big"). You can't say complex poetry yet, but you can hold a simple conversation.
- Phase 2 (The Hard Stuff): Only after you've mastered the basics do you start learning complex grammar, idioms, and subtle cultural nuances.
The Experiment: The researchers trained a model on real images (like cats and cars). Then, they tested it on two things:
1. Real Images: Full of complex details.
2. "Clone" Images: These were fake images made to look like the real ones statistically, but they were just blurry blobs with the same average brightness and color patterns (mean and covariance). They lacked the complex details.
- The Result: For the first 1,000 steps of training, the model performed equally well on the real images and the blurry "clones." This means the model was only using the "easy" statistics (average colors and basic shapes). It wasn't "seeing" the complex details yet. Only after thousands more steps did it start to outperform the clones, proving it had finally learned the "hard" complex details.

2. The "Diffusion Information Exponent" (The Difficulty Score)

The paper introduces a mathematical concept called the Diffusion Information Exponent ( $k^*$ ). Think of this as a "Difficulty Score" for different types of information.

Level 1 (Easy): Learning the average brightness or basic shape. This is like learning to walk. The model can do this very quickly with very few examples.
Level 4 (Hard): Learning complex correlations, like "if the eyes are blue, the fur is likely white." This is like learning to play the violin.
The Finding: The model needs cubic (much, much more) data to learn the "Level 4" stuff compared to the "Level 1" stuff. It's not just a little harder; it's exponentially harder.

3. The "Hidden Trap" (The Spherical Constraint)

One of the most surprising findings is about how the model moves while learning.

The Analogy: Imagine a hiker trying to climb a mountain (learning the pattern).
- Projected SGD (The Smart Hiker): The researchers found that if you force the hiker to stay on a specific path (a sphere), they climb efficiently.
- Unconstrained SGD (The Lost Hiker): If you let the hiker wander freely, they often get stuck in a "trivial solution." It's like the hiker decides, "Climbing is too hard; I'll just sit at the bottom of the hill and do nothing."
- The Result: Without a specific mathematical "tether" (the spherical constraint), the model often gives up and learns nothing, getting stuck at zero. The tether forces it to keep trying.

4. The "Secret Shortcut" (Correlated Latents)

The paper also found a way to speed up the learning of hard things.

The Analogy: Imagine you are learning to recognize a specific type of bird.
- Hard Way: You have to memorize every single feather pattern from scratch.
- Easy Way: You realize that in this forest, if you see a red beak, the bird always has blue wings. The features are linked.
The Finding: If the complex features in the data are naturally linked (correlated), the model learns them much faster. It's like having a cheat sheet that connects the hard concepts to the easy ones.

5. Why Deep Networks Help (The "Over-Parameterization" Magic)

Finally, the paper looks at why modern AI uses huge, deep networks.

The Analogy: Imagine a single person trying to solve a complex puzzle. They might get stuck. But if you give them a team of 100 people (a wide network), they can pass pieces of the puzzle around, helping each other escape dead ends.
The Finding: Even if the math says a simple model should get stuck, making the network wider (more neurons) or deeper (more layers) allows it to escape the "trivial solution" and learn the hard, complex patterns without needing to be manually tuned.

Summary

This paper tells us that Diffusion Models aren't magic; they are sequential learners.

They start by learning the simple, basic statistics (like average colors) very quickly.
They take much longer to learn the complex, high-order details (like specific textures and relationships).
They need mathematical constraints to avoid giving up.
Big networks help them escape these traps and learn the hard stuff.

This explains why training these models takes so long and so much data: they are literally climbing a mountain, starting with the easy slope at the bottom and slowly, painfully, making their way to the complex peak at the top.

1. Problem Statement

Diffusion models have achieved state-of-the-art performance in generative modeling, yet the theoretical understanding of their learning dynamics remains limited compared to supervised learning. A key phenomenon in supervised learning is the simplicity bias, where neural networks learn simple features (e.g., pairwise correlations) before complex features (e.g., higher-order correlations).

The paper addresses the open question: Do diffusion models exhibit a similar "distributional simplicity bias"? Specifically, do they learn low-order statistics (mean, covariance) before high-order statistics (cumulants), and what are the sample complexity requirements for learning these different orders of statistics?

2. Methodology

The authors employ a dual approach combining empirical experiments on real image data with rigorous theoretical analysis on a solvable mathematical model.

A. Empirical Analysis

Datasets: CIFAR-10 and CelebA.
Architecture: Standard U-Net denoisers trained via Denoising Diffusion Probabilistic Models (DDPM).
Experimental Design: The authors evaluate the model's test loss on "clone" datasets. These clones are Gaussian distributions fitted to the real data's statistics:
- Mean Clone: Matches the mean of real data, identity covariance.
- Mean + Cov Clone: Matches both mean and covariance of real data.
Logic: If the model performs equally well on real data and a specific clone, it implies the model has not yet learned statistics beyond those present in the clone.

B. Theoretical Framework

Data Model: The Mixed Cumulant Model (MCM). This generates inputs that are isotropically Gaussian in most directions but contain two "spikes":
1. Covariance Spike ( $u$ ): A direction with higher variance (pairwise correlation).
2. Cumulant Spike ( $v$ ): A direction with non-Gaussian distribution (higher-order correlation), requiring higher-order statistics to detect.
Denoiser Architecture: A rank-1 non-linear autoencoder with a skip connection: $S_w(x) = -x - \sigma(w \cdot x)w$ .
Optimization: Analysis of Projected Stochastic Gradient Descent (pSGD) on the unit sphere to ensure the weight vector maintains unit norm, preventing trivial solutions.
Mathematical Tools:
- Hermite Decomposition: Expanding the loss function and the data likelihood ratio in the Hermite polynomial basis.
- Stein's Lemma: Used to simplify gradients by integrating out the Gaussian noise.
- Information Exponent: Adapting concepts from single-index models to diffusion settings.

3. Key Contributions

1. Identification of the "Diffusion Information Exponent" ( $k^*$ )

The authors define a scalar invariant, the diffusion information exponent $k^*$ , which governs the sample complexity required to learn a specific statistical feature.

$k^*$ is the smallest integer $k$ such that the $k$ -th Hermite coefficient of the data likelihood ratio and the $(k-1)$ -th Hermite coefficient of the effective nonlinearity are non-zero.
It determines the "hardness" of the inference task.

2. Proof of Distributional Simplicity Bias

The paper rigorously proves that diffusion models learn statistics sequentially:

Pairwise Statistics (Covariance): Learned at linear sample complexity ( $O(d)$ or $O(d \log d)$ ).
Higher-Order Statistics (Cumulants): Learned at cubic sample complexity ( $O(d^3)$ ) if the latent variables are uncorrelated.
This creates a clear separation of timescales, confirming the simplicity bias.

3. Role of Latent Correlations

The authors prove that if the latent variables governing the covariance spike and the cumulant spike are correlated, the sample complexity for learning the higher-order spike drops significantly (from cubic to quasi-linear). This explains why real-world data (where features are often correlated) might be learned faster than the worst-case theoretical bounds suggest.

4. Critical Importance of Spherical Constraints

A major technical contribution is the analysis of unconstrained SGD vs. Projected SGD (pSGD):

Projected SGD: Successfully learns the spikes by keeping the weight vector on the unit sphere.
Unconstrained SGD: For many common activation functions (e.g., ReLU, tanh, sigmoid) and optimal denoisers, unconstrained SGD gets trapped in a trivial solution ( $w=0$ ).
Mechanism: The removal of the spherical constraint introduces a radial term in the gradient. For specific nonlinearities, this term acts as a strong weight decay that overwhelms the signal term, causing the weights to collapse to zero.
Mitigation: Over-parameterization (wide networks) and depth can help escape this trivial solution, even without spherical constraints.

4. Results

Empirical Results

Sequential Learning: On CIFAR-10, the model achieves the same test loss on real images and "Mean + Cov" clones for the first $\approx 10^3$ training steps.
Specialization: Only after $\approx 10^3$ steps does the loss on real images drop below the loss on the Gaussian clones, indicating the model has started exploiting higher-order correlations.
Generative Quality: Visual samples (Fig 1) show that early-stage models generate blurry, Gaussian-like images, while later-stage models generate sharp, realistic images.

Theoretical Results

Sample Complexity Thresholds:
- For a cumulant spike with information exponent $k^*=4$ (typical for 4th-order cumulants), the model requires $\Omega(d^3)$ samples to recover the direction $v$ .
- For a covariance spike ( $k^*=2$ ), the model requires $\Omega(d \log d)$ samples.
Convergence Failure: Proposition 4.7 proves that for unconstrained SGD with certain nonlinearities, the overlap between the learned weight and the true spike converges to 0, meaning the model fails to learn.
Over-parameterization: Experiments (Fig 3) show that wide two-layer autoencoders can learn the higher-order spike even with nonlinearities that cause failure in single-neuron models, suggesting repulsive interactions between neurons help escape the $w=0$ trap.

5. Significance

Theoretical Foundation: This work provides the first rigorous theoretical explanation for the "easy-to-hard" learning trajectory observed in diffusion models, linking it to the order of statistical cumulants.
Optimization Insights: It highlights a critical, often overlooked detail in diffusion training: the necessity of spherical constraints (or mechanisms like over-parameterization) to avoid trivial solutions. This has direct implications for how diffusion models should be optimized in practice.
Sample Complexity: It establishes precise bounds on how much data is needed to learn different levels of complexity, explaining why diffusion models might require massive datasets to capture fine-grained details (high-order correlations) compared to coarse structures.
Generalization: The framework extends the concept of "information exponents" from supervised learning to the generative diffusion setting, offering a new lens to analyze learning dynamics in score-based models.

In summary, the paper demonstrates that diffusion models learn by first mastering the "skeleton" of the data distribution (mean and covariance) before filling in the "flesh" (higher-order correlations), a process governed by the diffusion information exponent and heavily influenced by optimization constraints and network architecture.