📊 statistics

A Random Matrix Theory Perspective on the Consistency of Diffusion Models

This paper employs a random matrix theory framework to demonstrate that the consistency of diffusion models across non-overlapping data splits stems from shared Gaussian statistics, revealing how finite dataset size and spectral properties systematically shape the expectation and variance of generated samples.

Original authors: Binxu Wang, Jacob Zavatone-Veth, Cengiz Pehlevan

Published 2026-07-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Binxu Wang, Jacob Zavatone-Veth, Cengiz Pehlevan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Mystery: Why Do AI Artists Agree?

Imagine you hire two different chefs to bake a cake using the exact same recipe book, but they are given two different, non-overlapping piles of flour and sugar from the same giant warehouse. You might expect their cakes to taste slightly different because their ingredients came from different spots in the warehouse.

However, with modern AI image generators (called Diffusion Models), something strange happens. If you give two different AI models the exact same "seed" (a random starting noise pattern), they produce images that look almost identical—even if they were trained on completely different halves of the data.

The paper asks: Why do these independent models agree so perfectly?

The Core Discovery: It's All About the "Average"

The authors discovered that the reason for this agreement is surprisingly simple. Even though the AI is complex, the part of the data that matters most for this consistency is just the basic statistics: the average color, the average shape, and how things usually vary together (like how eyes and noses usually sit on a face).

Think of it like this: If you ask two people to describe a "typical face" based on a photo album, they will both describe a face with eyes in the middle and a nose below them. They might miss the tiny freckles or the specific hairline of a specific person, but they will agree on the "average" structure. The paper argues that diffusion models are mostly just learning this "average" structure, which is why they agree.

The Tool: Random Matrix Theory (The "Math Telescope")

To prove this, the authors used a branch of math called Random Matrix Theory (RMT).

The Analogy: Imagine trying to predict the weather by looking at a single raindrop. It's impossible. But if you look at a billion raindrops falling at once, patterns emerge. RMT is a mathematical telescope that lets us look at the "billion raindrops" of data (the huge matrices inside the AI) to see the underlying patterns without getting lost in the noise.

Using this telescope, the authors built a theory that explains exactly how the AI behaves when it doesn't have infinite data.

Three Key Findings (The "Rules of the Game")

The paper breaks down the behavior of these models into three main concepts:

1. The "Noise Filter" Effect (Renormalization)

When an AI tries to learn from a limited dataset, it gets a bit nervous. It starts to think, "Is this tiny detail real, or is it just random noise?"

The Metaphor: Imagine you are trying to hear a whisper in a crowded room. If you aren't sure, you might turn down the volume on the high-pitched sounds (the details) and focus only on the deep, rumbling bass (the main structure).
The Result: The AI "overshrinks" the details. It pulls the generated image closer to the average face, making the textures smoother and less detailed than they should be. The math shows that the AI effectively turns up the "noise level" in its own head, causing it to ignore the subtle, low-variance details unless it has a massive amount of data.

2. The "Directional Bias" (Anisotropy)

Not all details are equally hard to learn.

The Metaphor: Imagine a landscape with big, rolling hills (major features) and tiny pebbles (minor details). If you have a small map, you can easily draw the big hills. But the tiny pebbles? You might miss them entirely, or draw them in the wrong place.
The Result: The AI is very consistent about the "big hills" (major features like the shape of a face) because those are easy to see in any data split. But it disagrees more on the "pebbles" (fine details) because those are harder to pin down with limited data. The paper provides a formula to predict exactly which details will be shaky.

3. The "Location Matters" Effect (Inhomogeneity)

The AI is also more confused about some parts of the image than others.

The Metaphor: If you are drawing a crowd, you are very good at drawing the people standing in the front row (where the data is dense). But if you try to draw someone standing way off in the corner (where data is sparse), your drawing might wobble.
The Result: The AI's consistency depends on where the image "sits" in the data. If an image is a mix of common features, the AI is confident. If it's a weird mix of rare features, the AI's output becomes more variable across different training runs.

The "Magic Seed" Connection

The paper also explains why some random seeds produce "better" or more consistent images than others.

The Analogy: Think of the random noise seed as a set of instructions. Some instructions tell the AI to focus on the "big hills" (the main structure), while others tell it to focus on the "pebbles" (the noise).
The Result: If your seed aligns with the "big hills" (the main data patterns), the AI produces a consistent, clear image. If your seed tries to force the AI to focus on the "pebbles" (rare, noisy directions), the result is shaky and inconsistent.

The Bottom Line

The paper concludes that the "magic" of diffusion models isn't just magic; it's math.

Consistency is natural: Because different data splits share the same basic "average" statistics, the models naturally agree on the main structure.
Disagreement is predictable: The places where models disagree (the fine details) are not random; they follow a strict mathematical rule based on how much data the model has and how "noisy" the image is.
Deep Learning vs. Simple Math: Even though modern AI uses complex neural networks (Deep Learning), they often behave just like a simple, linear math formula when it comes to this consistency. The complex networks are just adding a little bit of extra flair, but the foundation is built on these simple statistical rules.

In short: Diffusion models are like two different artists looking at the same blurry photo. They will both agree on the general shape of the object, but they will argue about the tiny details unless you give them a much sharper photo (more data).

Technical Summary: A Random Matrix Theory Perspective on the Consistency of Diffusion Models

Problem Statement
Diffusion models trained on different, non-overlapping subsets of the same dataset often produce strikingly similar outputs when conditioned on the same noise seed, even when using different architectures (e.g., UNet vs. DiT). This phenomenon of "consistency" contrasts with other generative frameworks like GANs and VAEs, where latent space rotations lead to significant run-to-run variability. While previous work has observed this effect, the underlying mechanisms—specifically how finite dataset size and data statistics shape the stability of the learned denoiser and sampling map—remain theoretically unexplained. The paper seeks to quantify the expectation and variance of these learned mappings across independent data splits, distinguishing between memorization (where models reproduce training samples) and generalization (where they capture universal statistical regularities).

Methodology
The authors employ Random Matrix Theory (RMT) to analyze diffusion models, beginning with a tractable linear denoiser framework before extending insights to deep networks.

Linear Setting & Deterministic Equivalence: The core technical tool is deterministic equivalence (DE), which allows random empirical covariance matrices ( $\hat{\Sigma}$ ) to be replaced by deterministic surrogates in the high-dimensional limit ( $d, n \to \infty$ with $d/n = \gamma$ ).
- The authors derive a self-consistent equation for a renormalized noise scale, $\kappa(\sigma^2)$ , which replaces the raw noise variance $\sigma^2$ in the population covariance $\Sigma$ . This relation is governed by the Silverstein equation:
  $\kappa(\lambda) - \lambda = \gamma \kappa(\lambda) \text{tr}[\Sigma(\Sigma + \kappa(\lambda)I)^{-1}]$
- This framework is extended to fractional matrix powers (specifically $\Sigma^{1/2}$ ) required for the full sampling trajectory (Wiener filter) using integral representations (Balakrishnan's formula) and two-point deterministic equivalence identities.
Theoretical Derivations:
- Expectation: The theory predicts that finite data causes the learned denoiser to effectively operate with a higher noise level ( $\sigma^2 \to \kappa(\sigma^2)$ ), leading to an "overshrinkage" of low-variance directions toward the dataset mean.
- Fluctuation (Variance): The variance of the denoiser across dataset splits is shown to factorize into three components:
  - Anisotropy: Uncertainty is maximized along eigenmodes where the population eigenvalue $\lambda_k$ matches the renormalized noise $\kappa(\sigma^2)$ .
  - Inhomogeneity: Uncertainty is amplified for input points displaced along high-variance directions.
  - Global Scaling: The overall variance scales inversely with dataset size $n$ , modified by the renormalization effects at smaller $n$ .
Empirical Validation: The theoretical predictions are validated against deep diffusion models (UNet and DiT) trained on various datasets (FFHQ, AFHQ, CIFAR, LSUN) at different dataset sizes ( $n \in \{300, 1000, \dots, 30000\}$ ). The study distinguishes between a memorization regime (small $n$ , where models reproduce training data) and a renormalization/generalization regime (larger $n$ , where models align with the linear Gaussian predictor).

Key Contributions

Linear Origin of Consistency: The paper demonstrates that shared Gaussian statistics (mean and covariance) across data splits are sufficient to predict cross-split agreement, explaining why linear denoisers already exhibit high consistency.
Finite-Sample Renormalization: It formalizes how finite datasets induce a renormalized noise scale $\sigma^2 \to \kappa(\sigma^2)$ , explaining the systematic overshrinkage of low-variance modes and the pull toward the dataset mean.
Variance Law: The authors derive a factorized variance formula for cross-split disagreement, identifying anisotropy (dependence on eigenmode variance), inhomogeneity (dependence on input location), and global scaling (dependence on $n$ ) as the three governing factors.
Fractional-Power Equivalence: The work extends deterministic equivalence tools to fractional matrix powers, enabling closed-form predictions for entire sampling trajectories, not just single-step denoising.
Deep Network Validation: The paper confirms that these linear RMT principles qualitatively govern the consistency of deep UNet and DiT models in the non-memorization regime, predicting where and how samples deviate across splits.

Results

Consistency Mechanism: Experiments show that diffusion models trained on disjoint splits generate visually similar images from the same noise seed. This similarity is captured by a linear Gaussian predictor (Wiener filter) using the empirical moments of the splits.
Overshrinkage: In the generalization regime, generated samples exhibit reduced variance along low-to-mid eigenmodes compared to the population distribution, effectively "overshrinking" toward the mean. This bias vanishes as $n$ increases and the empirical spectrum converges to the population spectrum.
Anisotropy and Inhomogeneity:
- Spectral: Disagreement between models trained on different splits is highest for eigenmodes where the eigenvalue $\lambda_k \approx \kappa(\sigma^2)$ . Lower eigenmodes (fine details) require significantly larger datasets to achieve consistency compared to top eigenmodes (global structure).
- Spatial: Samples generated from noise seeds that project strongly onto high-variance directions of the data manifold exhibit larger cross-split deviations.
Deep Network Alignment: In the non-memorization regime ( $n \ge 3000$ for FFHQ64), deep network samples converge toward the linear predictor. The RMT predictions of spatial inhomogeneity (which seeds yield larger deviations) correlate significantly with empirical cross-split MSE in deep networks (e.g., Spearman $r \approx 0.33$ for UNet on FFHQ64 with $n=30k$ ).
Counterfactuals: When training data is intentionally partitioned to mismatch mean or covariance (e.g., stratifying by a principal component), the consistency effect breaks down, confirming that shared Gaussian statistics are the driver.

Significance
The paper provides a principled baseline for reproducibility in diffusion training. By linking the spectral properties of data to the stability of generative outputs, it clarifies that the remarkable consistency of diffusion models arises not from complex architectural inductive biases alone, but from the stability of the underlying Gaussian statistics shared across data splits. The work suggests that the "creativity" or generalization of diffusion models is constrained by these linear statistical regularities, offering a theoretical explanation for why independently trained models often map the same noise seed to similar structures. This framework helps distinguish between true generalization and memorization, providing a tool to predict the reliability of generated samples based on dataset size and data spectrum.