Information Spreading in Diffusion Models from Effective Field Theory

Technical Summary: Information Spreading in Diffusion Models from Effective Field Theory

Problem Statement
Diffusion models are a dominant paradigm in generative AI, capable of producing high-quality samples by gradually denoising pure noise. However, the underlying dynamics of these models, particularly how spatial structure and correlations emerge during the reverse (denoising) process, are often complex and difficult to analyze analytically. While previous works have utilized information theory (e.g., the I-MMSE relationship) to study mutual information evolution, there is a lack of a simplified, physics-based framework that explains how locality and symmetry constraints in convolutional architectures govern the spread of information and the formation of spatial correlations. Specifically, the transition from a "diffusive" regime (where information spreads) to a "snapping" or memorization regime (where patches lock onto training data) requires a unified theoretical description.

Methodology
The authors apply the formalism of Effective Field Theory (EFT), a concept from theoretical physics used to describe systems at specific length scales without needing full microscopic detail, to the reverse process of score-matching diffusion models.

1. Theoretical Framework:

   • The authors argue that the inductive biases of locality and translational equivariance inherent in convolutional neural networks (ConvNets) allow the score function (the learned gradient of the log-probability) to be expanded in a series of spatial derivatives.
   • By performing a scaling analysis near the pure noise endpoint ($\bar{\alpha}_t \to 0$), they demonstrate that higher-order derivative terms and nonlinearities become negligible.
   • This leads to a simplified evolution equation for the field $\tilde{\phi}$:
     $$\frac{\partial \tilde{\phi}(x, \bar{\alpha}_t)}{\partial \bar{\alpha}_t} = g_3(\bar{\alpha}_t) \nabla^2 \tilde{\phi}(x, t)$$
     This is identified as the heat equation (or diffusion equation), where the noise schedule parameter $\bar{\alpha}_t$ acts as time.

2. Microscopic Justification:

   • The authors ground this EFT approach in the Equivariant Local Score Machine (ELSM) framework (Kamb & Ganguli, 2025), which provides a microscopic description of the score function as a sum over training patches.
   • They show that in the limit of pure noise, the nonlinear terms in the ELSM expansion are suppressed by powers of $\bar{\alpha}_t$, validating the truncation to the linear diffusion equation.

3. Predictions:

   • The theory predicts that the mutual information (or correlation) between two spatial points $x$ and $y$ should evolve diffusively.
   • Specifically, the correlation coefficient $\rho(x, y; t)$ should depend on a self-similar, scale-invariant variable: $|x - y|^2 / \bar{\alpha}_t$.
   • Consequently, plots of mutual information against spatial distance at different time steps should collapse onto a single curve when plotted against this scaled variable.

4. Experimental Validation:

   • Toy Model ($\pm 1$ Grids): The authors trained a diffusion model on two simple samples (grids of all $+1$ and all $-1$). They derived an analytical solution for the ELSM and verified that the mutual information curves collapse according to the predicted scaling. They also derived the relationship between the diffusion constant $D$ and the network's kernel size $\Delta$, finding $D \sim \Delta^4$ and the correlation length $\xi \sim \Delta$.
   • MNIST Dataset: They trained a ConvNet on the MNIST dataset. Despite the lack of perfect symmetries (e.g., rotational invariance) present in the toy model, they observed that the mutual information growth still exhibited the predicted diffusive scaling at early times, with curve collapse observed against the self-similar variable.

Key Results

• Diffusive Regime: The reverse process begins with a "diffusive regime" where spatial mutual information spreads from uncorrelated noise. The growth of correlations is governed by a heat equation, where the "time" variable is determined by the noise schedule.
• Scaling Collapse: Empirical results on both toy data and MNIST confirm that mutual information curves at different time steps collapse when plotted against the variable $r^2 / \bar{\alpha}_t$, validating the EFT prediction.
• Kernel Size Dependence: In the toy model, the effective diffusion constant $D$ is shown to scale with the fourth power of the convolutional kernel size ($D \sim \Delta^4$), while the resulting correlation length $\xi$ scales linearly with the kernel size ($\xi \sim \Delta$). This suggests that the spatial scale of generated features is fundamentally tied to the network architecture.
• Transition to Snapping: The paper identifies a transition point where the diffusive regime ends and the "snapping" regime begins. In this later regime, local patches decouple and snap to the nearest training data patches (memorization), a behavior described by a different expansion near $\bar{\alpha}_t \to 1$.

Significance and Claims
The paper claims to provide a novel application of Effective Field Theory to deep learning, offering a simplified, analytical description of the complex dynamics of diffusion models.

• Simplification: It demonstrates that the complex, high-dimensional dynamics of the reverse process can be reduced to a simple diffusion equation near the noise endpoint, governed by a few parameters (like the diffusion constant) rather than the full microscopic details of the training data.
• Unification: It connects the "Gaussian" regime described in previous literature with the "Brownian motion" regime, providing a physical intuition for how spatial structure is built.
• Architectural Insight: The work suggests that the spatial scale of information spreading is directly controlled by the network's inductive biases (specifically the kernel size), offering a potential method to infer architectural properties (like effective kernel size) from the dynamics of the generated samples.
• Scope: The authors modestly note that while the theory works well for convolutional architectures, its applicability to other architectures (like Transformers) or more complex datasets requires further investigation, particularly regarding how locality is inherited from data in those contexts. They also acknowledge that the "snapping" regime and the transition between regimes are not fully captured by the simple linear EFT and require separate analysis.