Memorization to Generalization: Emergence of Diffusion Models from Associative Memory

1. Problem Statement

Diffusion Models (DMs) have achieved state-of-the-art performance in generative modeling but face significant challenges regarding memorization (overfitting to training data) and generalization (creating novel, unseen samples).

• The Gap: While recent literature acknowledges the transition from memorization to generalization as training data size ($K$) increases, the intermediate regime is poorly understood. Most works view memorization as a negative artifact to be eliminated by simply increasing data size, without exploring the theoretical mechanisms governing the transition.
• The Question: Can the theoretical framework of Dense Associative Memories (DenseAMs)—specifically the concept of "spurious states" (emergent local minima not corresponding to training data)—explain the generative capabilities and the memorization-generalization transition in Diffusion Models?

2. Methodology & Theoretical Framework

The authors bridge the gap between DMs and DenseAMs by analyzing the energy landscape of diffusion models.

A. Theoretical Link: DMs as DenseAMs

• Energy Formulation: The authors derive the energy function ($E_{DM}$) of a diffusion model from the score-matching objective. They show that for a dataset of size $K$, the energy landscape of a DM is mathematically isomorphic to a Dense Associative Memory model:
  $$E_{DM}(x_t, t) \approx -2\sigma^2 t \log \left[ \sum_{\mu=1}^K \exp\left(-\frac{\|x_t - \xi_\mu\|^2}{2\sigma^2 t}\right) \right]$$
  Here, training data points $\xi_\mu$ act as memories, and the noise variance $\sigma^2 t$ acts as an effective temperature ($\beta^{-1}$).
• Dynamics: The reverse (denoising) process of a DM is interpreted as a dynamical convergence toward attractor states (local minima) in this energy landscape.

B. Classification of Sample States

Based on the energy landscape theory, the authors define three distinct phases of sample generation:

1. Memorized States: Local minima corresponding exactly to training data points. These have large basins of attraction.
2. Spurious States: Emergent local minima that do not correspond to any training data point but are stable attractors. In traditional AM theory, these are "errors," but in DMs, they represent the onset of creativity.
3. Generalized States: Samples lying on a continuous, flat manifold of low energy. These have near-zero basins of attraction, meaning small perturbations lead to different outputs, ensuring diversity.

C. Empirical Detection Metrics

To identify these states empirically, the authors propose a detection framework using two datasets: the training set ($S$) and a large synthetic set ($S'$).

• Memorized: Samples in $S'$ that are duplicates of samples in $S$ (distance $d < \delta_m$).
• Spurious: Samples in $S'$ that are duplicates of other samples in $S'$ (indicating a stable basin) but are not in $S$ (distance $d > \delta_m$).
• Generalized: Unique samples in $S'$ with no duplicates in either set.
• Geometric Analysis: They further characterize these states using:
  • Basin Volume: Calculated via critical time ($t_c$) required to recover a sample from noise.
  • Energy Curvature: Measured via the singular values of the score function's Jacobian (Hessian of energy).

3. Key Contributions

1. Theoretical Unification: Established a rigorous theoretical connection showing that DMs are effectively Dense Associative Memories, where the generative process is a memory retrieval task.
2. Discovery of Spurious States as Generative Signatures: Identified spurious states as a critical intermediate phase. Unlike in Hopfield networks where spurious states are detrimental, in DMs, they are the first signs of generalization, representing the model's ability to create stable, novel patterns before entering the full generalization regime.
3. Characterization of the Transition: Mapped the memorization-to-generalization transition as a three-phase evolution:
   • Phase 1 (Small $K$): Dominance of memorized states (sharp minima at training points).
   • Phase 2 (Critical $K$): Emergence of spurious states (new minima appear; memorization drops).
   • Phase 3 (Large $K$): Collapse of discrete minima into a continuous manifold (generalization).
4. Scalability Validation: Demonstrated that these phenomena persist even in large-scale models like Stable Diffusion trained on massive datasets (LAION), proving the theory applies beyond toy models.

4. Key Results

• Phase Transition Dynamics: Experiments on MNIST, Fashion-MNIST, CIFAR-10, and LSUN-Church show a clear peak in the fraction of spurious samples at the boundary between memorization and generalization. As $K$ increases, memorized samples decrease, spurious samples rise and then fall, and generalized samples rise.
• Basin of Attraction Hierarchy:
  • Memorized: Largest basin volume (high stability).
  • Spurious: Intermediate basin volume (stable but novel).
  • Generalized: Near-zero basin volume (unstable, high diversity).
• Curvature Spectra:
  • Memorized and spurious samples exhibit high energy curvature (large singular values in the Jacobian), indicating sharp minima.
  • Generalized samples exhibit low curvature (flat manifold).
  • As $K$ increases, the curvature of training data points decreases, signaling the transition to a smooth manifold.
• Stable Diffusion Analysis: Even in Stable Diffusion, the authors identified candidate spurious states (e.g., a shirt with one sleeve, a plane with a unique wing shape) that are stable attractors (reproducible) but not present in the training data, confirming the theory holds for high-dimensional, real-world models.

5. Significance and Impact

• Reframing Memorization: The paper shifts the perspective on memorization from a purely negative security/privacy risk to a fundamental phase in the learning trajectory of generative models.
• Understanding Creativity: It provides a physical/mathematical explanation for "creativity" in AI: it emerges from the formation of spurious states (new attractors) before the model fully learns the continuous data manifold.
• Privacy and Security: Understanding the "spurious" phase helps in distinguishing between exact data leakage (memorization) and the generation of plausible but novel data (generalization/spurious), which is crucial for developing better privacy-preserving training methods.
• Theoretical Tool: The DenseAM lens offers a new toolkit for analyzing the internal mechanics of diffusion models, potentially guiding the design of architectures that control the memorization-generalization trade-off more effectively.

In summary, the paper argues that generalization in Diffusion Models is not the absence of memorization, but the emergence of a specific type of memory structure (spurious states) that bridges the gap between rote memorization and true creativity.