🤖 machine learning

Adynamical systems view of training generativemodels and the memorization phenomenon

This paper employs a dynamical systems perspective, leveraging two-time-scale dynamics in stochastic gradient descent and recent results on model collapse, to provide a system-theoretic explanation for the memorization phenomenon in generative models where outputs remain static for extended periods during training.

Original authors: Siva Athreya, Chiranjib Bhattacharya, Vivek S. Borkar

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siva Athreya, Chiranjib Bhattacharya, Vivek S. Borkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why AI Gets "Stuck" on the Same Ideas

Imagine you are teaching a robot to paint pictures. You want it to learn the general style of a landscape so it can create new beautiful landscapes. However, you notice a strange glitch: after a while, the robot stops creating new art and starts painting the exact same tree, or the same cloud, over and over again for hours. It has "memorized" a specific output instead of learning the general concept.

This paper explains why this happens. The authors argue that this isn't just a bug in the code; it's a natural consequence of how the robot learns, specifically because of how fast it takes steps while learning.

They view the training process not as a static math problem, but as a dynamic system—like a ball rolling down a bumpy hill.

The Core Concept: Two Speeds of Learning

The authors suggest that the robot's brain (the model) has two different parts that learn at very different speeds. Think of it like a hiker carrying a heavy backpack:

The Fast Hiker (The "Fast" Variables): This part of the brain learns quickly. It's like the hiker's legs, which are constantly adjusting to the terrain.
The Slow Hiker (The "Slow" Variables): This part learns very slowly. It's like the hiker's backpack, which shifts weight only occasionally.

In the real world, the "loss function" (the measure of how bad the robot's painting is) depends heavily on the fast variables and only slightly on the slow ones. Because of this, the fast variables zoom around trying to find a good spot, while the slow variables barely move.

The "Collapse" Phenomenon: Getting Trapped in a Hole

First, the authors explain a simpler problem called "Collapse."

Imagine the robot is trying to find the lowest point in a valley (the best solution).

The Ideal Scenario: If the robot takes tiny, shrinking steps (like a cautious explorer), it will eventually settle at the very bottom of the valley and stay there.
The "Collapse" Scenario: If the robot takes constant, steady steps (like a person walking at a fixed pace), it might overshoot the bottom, bounce back up, and get stuck in a small local dip.

In the paper's language, if the robot is just trying to generate data without any "noise" to shake it loose, it will eventually collapse into a single, fixed output. It stops being a generator and becomes a recorder of one specific image. Mathematically, this is like a ball rolling into a hole and staying there forever.

The "Memorization" Phenomenon: The Lazy Oscillator

Now, here is the twist. The authors explain that Memorization is a mix of "Collapse" and "Drift."

Because the robot has those two different speeds (fast and slow variables) and is taking constant steps, something interesting happens:

The Fast Part Collapses: The fast variables quickly settle into a local dip (a specific image style). The robot starts outputting that image repeatedly. This looks like "memorization."
The Slow Part Drifts: However, the slow variables are still moving, just very slowly. They are slowly pushing the "landscape" of the problem.
The Jump: Eventually, the slow drift pushes the fast variables out of their current dip. The robot suddenly jumps to a different dip (a different image) and settles there for a long time.

The Analogy: Imagine a relaxation oscillator (like a firefly blinking or a heartbeat).

The robot spends a long time "sleeping" in one spot (memorizing one image).
Then, slowly, the pressure builds up until it "jumps" to a new spot.
It stays there, sleeps again, and repeats.

This is Memorization: The robot isn't stuck on one image forever; it gets stuck on one image for a long stretch, then switches to another, then another. It's an intermittent cycle of getting stuck and then slowly drifting away.

Why Does the Step Size Matter?

The paper highlights that this happens specifically because the learning algorithm uses a constant step size (a fixed pace).

Small Steps: If the robot takes tiny steps, it settles down nicely and learns the general shape without getting stuck in loops.
Constant Steps: If the robot keeps taking the same size step, it creates a "tug-of-war." The fast variables want to settle, but the constant momentum keeps pushing them.
Big Steps (Noise): The authors note that if you make the steps very large, you introduce so much "noise" (shaking) that the robot can't settle into a memorization loop at all. It keeps bouncing around too much to get stuck. This explains why some training methods that use large steps avoid memorization.

Summary of the Authors' Argument

High Dimensions create Two Speeds: Because AI models have so many parameters, some learn fast and some learn slow.
Constant Steps create Loops: Using a fixed learning rate prevents the system from settling perfectly.
Collapse is the Base: Without noise, the system would just freeze on one output.
Memorization is "Drifting Collapse": Because the slow variables keep moving, the system gets stuck on an output for a while, then slowly drifts to a new one, creating a cycle of repetition.

The paper doesn't offer a new tool to fix this immediately, but it provides a mathematical map of why this happens. It tells us that memorization isn't a random error; it's a predictable behavior of a system moving at two different speeds with a fixed pace.

Technical Summary: A Dynamical Systems View of Training Generative Models and the Memorization Phenomenon

Problem Statement
The paper addresses the phenomenon of "memorization" in generative models, particularly diffusion models. Empirically, during the training process, these models occasionally enter phases where they regenerate the same or highly similar samples for significant durations, rather than producing diverse outputs. While existing literature often analyzes this through static statistical mechanics or loss landscape thermodynamics, this work proposes a dynamical systems perspective. The authors argue that memorization is not merely a static property of the loss landscape but an emergent behavior of the training algorithm's temporal evolution, specifically driven by the use of constant step sizes in Stochastic Gradient Descent (SGD) and the presence of Markov noise.

Methodology and Framework
The authors construct a theoretical framework based on three core components:

Two-Time-Scale Dynamics via Austin's Theorem:
Drawing on Austin [2016], the authors posit that in high-dimensional parameter spaces with Lipschitz constraints, the loss function exhibits a strong dependence on a subset of parameters and a weak dependence on the rest. This motivates a stylized model where the objective function $F(z)$ (with $z=(x,y)$ ) takes the form $F(z) = f(x, \epsilon y)$ for $0 < \epsilon \ll 1$ . This structure naturally induces two distinct time scales in the SGD updates: a fast scale for $x$ and a slow scale for $y$ .
SGD with Markov Noise:
The training algorithm is modeled as a stochastic approximation scheme with Markov noise. The noise $\{Z_n\}$ arises from the generative model itself (e.g., the diffusion process), which is being tuned by the parameters. The transition probabilities of this noise are modulated by the current parameters. The authors utilize the theory of stochastic approximation with Markov noise (Benveniste et al. [1990]; Borkar [2025c]) to analyze the system, noting that the noise evolves on a "natural" time scale much faster than the "algorithmic" time scale of the parameters.
Collapse as an Absorbing Markov Chain:
The paper defines "collapse" formally using results from Borkar [2025a]. In a generative model setting, the sequence of probability measures generated by the model forms a time-homogeneous Markov chain. Under certain conditions (specifically when the mixing parameter $a=0$ ), this chain converges almost surely to a Dirac measure concentrated at a random point. This asymptotic collapse to a single output is the mathematical definition of the collapse phenomenon.

Key Contributions and Results

Differentiation of Step Sizes:
The paper contrasts decreasing step size SGD (which leads to convergence) with constant step size SGD (the focus of this work). In the decreasing step size regime, the system converges to a global minimum, leading to pure collapse (Theorem 7). However, the authors argue that the practical phenomenon of memorization is specific to constant step size SGD.
Asymptotic Behavior of Constant Step Size SGD:
Leveraging recent results by Azizian et al. [2024] and Hwang [1981], the authors analyze the limiting distribution of constant step size SGD. They establish that:
- The iterates concentrate on the minima of a "large deviations rate function" (an energy functional), which does not necessarily coincide with the global minimum of the loss function.
- The stationary distribution favors "flatter" minima (those with lower determinants of the Hessian of the rate function), explaining the empirical preference of SGD for simpler minima.
- The system behaves like a diffusion process where the step size acts as a temperature parameter.
Mechanism of Memorization (Theorem 11):
The central result is the explanation of memorization as an intermittent collapse.
- The system operates on two time scales: the fast variable $x$ tracks a local minimum $\lambda(y)$ of the fast dynamics, while the slow variable $y$ drifts.
- Because the step size is constant, the system does not converge to a single point. Instead, the slow variable $y$ drifts, causing the fast variable $x$ to track a moving minimum.
- When the slow variable $y$ remains in a region where the fast dynamics have a stable attractor, the system exhibits "memorization" (repeated outputs).
- However, because $y$ continues to drift (aided by noise and the slow time scale), the system eventually exits this region, causing a transition to a different attractor.
- This results in a "relaxation oscillation" behavior: long stretches of repetition (memorization) punctuated by relatively rapid transitions to new states. This distinguishes memorization from the rare, noise-driven jumps between equilibria seen in standard Freidlin-Wentzell theory; here, the transitions are driven by the deterministic drift of the slow variable.
Role of Noise and Step Size:
The authors explain that larger step sizes increase the noise level in the iteration, which can disrupt the stable tracking required for memorization, thereby discouraging the phenomenon. This aligns with empirical observations (e.g., Wu et al. [2025]) that large step sizes mitigate memorization.

Significance and Claims
The paper claims to provide a "system theoretic explanation" for memorization, shifting the focus from static loss landscape analysis to the temporal dynamics of the learning algorithm.

Unified Perspective: It unifies the phenomena of "collapse" and "memorization," arguing that memorization is essentially a collapse phenomenon that is modulated by the score learning scheme and the slow drift of parameters, rendering it intermittent rather than permanent.
Emergent Time Scales: It highlights that the two time scales in the training of high-dimensional generative models are emergent properties of the problem structure (via Austin's theorem) and the constant step size, rather than being artificially imposed.
Modest Scope: The authors acknowledge that their model involves idealizations, such as assuming the generated measures form a conditional barycenter and using a stylized two-scale loss function. They note that in reality, approximation errors act as persistent external noise, meaning memorization is often a "corruption of collapse" rather than a pure Dirac delta concentration. Consequently, memorization manifests as a bias toward noisy versions of past outputs rather than exact duplication.

The work does not propose new experimental algorithms or specific engineering fixes but offers a theoretical lens to understand why memorization occurs in constant step size SGD and how it relates to the broader dynamics of stochastic approximation in high dimensions.