💻 computer science

Nonequilibrium training dynamics and scaling laws of language models

This paper proposes a stochastic multiscale theory grounded in nonequilibrium physics to explain the mechanistic origin of language model scaling laws, modeling training as a scale-dependent biased random walk that yields analytically derived power laws for loss versus data and model size while identifying a bifurcated two-phase dynamics of context compression and frontier absorption.

Original authors: Zhijie Xu

Published 2026-08-03

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhijie Xu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Learning Curve: From Coffee Swirls to AI Brains

Imagine you are watching a cup of coffee being stirred. At first, the cream swirls in big, slow loops. But as you keep stirring, those big loops break apart into smaller and smaller whirlpools, until the liquid is a chaotic mix of tiny eddies. This is called turbulence, and it's a classic example of a system that is constantly being pushed away from a calm state by an outside force (your spoon). In physics, scientists have long known that these swirling patterns follow strict mathematical rules, where energy flows from big swirls to tiny ones in a predictable cascade.

Now, picture a different kind of chaos: the universe itself. After the Big Bang, tiny clumps of dark matter started to pull together, merging into bigger and bigger galaxies. This is the opposite of the coffee cup—instead of big things breaking into small ones, small things are building up into giants. Yet, surprisingly, both the swirling coffee and the forming galaxies follow similar mathematical patterns. They are both "nonequilibrium" systems, meaning they are busy, changing, and driven by forces that keep them from settling down.

For years, scientists have been trying to understand how Artificial Intelligence (specifically, Large Language Models or LLMs) learns. We know these AI models get better as we feed them more data and give them more "brain power" (parameters), following a neat mathematical rule called a "scaling law." But why? Why do they improve in this specific way? Until now, we've mostly just watched the numbers go up without understanding the engine under the hood. This paper asks: Could the way an AI learns be just like the way coffee swirls or galaxies form? Could the messy, chaotic process of training an AI be described by the same physics that governs the universe?

The Coffee Cup in the Computer

In this paper, author Zhijie (Jay) Xu proposes a wild idea: training a language model is exactly like a game of "hot potato" played across different distances in a story. To understand this, imagine you are trying to predict the next word in a sentence. Sometimes, the answer is right there in the word you just said (a short distance). Other times, you need to remember a character's name from three paragraphs ago (a long distance).

The paper suggests that as the AI trains, it doesn't just learn "everything" at once. Instead, it learns in a specific order, moving information around like energy in a storm. The author introduces a new way to look at the AI's mistakes, called the "loss spectrum." Think of this like a map of how much the AI is struggling at different distances. Is it struggling to remember the word from two steps ago? Or the word from two thousand steps ago?

The paper finds that this map looks exactly like the energy map of a swirling cup of coffee or the distribution of galaxies in space. It suggests that the AI's learning process is a "biased random walk." Imagine a drunk person trying to walk through a city where the streets get harder to navigate the further they go from home. The AI starts by learning the easy, short-distance patterns (like grammar and immediate context). Then, it starts to "walk" further out, learning long-distance connections (like the plot of a whole story).

The Two-Phase Dance: Stretching and Shrinking

The most exciting discovery in this paper is that the AI's training isn't a straight line; it's a two-part dance that happens after a critical moment.

Phase 1: The Big Stretch (The Upscale Cascade)
At the beginning, the AI is like a kid stretching their arms wide. It rapidly learns to reach out and grab information from further and further away. It's expanding its "learning frontier," grabbing long-range connections it didn't understand before. During this phase, the AI is getting better at using the whole story, not just the last sentence.

Phase 2: The Great Split (Bifurcation)
Once the AI reaches a certain point (a specific number of training steps), something magical happens. The learning process splits into two simultaneous modes, like a river splitting into two streams:

Mode A (The Downscale Compressor): This is the "refinement" phase. The AI takes all those big, messy, long-range connections it just learned and compresses them into efficient, short-range shortcuts. It's like taking a long, winding road and building a tunnel through the mountain to get to the same place faster. The AI is getting smarter by making its internal representations more compact and efficient.
Mode B (The Frontier Absorber): At the same time, the AI keeps pushing its frontier outward, continuing to learn new, even longer-range patterns that it hasn't seen yet. It's still stretching its arms, but now it's also tightening its grip on what it already knows.

The paper suggests that this split is why AI models get so good. They aren't just memorizing; they are constantly balancing between "stretching out" to learn new things and "compressing in" to make what they know more efficient.

The Magic Numbers and the "Why"

The author doesn't just guess these patterns; they derive them from the math of how language works. They use a famous rule about language called Heaps' Law, which describes how the number of unique words grows as you read more text. By combining this with the physics of turbulence, the author calculates a specific number, 1/6, that describes how long it takes the AI to learn at different distances.

Using this number, the paper predicts exactly how the AI's performance should improve as you give it more data. The prediction is that the error rate should drop by a factor of 1/10 as the data increases. This matches incredibly well with what we see in real-world experiments (where the number is about 0.095, or very close to 1/10). The paper also suggests that if you make the AI bigger (add more parameters), it should improve in a similar way, but perhaps slightly less efficiently, like a biological organism where bigger bodies need more support systems that don't grow at the same rate as the muscles.

What This Means (and What It Doesn't)

This paper is a "theory," meaning it's a mathematical model that explains how things might be working, based on observations and analogies to physics. It hasn't "proved" that AI is literally a fluid or a galaxy, but it suggests that the math describing those things also describes AI learning.

The author is careful to say that this is a "stochastic multiscale theory"—a fancy way of saying it's a theory about random processes happening at many different sizes. The paper argues against the idea that AI learning is a simple, smooth curve. Instead, it suggests a complex, two-phase process where the AI is constantly juggling between expanding its knowledge and compressing it.

The paper also notes that while the math fits the data for the models they tested (like the Pythia family of models), we need to test this on many different types of AI to be sure. It's a promising new lens, like putting on a pair of glasses that lets us see the hidden physics of how machines learn, but the author admits there is still work to be done to confirm if this "turbulence" is truly the engine driving all AI.

In short, this paper suggests that the secret to how AI learns might be hidden in the same swirling chaos that makes coffee cream mix and galaxies form. By treating the AI's learning process as a physical system driven by the statistics of language, the author provides a new, vivid way to understand why these models get better, and exactly how they do it.

=== SUMMARY ===

Technical Summary: Nonequilibrium Training Dynamics and Scaling Laws of Language Models

Problem Statement

Large Language Models (LLMs) exhibit robust empirical scaling laws where held-out cross-entropy loss decreases as a power law with respect to parameter count ( $N$ ), dataset size ( $D$ ), and compute ( $C$ ). However, the mechanistic origin of these power laws remains poorly understood. Existing literature has identified that natural language possesses multiscale statistical structures (e.g., power-law decay of mutual information) and that training dynamics involve abrupt transitions (e.g., induction heads), but no existing framework quantitatively unifies the static structure of data with the dynamic evolution of training to explain the emergence of scaling laws.

Methodology

The paper proposes a stochastic multiscale theory of LLM training dynamics, drawing a formal analogy between language model training and driven, multiscale nonequilibrium physical systems, specifically turbulence (energy cascade) and cosmic structure formation (mass cascade).

Core Framework

Scale-Dependent Loss Spectrum: The author introduces a context scale variable $r$ (dependency distance in tokens) and define a scale-dependent held-out loss $L_r(r, s)$ . The negative gradient of this loss with respect to scale, $E_X(r, s) \equiv -\partial L_r / \partial r$ , is defined as the loss spectrum, analogous to the energy spectrum in turbulence.
Random Walk in $r$ -Space: Training dynamics are modeled as a biased random walk in the context-scale space ( $r$ -space). The model learns to exploit dependencies at scale $r$ by waiting a characteristic time $\tau_r$ before jumping to the next scale.
Fokker-Planck Equation: The evolution of the loss spectrum is governed by a Fokker-Planck equation with a position-dependent diffusivity derived from a scale-dependent waiting time $\tau_r \propto r^{-\gamma}$ .
Decomposition of Loss: The total loss is decomposed into two components representing distinct learning modes:
- $\Delta L_r(s)$ (Depth): The context-dependent loss, measuring the predictive gain from using full context versus a single token.
- $L_\infty(s)$ (Breadth): The residual full-context loss, measuring the model's inability to exploit available context at the learning frontier.

Key Contributions and Theoretical Derivations

1. Two-Phase Training Dynamics

The theory identifies a bifurcation in training dynamics at a critical step $s^*$ (where $\Delta L_r = L_\infty$ ):

Phase I (Upscale Cascade): Before $s^*$ , the model rapidly expands its predictive reach. The effective context scale $\xi$ grows, and the model captures long-range dependencies (upscale transfer).
Phase II (Bifurcation): After $s^*$ $s^{*}$ , the dynamics split into two simultaneous modes:
- Mode A (Downscale Compression): The model compresses learned large-scale structures into efficient short-range representations. The effective scale $\xi_A$ shrinks ( $\xi_A \propto s^{-3/5}$ ), while the depth $\Delta L_r$ grows slowly ( $\propto s^{1/10}$ ).
- Mode B (Upscale Frontier Absorption): The model continues to absorb new long-range predictive structures at the frontier. The effective scale $\xi_B$ expands ( $\xi_B \propto s^{3/5}$ ), while the residual loss $L_\infty$ decreases ( $\propto s^{-1/10}$ ).

2. Self-Consistent Scaling Exponents

The paper derives the scaling exponents from first principles based on the statistical properties of natural language:

Waiting Time Exponent ( $\gamma$ ): By modeling the waiting time $\tau_r$ as a function of attempt frequency (driven by residual loss) and success probability (driven by the number of independent predictive cues), the author links $\gamma$ to Heaps' law ( $M(r) \propto r^{\alpha_H}$ ). Assuming $\alpha_H \approx 1/2$ and incoherent (RMS) aggregation of gradient signals, they derive:
$\gamma = \frac{\alpha_H}{3} = \frac{1}{6}$
Loss Spectrum: This yields a universal loss spectrum in the inertial range:
$E_X(r) \propto r^{-(1+\gamma)} = r^{-7/6}$
This is presented as the language-modeling analogue to the Kolmogorov $-5/3$ energy spectrum in turbulence.

3. Derivation of Empirical Scaling Laws

Using the derived exponents, the theory predicts the observed scaling laws without fitting:

Loss vs. Data ( $D$ ): In Phase II, the loss scales as $L \propto D^{-\alpha_D}$ . The theory predicts:
$\alpha_D = \frac{\gamma}{2 - 2\gamma} = \frac{1/6}{2 - 1/3} = \frac{1}{10} = 0.1$
This aligns closely with the empirical Kaplan scaling of $L \propto D^{-0.095}$ .
Loss vs. Model Size ( $N$ ): Treating model size as an alternative "clock" for the Langevin time, the theory predicts $L \propto N^{-\alpha_N}$ .
$\alpha_N = \frac{\lambda \gamma}{2 - 2\gamma}$
where $\lambda \leq 1$ encodes the efficiency of parameter utilization. If $\lambda = 3/4$ (suggesting sublinear capacity gains due to redundancy), $\alpha_N = 3/40 \approx 0.075$ , consistent with the empirical Kaplan value of $\approx 0.076$ .
Combined Scaling: The full scaling form $L(N, D) = [(N_c/N)^\lambda + D_c/D]^{\alpha_D}$ is recovered by harmonically combining the two independent bottlenecks (data and capacity).

Results and Validation

The author validates the theory using checkpoints from the Pythia model suite (70M to 1.4B parameters) and GPT-2:

Loss Spectrum: Empirical measurements of $E_X(r, s)$ confirm the predicted power-law tail $P_X(r) \propto r^{-7/6}$ in Phase II.
Phase Transitions: The data shows a clear transition at $s^*$ where the effective scale $\xi$ stops growing monotonically and bifurcates, with $\xi_A$ contracting and $\xi_B$ expanding.
Scaling Fits: The evolution of $\Delta L_r$ and $L_\infty$ follows the predicted power laws ( $s^{1/10}$ and $s^{-1/10}$ respectively) in Phase II.
Model Size Scaling: Fits to Pythia models at convergence show $L_\infty \propto N^{-1/10}$ (for total parameters) and $\Delta L_r \propto N^{3/40}$ (for non-embedding parameters), supporting the hypothesis of sublinear capacity scaling.

Significance and Claims

The paper claims to provide a mechanistic origin for LLM scaling laws by framing training as a nonequilibrium cascade process. Key claims include:

Unification: It quantitatively ties the static structure of language (Heaps/Zipf laws) to the dynamic evolution of training via a stochastic framework.
Predictive Power: The scaling exponent $\alpha_D = 1/10$ is derived, not fitted, from the heavy-tailed statistics of natural language.
Diagnostic Utility: The framework offers new diagnostics for long-context generalization by separating "depth" (context dependence) from "breadth" (frontier expansion), suggesting distinct optimization strategies for data selection and curriculum design.
Universality: The theory posits that the loss spectrum $E_X \propto r^{-7/6}$ is a universal feature of language model training, analogous to the universality of turbulence spectra.

The author explicitly notes that the mapping of optimizer steps to Langevin time and the harmonic combination of $N$ and $D$ are structural analogies motivated by empirical consistency rather than strict first-principles derivations, and that the sublinear capacity hypothesis ( $\lambda=3/4$ ) requires further validation on larger model suites.