📊 statistics

Universal One-third Time Scaling in Learning Peaked Distributions

This paper demonstrates that the computationally expensive, slow power-law convergence observed in training large language models is an intrinsic consequence of using softmax and cross-entropy to learn peaked distributions, which universally results in a loss time-scaling exponent of 1/3.

Original authors: Yizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff Gore

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff Gore

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot to finish your sentences. You give it millions of books, and it starts to learn the patterns of human language. But here's the catch: as the robot gets bigger and you feed it more data, it doesn't get perfect instantly. Instead, it improves in a very specific, slow way. It's like watching a car drive up a hill where the steeper it gets, the slower it goes, following a predictable curve. Scientists call this "neural scaling," and for a long time, they thought this slow curve happened because the data itself was messy or complicated, like a library where some books are rare and others are everywhere. They assumed the robot was just struggling to find the rare books. But what if the robot isn't struggling because of the library, but because of the way it thinks? This paper dives into that mystery, looking at the mathematical "gears" inside the robot's brain to see why learning slows down and if we can predict exactly how slow it will be.

The authors of this study, Yizhou Liu, Ziming Liu, Cengiz Pehlevan, and Jeff Gore, decided to stop guessing about the messy library and instead look at the robot's internal engine. They built a tiny, simplified version of a language model—a "toy model"—to see what happens when it tries to learn a very specific kind of pattern: a "peaked" distribution. Imagine a weather forecast that says there is a 99% chance of rain and a 1% chance of sun. That's a peaked distribution; the answer is almost certain, but the model has to be very precise to get it right.

When they trained their toy model to predict these sharp, certain outcomes, they discovered something surprising. The slow, power-law learning curve wasn't because the data was hard; it was because of two specific tools the model uses: softmax and cross-entropy. You can think of softmax as a "voting machine" that turns raw numbers into probabilities (making sure they add up to 100%), and cross-entropy as the "scorecard" that tells the model how wrong it is. The paper shows that when you combine these two tools to learn a very sharp, certain answer, the math forces the learning speed to slow down in a very specific way. No matter how you tweak the data or the model's size, the loss (the error) drops at a rate that follows a universal rule: it scales with time to the power of 1/3.

To put it in a playful analogy: Imagine you are trying to balance a pencil on its tip. At first, it's easy to make small adjustments. But as the pencil gets closer to perfectly upright (the "peaked" state), the tiniest wobble matters more and more. The paper suggests that the way the model calculates its mistakes makes it feel like it's walking through thick honey as it gets closer to the perfect answer. The "honey" isn't the data; it's the math of the voting machine and the scorecard. The authors found that in this "honey," the error doesn't drop quickly; it drops at a steady, predictable pace where if you double the time you spend training, the error only drops by a specific fraction, not by half. This fraction is exactly the cube root of the time, or 1/3.

The researchers didn't just stop at their tiny toy model. They wanted to know if this "honey" effect was real in the giant, real-world language models we use today, like the Pythia and Olmo models. They looked at the training data of these massive models and found that, indeed, the models were operating in this "low-temperature" (very sharp, certain) regime. When they plotted the error rates of these giant models against the time they spent training, the data lined up perfectly with their prediction. The error was dropping with a 1/3 power-law exponent. This suggests that the slow learning we see in big AI isn't a bug or a sign of messy data; it's a fundamental feature of how these models are built.

The paper also explored what happens if you try to speed things up. They found that if you make the learning rate (how big of a step the model takes) too big, the model gets confused and can't follow the path through the honey. But if you keep the steps small and steady, the model follows the 1/3 rule perfectly. Interestingly, they also found that even if the model isn't perfectly aligned with the teacher at the start, it eventually falls into this rhythm. The only time this rule breaks is if the data is very "flat" or uncertain (like a weather forecast with 50% rain and 50% sun), in which case the learning is fast and exponential, not slow and power-law.

So, what does this mean for the future? The authors suggest that if we want to train AI faster, we might need to rethink the "voting machine" and "scorecard" we use. Maybe we need new tools that don't get stuck in the honey when the answer is very certain. They also point out that this discovery explains why previous experiments saw a 1/3 exponent (or close to it, like 0.28 or 0.30) without realizing why. It wasn't a coincidence; it was the math of the model itself.

In short, this paper reveals that the slow, steady improvement of large language models is a built-in feature of their design, not a flaw in the data. It's like a law of physics for AI learning: when you try to learn a very certain answer using these specific tools, you are destined to move at a 1/3 speed. The authors are careful to say this is based on their simulations and analysis of existing models, suggesting a new direction for making AI training more efficient, but they don't claim to have solved the problem of training speed entirely. Instead, they've handed us a map showing exactly where the speed bumps are and why they exist.

Technical Summary: Universal One-third Time Scaling in Learning Peaked Distributions

Problem Statement
Training large language models (LLMs) is computationally expensive, characterized by a slow power-law convergence of the loss function. While empirical "neural scaling laws" have successfully predicted performance based on model size and dataset size, the underlying mechanisms remain largely empirical and debated. Existing theories often attribute power-law loss scaling to power-law structures inherent in the data (e.g., feature frequencies or manifold dimensions). However, key questions remain unanswered: What specific mechanism drives the power-law decay of loss over training time, and what determines the specific exponent? This paper investigates whether the origin of these scaling laws lies in the data or in the architectural components of the model itself.

Methodology
The authors employ a combination of theoretical analysis, toy model simulations, and empirical evaluation on large language models to isolate the mechanisms driving loss scaling.

Toy Model Construction: The authors identify a minimal setup mimicking the language model (LM) head: a one-layer network with a softmax non-linearity and a cross-entropy (or KL-divergence) loss function. They utilize a teacher-student framework where a student network learns to match the output distribution of a fixed teacher network. The teacher's output is controlled by an "inverse temperature" parameter ( $\beta^*$ ), which dictates how "peaked" (low-entropy) the probability distribution is.
Theoretical Analysis: The authors analyze the continuous-time gradient flow dynamics of the student weights. They introduce an "aligned student ansatz," hypothesizing that the student's weight matrix remains proportional to the teacher's during training, with the primary dynamic variable being the norm (or inverse temperature $\beta$ $β$ ) of the weights.
- They perform a low-temperature expansion (large $\beta$ ) of the free energy and internal energy derived from the softmax and cross-entropy definitions.
- By integrating the resulting differential equations, they derive the scaling relationship between loss and training time.
Empirical Validation:
- Toy Model Experiments: They train the toy model across a wide range of inverse temperatures ( $\beta^*$ ) and learning rates using optimizers like Adam and SGD. They verify the theoretical predictions regarding loss decay and weight alignment.
- LLM Evaluation: They evaluate checkpoints of Pythia and OLMo models at various training stages on the FineWeb dataset. They measure the test loss and the standard deviation of the logits (a proxy for inverse temperature) to determine if real LLMs operate in the low-temperature regime and exhibit the predicted scaling.

Key Contributions and Results

Intrinsic Origin of Power Laws: The paper demonstrates that power-law training behaviors can arise intrinsically from the use of softmax and cross-entropy when learning peaked distributions (low-entropy targets), independent of power-law structures in the data. This challenges the prevailing view that data structure is the sole driver of scaling laws.
Universal 1/3 Exponent: Through Taylor expansion and time integration of the gradient flow dynamics in the low-temperature regime, the authors derive a universal scaling exponent of 1/3. Specifically, the loss $L$ $L$ scales with dynamic training time $\tau$ $τ$ as $L \sim \tau^{-1/3}$ $L \sim τ^{- 1/3}$ .
- The derivation relies on the fact that in the low-temperature limit, the free energy and internal energy converge smoothly to the minimum energy, leading to power-law vanishing gradients.
- This result is described as "universal" in the statistical physics sense: the exponent depends on broad system properties (non-linearities and peakedness) rather than microscopic details (specific data distributions).
Validation in Toy Models: Experiments confirm that when the target distribution is sufficiently peaked (low temperature), the loss follows a $1/3$ power law. The "aligned student ansatz" accurately describes the training dynamics under optimal learning rates, where the student weight norm grows algebraically with an exponent close to $1/3$ .
Validation in LLMs: Analysis of Pythia and OLMo models reveals that:
- Next-token distributions in LLMs are sufficiently peaked to place them in the low-temperature regime (estimated $\beta^* \approx 6.87$ , exceeding the critical threshold $c_0 \approx 4.55$ ).
- The training loss of these models, when fitted against dynamic time $\tau$ , exhibits a power-law decay with an exponent very close to $1/3$ (e.g., fitted values around 0.34–0.38).
- The logit standard deviation grows with an exponent near $1/3$ before saturating, indicating the system enters the low-temperature regime and that later loss decay is driven by parameter rotation rather than norm growth.

Significance and Claims
The authors claim this work provides the first theoretical study identifying softmax and cross-entropy as the key architectural drivers of power-law training behaviors in LLMs.

Mechanistic Explanation: The paper offers a mechanistic explanation for observed neural scaling laws, suggesting that the $1/3$ time scaling is a fundamental consequence of optimizing peaked distributions with these specific non-linearities.
Universality: The findings suggest that the $1/3$ exponent is robust across different data structures, model depths, and initialization scales, provided the system operates in the intermediate low-temperature regime.
Implications for Training: The authors suggest that understanding this mechanism could lead to improved training efficiency. For instance, they note that large weight decays might hinder the necessary logit growth for peaked distributions, and that future optimizers might benefit from being more effective at rotating parameters in the low-temperature regime. They also propose that architectures avoiding full-vocabulary distributions (e.g., via token hierarchies) could fundamentally alter these scaling laws.

The paper concludes that while previous works attributed scaling to data properties, the specific $1/3$ time scaling observed in LLMs is largely a result of the optimization landscape created by the model's non-linearities when learning high-confidence predictions.

Technical Summary: Universal One-third Time Scaling in Learning Peaked Distributions

More like this