🤖 machine learning

Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients

This position paper argues that while the exponents of neural scaling laws are fixed by universal mechanisms like Softmax nonlinearity and representational superposition, the coefficients—which dictate practical performance and optimal model configurations—are sensitive to specific data and architectural details, making their understanding the key to near-term AI improvements.

Original authors: Yizhou Liu, Jeff Gore

Published 2026-06-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yizhou Liu, Jeff Gore

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Recipe" vs. The "Ingredients"

Imagine you are trying to bake the perfect loaf of bread (a Large Language Model). For a long time, scientists have noticed a pattern: if you double the amount of flour, water, or oven time, the bread gets better in a predictable way. This is called a "scaling law."

This paper argues that the rules of how the bread improves (the math behind the recipe) are actually fixed and unchangeable, like the laws of physics. However, the quality of the ingredients (the data and specific design choices) changes how good the bread tastes, even if the rules stay the same.

The authors call this "Neural Scaling Universality." It means that no matter how you tweak the model, the "speed" at which it learns follows three fixed rules. The only thing that changes is the "starting point" or the "efficiency" of the process.

The Three Fixed Rules (The Exponents)

The paper says that as you train a model, the errors (mistakes) it makes drop down following three specific patterns. Think of these as three different "speed limits" for learning:

1. The "Softmax" Speed Limit (Time)

The Analogy: Imagine you are trying to find a needle in a haystack, but the haystack is made of magnets that get stronger the closer you get. At first, you move slowly. But as you get closer to the needle, the magnets pull you in faster and faster.
The Science: The paper says that because of a specific math function called "Softmax" (used to make predictions), the model learns in a specific way. The errors drop at a fixed rate related to time (specifically, the cube root of time). This happens because the math gets very "non-linear" (curvy) when the model is confident.
The Takeaway: You can't change this speed limit. It's built into the math of how the model thinks.

2. The "Crowded Room" Limit (Width)

The Analogy: Imagine a small room (the model's memory) where you are trying to fit a million different people (concepts/words). If the room is too small, people have to stand on top of each other or share space. This causes "geometric interference"—people bumping into each other, making it hard to hear clearly.
The Science: This is called "superposition." The model tries to pack more features into fewer dimensions. The errors drop as you make the room wider (increase the model's width), but the improvement follows a strict rule: double the width, halve the error.
The Takeaway: The limit here is how much information you can squeeze into the model's "brain" without it getting messy.

3. The "Teamwork" Limit (Depth)

The Analogy: Imagine a relay race with many runners (layers in the model). If every runner makes a tiny mistake, the team can still win if they average out their errors. But if the runners are all doing the exact same thing, they don't help each other much.
The Science: The paper suggests that Transformer layers work like a team averaging out their mistakes. The more layers you add, the better the team gets at correcting errors, but again, it follows a fixed rule: double the layers, halve the error.
The Takeaway: Adding more layers helps, but the benefit follows a predictable, fixed pattern.

The Real Problem: The Coefficients (The "Ingredients")

If the rules (exponents) are fixed, why do some models perform better than others? The answer lies in the coefficients.

The Analogy: Think of the fixed rules as the laws of physics for baking. You can't change the fact that heat cooks dough. But, the coefficient is the quality of your flour, the temperature of your oven, and the skill of your baker.
What the Paper Says: The "coefficients" are the numbers in the math that tell you how much error you have left. These numbers depend entirely on:
- The Data: How diverse and high-quality the text is.
- The Architecture: The specific design choices (like how the model handles attention or normalization).
Why It Matters: Since the "speed limits" (exponents) are fixed, the only way to make a better model right now is to improve these coefficients. If you can tweak your design or data to lower the coefficient, you get a better model without breaking the laws of physics.

Practical Results: Finding the "Sweet Spot"

The paper uses these rules to figure out the perfect shape for a model.

The Shape: How wide should the model be compared to how deep?
- The paper calculates that there is a "Goldilocks" ratio. If you have a fixed budget of computing power, you shouldn't just make the model huge in one direction. You need a specific balance (roughly 64 times wider than it is deep, according to their data on Chinchilla models).
- Good News: The paper says the "landscape" is flat here. This means you don't need to be perfect. Being close to the right ratio is almost as good as being exact.
The Data vs. Steps:
- Many people worry we are "running out of data." This paper argues that pre-training is actually step-limited, not data-limited.
- The Analogy: It's not that you ran out of books to read; it's that you stopped reading before you finished the sentence. You can get better results by taking more "steps" (optimization steps) with the data you have, rather than just hoarding more unique data.
The Compute Frontier:
- The paper confirms that if you balance your model size and data size perfectly, the error drops at a rate of one-sixth of the total computing power used. This is the "compute-optimal frontier."

Summary

The Rules are Fixed: The way AI models learn follows three unchangeable mathematical patterns (Time, Width, Depth) caused by the nature of the math (Softmax) and how layers work together.
The Variables are Flexible: The quality of the learning (the coefficients) depends on your data and design choices.
The Goal: To build better AI, we shouldn't just look for new "laws of physics." Instead, we should focus on tweaking our "ingredients" (data and architecture) to lower the error coefficients, finding the perfect balance between model size and data to get the most bang for our buck.

The paper concludes that we are currently in a specific "universality class" (a specific set of rules). To get even better AI in the future, we might need to find a way to break out of these current rules, but for now, mastering the coefficients is the key to immediate improvement.

Technical Summary: Neural Scaling Universality

Problem Statement

Neural scaling laws describe the decay of pre-training loss as power laws with respect to training time, model size, and compute. While these laws have been instrumental in the success of Large Language Models (LLMs), their mechanistic origins remain largely empirical. The prevailing view attributes these power laws to the power-law distributions inherent in the training data (e.g., feature frequencies), suggesting that loss exponents are sensitive to specific data structures. This leaves open questions regarding the generalizability of scaling laws at larger scales and the potential for further speedups. The paper addresses the need to determine whether scaling exponents are fixed by generic architectural mechanisms or are contingent on data details, and whether the coefficients of these laws hold the key to near-term performance improvements.

Methodology

The paper synthesizes and extends a trilogy of recent works [15–17] to propose a hypothesis of Neural Scaling Universality. The methodology involves:

Theoretical Decomposition: The authors decompose the pre-training loss ( $L$ ) into three distinct asymptotic terms based on generic mechanisms:
- Time Scaling: Driven by the strong nonlinearity of the Softmax function when learning peaked (low-entropy) distributions, leading to a loss decay of $L \sim \tau^{-1/3}$ , where $\tau$ is dynamic time.
- Width Scaling: Driven by "superposition," where models encode more features than available dimensions, causing geometric interference. This results in a residual loss $L \sim m^{-1}$ , where $m$ is the embedding dimension (width).
- Depth Scaling: Driven by the ensemble averaging of errors across Transformer layers, which update hidden states incrementally rather than hierarchically. This yields $L \sim \ell^{-1}$ , where $\ell$ is the depth.
Unified Loss Formula: These mechanisms are combined into an additive asymptotic loss formula:
$L = \frac{c_\tau}{\tau^{1/3}} + \frac{c_m}{m} + \frac{c_\ell}{\ell} + L_0$
where $c_\tau, c_m, c_\ell$ are coefficients absorbing data and architecture specifics, and $L_0$ is irreducible loss.
Empirical Validation: The authors fit this fixed-exponent formula to empirical data from multiple model families, including Pythia, Chinchilla, GPT-3, and Farseer.
- They treat exponents as fixed (e.g., $1/3$ for time, $1$ for width/depth) and vary only the coefficients.
- They analyze the relationship between dataset size ( $D$ ) and dynamic time ( $\tau$ ), arguing that in online training regimes, $D \propto \tau$ , allowing the transfer of time scaling to dataset size scaling.
- They derive optimal model shapes and compute-optimal frontiers by minimizing the loss function with respect to model parameters ( $N$ ) and compute ( $C$ ).

Key Contributions

1. The Hypothesis of Neural Scaling Universality

The paper posits that neural scaling laws belong to a specific "universality class" defined by fixed exponents determined by generic mechanisms (Softmax nonlinearity, superposition, and layer averaging). Within this class, variations in data structures or architectural details primarily alter the coefficients ( $c_\tau, c_m, c_\ell$ ) rather than the exponents.

2. Mechanistic Origins of Fixed Exponents

The paper provides specific mechanistic explanations for the observed exponents:

$1/3$ Time Scaling: Arises from the nonlinear dynamics of Softmax in reaching peaked distributions, analogous to low-temperature expansion.
Inverse Width Scaling ($1$): Arises from the geometric interference of superposed features.
Inverse Depth Scaling ($1$): Arises from the averaging of errors across layers with similar functional updates.

3. Derivation of Optimal Architectures and Frontiers

By treating the coefficients as the primary variables, the paper derives:

Optimal Model Shape: The optimal ratio of width to depth ( $m/\ell$ ) is determined by the ratio of coefficients $c_m / c_\ell$ . At this optimum, the loss contributions from width and depth follow a $2:1$ ratio.
Model Size Scaling: At the optimal shape, the loss decays as $N^{-1/3}$ (where $N$ is parameter count).
Compute-Optimal Frontier: By minimizing loss at fixed compute ( $C \approx 6ND$ ), the paper derives an optimal token-to-parameter ratio ( $D/N$ ) and a compute-optimal loss scaling of $C^{-1/6}$ .

4. Shift from Data-Limited to Step-Limited Pre-training

The authors argue that pre-training is fundamentally step-limited (constrained by optimization dynamics) rather than data-limited. Since dataset size scaling is a secondary consequence of the time scaling ( $D \propto \tau$ ), further loss reduction can be achieved by taking more optimization steps (potentially reusing data) rather than solely increasing the volume of unique data, provided data quality and diversity are sufficient.

Results

The paper validates the proposed universality class through extensive fitting of empirical data:

Pythia Models: Loss decay across training time, width, and depth aligns well with the fixed-exponent formula ( $1/3, 1, 1$ ).
Chinchilla Models: The data supports the derived $1/3$ dataset size scaling and $1/3$ model size scaling. The optimal aspect ratio predicted by the fitted coefficients ( $m/\ell \approx 64$ ) closely matches the actual Chinchilla model shapes. The compute-optimal points follow the predicted $N \propto C^{1/2}$ and $L \propto C^{-1/6}$ scaling.
Farseer Models: This family, which strictly follows $m \propto \ell$ , falls precisely on the predicted optimal line. Despite the degeneracy in identifying individual width/depth coefficients, the model size and dataset size components of the loss still agree well with the $1/3$ scaling.
Coefficient Variability: Analysis across different model families (Pythia, GPT-3, Farseer, Chinchilla) shows that while exponents remain fixed, the coefficients ( $c_\tau, c_m, c_\ell$ ) and derived optimal ratios vary, confirming their sensitivity to data and architectural choices.

Significance and Claims

The paper claims that understanding the coefficients is the key to near-term performance improvements in LLMs, rather than searching for new exponents.

Design Prescription: The coefficients provide a principled basis for architecture design (e.g., determining the optimal $m/\ell$ ratio) and compute allocation (determining the optimal $D/N$ ratio).
Universality vs. Data Dependence: The paper challenges the mainstream view that loss exponents depend on data power laws. Instead, it argues that exponents are robust and fixed by high-level properties (Softmax usage, low entropy targets), while data structures mainly influence the coefficients.
Pathways to Improvement: By closely examining the coefficients, researchers can identify pathways to better universality classes or improvements within the current class. The paper suggests that modifying architectures (activation functions, attention mechanisms) or data composition can shift coefficients, thereby improving the loss floor and compute efficiency without changing the fundamental scaling exponents.

The authors conclude that while the hypothesis of fixed exponents requires further verification, the evidence from [15–17] and the current empirical fits make it plausible. Consequently, the focus of the community should shift toward characterizing and optimizing the coefficients that define the current universality class.