On the Optimizer Dependence of Neural Scaling Laws — Plain-Language Explanation

The Big Idea: It's Not Just About Size, It's About How You Learn

Imagine you are trying to learn a massive amount of information (like reading every book in a library). In the world of Artificial Intelligence (AI), we have a rule called a Scaling Law. This rule says: "If you make your AI brain bigger (add more neurons), it gets smarter, but the amount it improves follows a specific, predictable curve."

For a long time, scientists thought this curve was fixed. They believed that if you doubled the size of the AI, you would get a specific, unchangeable amount of improvement, no matter what tool you used to teach it.

This paper says: "Actually, the tool you use to teach the AI changes the curve."

The "tool" is called an optimizer. Think of the optimizer as the teacher or the study method. The paper argues that some teachers are so much better at organizing the learning process that they don't just make the AI learn faster; they actually change the fundamental math of how much the AI improves as it gets bigger.

The Analogy: The Library and the Librarian

To understand the paper's findings, let's use an analogy of a Library and a Librarian.

1. The Library (The Data)

Imagine the data the AI needs to learn is a library.

The "Steep" Library: Most books are about one popular topic (like "How to bake bread"), and very few books are about obscure topics (like "18th-century moss farming"). In the real world, this is like language: a few words are used constantly, and millions are used rarely. This is called a steep spectrum.
The "Flat" Library: Every topic has exactly the same number of books. This is rare in nature but easy to imagine.

2. The Student (The AI Model)

The AI is a student trying to read these books to pass a test. The student has a limited amount of time (computing power) and a limited memory (model size).

3. The Teachers (The Optimizers)

This is where the paper gets interesting. The authors tested different "teachers" (optimizers) to see how they help the student learn.

Teacher A (Standard Gradient Descent / GD): This teacher is like a student who reads the library in order. They read the popular "bread" books first because they are easy to find. By the time they get to the "moss farming" books, they run out of time.
- Result: They get good at bread, but terrible at moss. As the library gets bigger, they get stuck on the popular stuff and ignore the rest. Their improvement curve flattens out quickly.
Teacher B (Preconditioned Optimizers like Muon/Full NG): This teacher is a genius organizer. They realize that even though the "moss" books are rare, they are still important. They use a special map (called a preconditioner) to make the "moss" books just as easy to reach as the "bread" books. They force the student to learn everything evenly.
- Result: The student learns a little bit of everything. Because they aren't wasting time just on the popular stuff, they get much smarter as the library grows.

The Key Discovery: The "Magic Multiplier"

The paper ran computer experiments (using a simplified math model called "random-feature regression") to measure exactly how much smarter the students got.

They found that when the library has a steep spectrum (like real human language, where a few things are very common and many are rare):

Teacher A (Standard) hits a wall. The student stops improving much after a certain size.
Teacher B (Advanced) keeps climbing. The student gets significantly smarter with every step up in size.

The "Magic" Number:
The paper measured a number called $\alpha$ (alpha). This number represents the "rate of improvement."

For the standard teacher, $\alpha$ was low (about 0.12).
For the advanced teacher, $\alpha$ was much higher (about 0.31).

Why does this matter?
Because these numbers are exponents, a small difference creates a huge gap over time.

If you double the size of the AI with the standard teacher, you get a small boost.
If you double the size with the advanced teacher, you get a 2.6 times larger boost.
If you keep doubling the size, the advanced teacher pulls further and further ahead. It's like compound interest: the advantage grows with every step.

The Catch: It Depends on the Library

The paper also found a crucial condition: This advantage only happens if the library is "steep."

If the library is steep (like real language): The advanced teacher is a game-changer. They fix the imbalance and let the AI learn efficiently.
If the library is flat (everything is equal): The standard teacher is already doing a good job because there's no imbalance to fix. The advanced teacher doesn't offer much extra help here.

What About Real AI? (The "Open Question")

The authors are careful to say their results come from a simplified math model, not a real, giant AI like the ones used by Google or OpenAI today.

They acknowledge a conflict in the real world:

Some recent studies say advanced teachers (like Muon) are great at small scales but their advantage fades away as the AI gets huge.
This paper suggests that in their simplified model, the advantage should keep growing.

The Conclusion:
The paper proposes that the "fading advantage" seen in real life might happen because real AI learns features on its own (changing the library structure), whereas their model assumes the library stays the same.

Summary for the General Audience

The Myth: "Bigger AI always gets smarter at a fixed rate."
The Reality: The rate at which AI gets smarter depends heavily on the mathematical tool (optimizer) used to train it.
The Discovery: Advanced tools can act like a "spectral equalizer." They stop the AI from ignoring rare but important information, allowing it to learn much more efficiently as it grows.
The Condition: This super-power works best when the data is "unbalanced" (like human language). If the data is perfectly balanced, the fancy tools don't help much.
The Future: We need to test this on real, giant AI models to see if the "magic multiplier" holds up or if it fades away as the models get massive.

In short: Choosing the right teacher doesn't just make the student learn faster; it changes the ceiling of how smart the student can become.

Technical Summary: On the Optimizer Dependence of Neural Scaling Laws

Problem Statement
Neural scaling laws posit that test loss $L$ decreases as a power law with model size $N$ ( $L(N) \propto N^{-\alpha}$ ), where the exponent $\alpha$ is traditionally treated as a fixed constant determined by architecture and data distribution. This assumption underpins compute allocation strategies for frontier models. However, recent empirical results regarding second-order optimizers (e.g., Muon, SOAP, Shampoo) present conflicting evidence: some suggest significant compute speedups over AdamW, while others indicate these gains attenuate as model size increases. This paper investigates a fundamental question: do advanced optimizers merely improve the constant factor of the scaling law, or do they fundamentally alter the scaling exponent $\alpha$ itself? If $\alpha$ shifts, the implications for scaling forecasts are profound, as exponent changes compound with every model-size doubling, whereas constant-factor improvements do not.

Methodology
The authors investigate this question within the random-feature regression framework (Maloney et al., 2022; Bordelon et al., 2024). This theoretical setting isolates the interaction between data spectral structure, optimizer preconditioning, and model capacity by fixing feature weights and training only the top-layer linear weights.

Experimental Setup: The study simulates regression tasks with inputs drawn from a Gaussian distribution with power-law eigenvalues $\lambda_i \propto i^{-(1+s)}$ , where $s$ is the spectral exponent. Natural language data is characterized by $s \approx 0.8\text{--}1.2$ .
Optimizer Variants: Five optimizers are implemented as gradient preconditioners:
1. GD: Baseline (Identity preconditioner).
2. Diagonal: Proxy for AdamW ( $P = \text{diag}(F^\top F)^{-1/2}$ ).
3. Full NG: Proxy for Shampoo/K-FAC ( $P = (F^\top F)^{-1}$ ).
4. Sign-GD: A negative control using sign updates.
5. Matrix-Sign: Proxy for Muon ( $P = (F^\top F)^{-1/2}$ ), which equalizes singular values.
Protocol: Experiments sweep six spectral conditions ( $s \in [0.25, 2.0]$ ) and eight model sizes ( $N \in [25, 5000]$ ). Scaling exponents $\alpha$ are fitted via Ordinary Least Squares (OLS) on log-log plots of loss versus model size for $N \ge 200$ .

Key Contributions

Systematic Measurement: The paper provides the first systematic measurement of $\alpha$ as a joint function of optimizer type and data spectral decay across five preconditioner variants.
Mechanistic Explanation: It offers a spectral heuristic (Proposition 1) explaining that preconditioned optimizers equalize per-mode learning rates. While Gradient Descent (GD) "overspends" on high-variance modes and "underspends" on low-variance modes, preconditioners rescue underserved modes, increasing the effective capacity utilized at a given $N$ .
Spectral Diagnostic: The authors propose a diagnostic rule: the benefit of preconditioning depends on the spectral steepness $s$ . The advantage is negligible for flat spectra ( $s < 0.3$ ) but grows significantly as $s$ increases.
Falsifiable Prediction: The paper predicts that training on artificially flattened-spectrum data (e.g., via whitening) should cause the advantage of advanced optimizers like Muon to shrink or vanish.

Results

Optimizer Dependence of $\alpha$ : Preconditioned optimizers consistently yield steeper scaling (larger $\alpha$ ) than GD. The ordering is generally $\alpha_{\text{Full NG}} \approx \alpha_{\text{Matrix-Sign}} > \alpha_{\text{Diagonal}} > \alpha_{\text{Sign-GD}} \approx \alpha_{\text{GD}}$ .
Spectral Dependence: The gap $\Delta \alpha = \alpha_{\text{precond}} - \alpha_{\text{GD}}$ $Δ α = α_{precond} - α_{GD}$ increases with the spectral exponent $s$ $s$ .
- At $s \approx 1.0$ (characteristic of natural language), Full Natural Gradient achieves $\alpha \approx 0.31$ versus $\alpha \approx 0.12$ for GD, representing a $2.6\times$ larger fitted exponent.
- At $s \approx 0.5$ , the curves are nearly parallel, and the advantage diminishes.
- At $s \approx 2.0$ , GD's exponent collapses to near zero ( $\alpha \approx 0.01$ ), while preconditioned optimizers degrade gracefully ( $\alpha \approx 0.20$ ).
Compute Multipliers: Translating the $\alpha$ -shift to an effective compute multiplier, the random-feature model predicts a substantial parameter-equivalent advantage for strong preconditioners at $s \approx 1.0$ . This aligns directionally with recent empirical reports of $\sim 1.4\times$ compute speedups for Muon over AdamW at moderate scales (Chen et al., 2025).

Significance and Claims
The paper argues that scaling-law forecasts must account for optimizer choice. If preconditioned optimizers change the exponent $\alpha$ , forecasts calibrated under AdamW may significantly underestimate the returns to scaling for models trained with advanced optimizers.

However, the authors maintain a modest stance regarding the transfer of these findings to large-scale LLMs:

Limitations: The random-feature model operates in a "lazy" regime (fixed features) and omits depth, attention, and tokenization.
Scale Decay: The paper acknowledges conflicting evidence (Wen et al., 2025) suggesting that the Muon/SOAP advantage may attenuate at very large scales (e.g., 1.2B parameters). The authors propose that this attenuation may be due to finite- $N$ effects, feature learning naturally mitigating spectral imbalance, or changes in the effective data spectrum at scale.
Conclusion: The primary contribution is establishing that the scaling exponent is not a fixed constant of the architecture/data pair but is systematically dependent on the optimizer's interaction with the data spectrum. The paper calls for real-model validation to determine if this exponent shift persists or attenuates in the "rich" regime of feature learning.