💻 computer science

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

This paper demonstrates that the choice of optimizer fundamentally alters the spectral scaling laws of Transformer models, revealing that optimizers like Muon can achieve significantly better utilization of spectral capacity in hard-to-learn regimes compared to AdamW, thereby establishing optimization as a critical, independent axis of representation scaling that rivals architectural design.

Original authors: Nandan Kumar Jha, Brandon Reagen

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nandan Kumar Jha, Brandon Reagen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive library (a Large Language Model) to store and retrieve human knowledge. You have a blueprint for the library's shelves (the Architecture), and you have a librarian who organizes the books (the Optimizer).

For years, researchers believed that if you just made the library bigger (adding more shelves/parameters) and fed it more books (data), the librarian's job would automatically get better in a predictable way. They assumed the type of librarian didn't matter much, as long as they were doing the job.

This paper argues that who the librarian is matters just as much as how big the library is. In fact, the librarian determines how the new shelves are actually used.

Here is the breakdown of the paper's findings using simple analogies:

1. The Two Types of "Library Capacity"

The researchers looked at how the model uses its internal "brain space" (specifically the Feed-Forward Networks, or FFNs). They measured this in two ways:

Soft Rank (The "Spread"): Imagine a library where books are scattered evenly across every single shelf. The space is used broadly, but maybe not deeply. This is "diffuse" capacity.
Hard Rank (The "Focus"): Imagine a library where the most important, rare, and complex books are packed tightly onto specific, high-quality shelves, while other shelves are empty. This is "dominant" capacity. It's about having a few very strong ways to represent complex ideas.

2. The Problem with the "Standard Librarian" (AdamW)

The most common librarian, AdamW, is good at spreading books out. When you give it a bigger library (more width), it fills up the shelves broadly (Soft Rank goes up).

However, when it comes to the rare and difficult books (the "Tail" tokens—like obscure facts or complex concepts), AdamW struggles to organize them into strong, focused shelves.

The Result: Even if you double the size of the library, AdamW doesn't double its ability to handle the hard stuff. It just spreads the confusion out over more space. The paper calls this "weak hard-rank scaling."

3. The "Super Librarian" (Muon)

The researchers tested a different librarian called Muon.

The Result: When Muon gets a bigger library, it doesn't just spread things out. It actively builds strong, focused shelves for the rare and difficult books.
The Magic: Muon turns added space into usable power much faster. If AdamW's ability to handle hard concepts grows slowly (like a slow walk), Muon's ability grows in a straight line (like a sprint). In the paper's terms, Muon achieves linear scaling where AdamW is stuck in a "weak" growth pattern.

4. The "Perplexity Trap" (Why we missed this before)

You might ask: "If Muon is so much better at organizing, why didn't we notice earlier? Doesn't it just get lower error scores?"

The paper reveals a trap: You can have the same final score with totally different internal structures.

If you train the "Standard Librarian" (AdamW) for a very long time, it can eventually match the final test score (perplexity) of the "Super Librarian" (Muon).
However: Inside the brain, they are completely different. The Standard Librarian has a messy, diffuse structure. The Super Librarian has a sharp, organized structure.
The Takeaway: Just because two models get the same test score doesn't mean they "think" the same way. The Super Librarian is actually building a better internal map of the world, even if the final grade looks similar.

5. The "Rare Book" Problem

Language is like a Zipf's Law distribution: a few words are used constantly (like "the" or "is"), but most words are rare.

AdamW is okay with the common words but gets lost with the rare ones.
Muon shines specifically with the rare, long-tail knowledge. It scales its ability to handle these rare tokens almost perfectly as the model gets bigger.

6. The Librarian vs. The Blueprint (Architecture)

The researchers asked: "Is the librarian more important than the library design?"
They tested changing the library design (like changing the number of attention heads or removing positional signals).

The Finding: Changing the Librarian (the optimizer) had a bigger impact on how the model learned than changing the Blueprint (the architecture).
In fact, a "Super Librarian" could make a standard library design work better than a "Standard Librarian" could make a fancy new design work. The librarian and the blueprint are a team; you can't just pick a blueprint and assume any librarian will do.

Summary

The paper concludes that Optimization is a first-class citizen in AI design.

Old View: "Make the model bigger, and it will get smarter."
New View: "Make the model bigger, but choose the right optimizer to ensure that extra size actually turns into useful, focused intelligence, especially for the hard, rare things."

If you want a model that truly understands the "long tail" of human knowledge, you can't just rely on the standard tools. You need an optimizer that knows how to build strong, focused shelves for the difficult books, not just fill up empty space.

Technical Summary: Optimizer-Induced Spectral Scaling Laws

Problem Statement

Classical scaling laws for large language models (LLMs) have established predictable power-law relationships between validation loss and resources such as model size, data volume, and compute. However, these frameworks typically treat the optimizer as a fixed training detail, assuming it only influences convergence speed or acts as a multiplicative efficiency factor. This paper challenges that assumption, positing that the optimizer is a fundamental axis of representation scaling. Specifically, the authors investigate how effectively different optimizers convert increased Feed-Forward Network (FFN) width into utilized spectral capacity—the effective dimensionality of the latent space. They ask whether matched validation loss implies matched internal representation geometry, or if different optimizers realize the same architectural capacity through fundamentally different spectral structures.

Methodology

The study employs a spectral analysis framework centered on the eigenspectra of FFN representations within Transformer architectures.

Spectral Metrics: The authors utilize the Rényi effective-rank family to quantify spectral spread. They anchor their analysis on two primary metrics:
- Soft Rank ( $R_1$ ): Corresponds to $\alpha=1$ (Shannon entropy), measuring entropy-weighted spectral spread and capturing diffuse capacity.
- Hard Rank ( $R_2$ ): Corresponds to $\alpha=2$ (participation ratio), measuring concentration-sensitive effective dimensionality and capturing dominant-mode capacity.
- Hard-Soft Asymmetry ( $\Delta_{1,2}$ ): Defined as $\log R_1 - \log R_2$ , this metric quantifies the gap between diffuse and dominant capacity.
Experimental Setup: Experiments were conducted on GPT-style decoder-only Transformers (160M and 350M parameters) trained on the FineWeb-Edu dataset. The FFN hidden dimension ( $D$ ) was varied to establish scaling laws ( $R(D) \propto D^\beta$ ).
Optimizer Comparison: The study compares standard AdamW against matrix-aware optimizers (Muon, NorMuon) and rank-constrained variants (Dion with varying rank fractions $r$ ).
Stratification: To address the Zipfian distribution of language data, FFN representations were stratified by token frequency into HEAD (frequent), MID, and TAIL (rare) regimes.
Architectural Controls: The authors compared optimizer-induced shifts against architectural interventions, specifically increasing per-head attention rank (by reducing head count) and removing Rotary Positional Embeddings (RoPE).

Key Results

1. Optimizer-Dependent Spectral Scaling Laws

The same Transformer architecture exhibits markedly different spectral scaling laws depending on the optimizer.

Hard-Rank Scaling: AdamW exhibits weak hard-rank scaling ( $\beta_{hard} \approx 0.29$ to $0.44$), particularly for rare (TAIL) tokens. In contrast, Muon achieves near-linear scaling ( $\beta_{hard} \approx 1.02$ ) in the same regimes. This represents a 2.3 $\times$ increase in the scaling exponent.
Hard-Soft Asymmetry: AdamW displays a large positive asymmetry ( $\Delta_{1,2} \approx 0.37$ ), indicating that added FFN width expands diffuse spectral mass rather than dominant representational directions. Muon and NorMuon reduce this asymmetry to near zero ( $\Delta_{1,2} \approx 0.14$ ), meaning added width is efficiently converted into dominant-mode capacity.

2. Matched Loss $\neq$ Matched Geometry

The paper demonstrates that validation loss is an insufficient proxy for representation structure.

Extended Training: Training AdamW for extended steps (12K vs. 6K) allowed it to match the perplexity of a Dion (1/16) variant. However, the spectral geometry remained distinct: AdamW's hard-rank scaling nearly vanished ( $\beta_{hard} \to 0.03$ ), while Dion maintained robust power-law scaling.
Learning Rate: Sweeping learning rates confirmed that the AdamW-Muon spectral gap is not a hyperparameter artifact; Muon's lowest hard-rank exponent consistently exceeded AdamW's highest across the sweep.

3. Frequency and Rank Dependencies

Token Frequency: The optimizer effect is most pronounced in MID and TAIL token regimes. AdamW struggles to scale hard capacity for rare tokens, whereas Muon scales linearly.
Update Rank: Using Dion, the authors isolated the effect of update rank. Reducing the rank fraction ( $r$ ) from $1/2$ to $1/16$ caused hard-rank scaling to drop from $\beta \approx 0.88$ to $\beta \approx 0.40$ (approaching AdamW levels), while soft-rank scaling degraded more gradually. This indicates that update rank acts as a control knob for converting width into dominant-mode capacity.

4. Dominance Over Architectural Interventions

Optimizer-induced spectral shifts often exceed or reshape the effects of architectural changes:

Attention Rank: Increasing per-head attention rank (by reducing head count) produced smaller spectral scaling shifts than switching optimizers in 28 of 30 frequency/metric regimes.
Positional Encoding: Removing RoPE induced optimizer-dependent redistribution. For Muon, NoPE increased HEAD hard-rank scaling but decreased MID/TAIL scaling; for NorMuon, it decreased scaling across all regimes. This shows that architectural signals are expressed through optimizer geometry.

Significance and Claims

The paper argues that optimization should be treated as a first-class axis of representation scaling, alongside model size, data, and compute.

Architecture-Optimizer Co-Design: The realized spectral capacity is a property of the architecture-optimizer pair, not the architecture alone. Architectural interventions (like changing attention rank or positional encodings) do not induce fixed, optimizer-independent shifts; their effects are mediated by the optimizer's geometry.
Representation Structure: The choice of optimizer determines how model capacity is structured across eigenmodes and allocated across token-frequency regimes. It is not merely an efficiency knob for convergence speed.
Limitations: The authors note that these findings are established within a controlled empirical regime (GPT-style models up to 350M parameters). While the 350M results replicate the 160M patterns, further validation at frontier scales (1B+) is needed. Additionally, while spectral ranks quantify capacity utilization, they do not fully characterize downstream task behavior.

In conclusion, the study motivates a shift toward optimizer-architecture co-design, suggesting that future scaling laws must account for the implicit inductive biases of the optimizer as a primary determinant of how model capacity is realized.