📊 statistics

Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

This paper introduces "Spectral Lens," a diagnostic protocol using activation covariance and gradient spectra to reveal how batch size influences representation geometry, predict token efficiency, and distinguish between architectural and execution-side optimization gains in large language models.

Original authors: Andy Zeyi Liu, Elliot Paquette, John Sous

Published 2026-05-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Andy Zeyi Liu, Elliot Paquette, John Sous

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the perfect loaf of bread. Usually, you judge how well the baking is going by looking at the final result: does the bread taste good? Is it the right size? In the world of Large Language Models (LLMs), this "taste test" is called training loss. If the loss goes down, everyone assumes the model is learning better.

However, this paper argues that looking only at the final taste is like judging a car race only by who crosses the finish line first, without looking at the engine. Two cars might cross the line at the exact same time, but one could be running on a perfectly tuned engine while the other is sputtering, overheating, and about to break down. The "internal geometry" of the model—the way it actually organizes its knowledge inside its brain—might be completely different, even if the final score is the same.

The authors introduce a new way to look inside the model's brain using a "Spectral Lens." Instead of just looking at the final score, they look at the "music" or the "vibrations" of the model's internal data. They call these vibrations spectra.

Here is a breakdown of their three main discoveries, using simple analogies:

1. The "Batch Size" Secret (The Group Size Effect)

In training AI, you don't show the model one sentence at a time; you show it a "batch" of sentences. The size of this batch is called the batch size.

The Old View: If you train with a small batch or a huge batch, and they both end up with the same "loss" (same final score), you assume they learned the same thing.
The Paper's Discovery: This is an illusion. The authors found that batch size acts like a hidden architect. Even if two models finish with the exact same score, the one trained with a small batch has a very different internal structure than the one trained with a large batch.
The Analogy: Imagine two groups of people trying to solve a puzzle.
- Group A (Small Batch): They work in small teams, passing pieces back and forth constantly. They might solve the puzzle, but they end up with a very specific, rigid way of holding the pieces.
- Group B (Large Batch): They work in one giant circle, sharing everything at once. They solve the puzzle too, but they hold the pieces in a completely different, more fluid way.
- The Result: If you only look at the finished puzzle (the loss), you think they did the same job. But if you look at how they held the pieces (the activation spectrum), you see they are fundamentally different. The paper shows that the "large batch" way usually leads to a more efficient, smoother way of learning.

2. The Crystal Ball (Predicting the Future)

The most exciting part of the paper is that you don't have to wait until the model is finished to know if it's going to be efficient.

The Discovery: By looking at the "tail" of the internal vibrations very early in the training process (like after just 20% of the work is done), the authors can predict exactly how many tokens (words) the model will need to finish the job.
The Analogy: Imagine you are listening to a band rehearse. You don't need to wait for the final concert to know if they are going to be a hit. If you listen to the tail end of their sound (the quiet, fading notes) during the first few songs, you can tell if they are going to be tight and efficient or if they are going to struggle and need to rehearse for weeks.
Why it matters: This allows researchers to pick the best "batch size" and training settings before spending millions of dollars on computing power. They can spot the "winning" setup early by just listening to the model's internal "tail."

3. The "Learning vs. Speed" Detector

The paper also helps distinguish between two types of improvements:

Real Learning: The model actually got smarter and learned new features.
Execution Speed: The model didn't get smarter, but the computer just ran the code faster (like putting a turbocharger on a car without changing the engine).

The Discovery: By looking at two different "spectra" (one for what the model knows and one for how it updates its knowledge), they can tell the difference.
The Analogy:
- Learning-side gains: This is like a student actually studying harder and understanding the math better. The internal "map" of their brain changes.
- Execution-side gains: This is like the student just getting a faster calculator. They solve the same problems, but the math inside their head hasn't changed; they just did it quicker.
- The authors created a "taxonomy" (a classification system) that looks at the model's internal vibrations to say, "Ah, this change made the model smarter," or "This change just made the computer run faster."

The "Toy Model" Proof

To prove these ideas aren't just lucky guesses, the authors built a tiny, simplified "toy" model (like a Lego version of a real brain) where they could see exactly how the learning happened. They showed mathematically that when the model learns a specific pattern, the "vibrations" in its brain shift in a predictable way. This confirmed that the "spectral lens" isn't magic; it's a direct reflection of how the model is actually learning features.

Summary

In short, this paper says: "Don't just look at the final score of an AI model. Look at its internal vibrations."

By using this "Spectral Lens," researchers can:

See that different training settings create different internal brains, even if the scores are the same.
Predict how efficient a model will be just by looking at it early in training.
Tell the difference between a model that is actually getting smarter and one that is just running faster.

This gives scientists a much clearer, more honest view of what is happening inside the "black box" of AI training.

Technical Summary: Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

Problem Statement

The optimization of Large Language Models (LLMs) is traditionally guided by neural scaling laws, which predict that training loss evolves predictably with compute, data, and model size. However, the authors argue that training loss alone creates an "optimization mirage." While loss decreases monotonically, it fails to capture profound, non-monotonic transformations in the model's internal geometry. This disconnect creates a blind spot: practitioners cannot reliably diagnose or tune training processes if loss metrics mask distinct internal representation states. The paper posits that scalar loss is insufficient for distinguishing between different training regimes (e.g., varying batch sizes) or identifying the mechanistic source of architectural improvements (e.g., whether a change improves learning dynamics or merely execution throughput).

Methodology

The authors introduce a "spectral lens" approach, utilizing activation covariance and per-sample gradient singular value decomposition (SVD) spectra as operational diagnostics. Their methodology involves:

Controlled Experimental Setup: Using a modified NanoGPT codebase, they generated a controlled intervention chain of decoder-only models. They evaluated variants across three architectural tiers: 12-layer (d12), 36-layer (d36), and 48-layer (d48) models, trained on the FineWeb dataset.
Spectral Metrics:
- Activation Spectra: They compute the centered covariance of hidden states ( $\hat{\Sigma}_h$ ) from a fixed validation pool. They analyze the power-law decay of the trace-normalized eigenvalues ( $\tilde{\lambda}_j \propto j^{-\alpha}$ ). Specifically, they define a band-restricted exponent $\alpha_{tail}$ over rank windows that scale with model depth (e.g., ranks [100, 200] for d12, shifting to [400, 800] for d48) to capture the "unresolved tail" of the spectrum.
- Gradient Spectra: They compute the SVD of the per-sample gradient matrix ( $G$ ) for specific weight matrices (primarily the final attention output projection $W_O$ ). This measures the concentration of the update direction.
Comparison Protocols: To isolate internal geometric differences, they compare runs under matched conditions: matched final loss, matched token budgets, and matched schedule fractions.
Mechanistic Grounding: To explain empirical observations, they employ a modular-arithmetic toy model with known Fourier features. This allows them to analytically link spectral shifts to the recruitment of task-aligned features, distinguishing between optimization geometry changes and representational capacity changes.

Key Contributions and Results

1. Batch Size as a Latent Determinant of Geometry

The study demonstrates that reaching the same training loss with different effective batch sizes does not imply a shared internal state.

Distinct Spectra: Runs with matched loss but different batch sizes settle into systematically distinct activation covariance spectra.
Muon Sensitivity: While this phenomenon exists under Adam, it is amplified under the Muon optimizer, which produces sharper head concentration and greater spectral separation due to its direction-equalizing update geometry.

2. Early Spectral Signatures Predict Token Efficiency

The authors establish that early-training spectral statistics can forecast downstream efficiency without full-scale training.

Predictive Power: The early activation tail exponent ( $\alpha_{tail}$ ), measured at a fixed token budget (e.g., 20% of training progress), exhibits a strong positive correlation ( $\rho_S \approx 0.97$ ) with the total tokens required to reach a target loss.
Cross-Scale Robustness: This predictive signal generalizes from 12-layer models to 36- and 48-layer models (up to 3.57B parameters), allowing practitioners to identify optimal batch regimes early in the training process.

3. Spectral Taxonomy of Architectural Acceleration

The paper proposes a diagnostic framework to categorize architectural interventions based on whether they drive "learning-side" gains or "execution-side" gains:

Activation-Led Learning: Interventions that reshape representational degrees of freedom (e.g., RoPE, untied heads, SoftCap) manifest as clear displacements in the final activation covariance spectrum.
Gradient-Led Learning: Interventions that alter information routing or attention horizons (e.g., FixedWin, VTE) show dominant geometric shifts in gradient spectra or checkpoint-wise path separation, even if the final activation endpoint remains relatively static.
Throughput Execution-Leaning: Interventions focused on system efficiency (e.g., FP8, sparse value embeddings) primarily improve throughput with comparatively smaller activation-side geometric movement.
Mixed/Tradeoff: Interventions that improve one objective while weakening another.

4. Mechanistic Explanation via Toy Models

Using a modular-arithmetic toy model, the authors provide theoretical grounding for their empirical findings:

Loss vs. Spectrum: They prove that distinct batch regimes can achieve identical loss while possessing fundamentally different activation spectra because Fourier modes approach their target coefficients at different rates.
Tail Exponent and Efficiency: They derive that a flatter (smaller) tail exponent $\alpha_{tail}$ corresponds to a faster recruitment of slow, task-relevant features, mathematically guaranteeing shorter time-to-target for runs with flatter tails.
Intervention Mechanisms: The toy model distinguishes how different tricks work: RoPE restores cyclic symmetry (symmetry bias), Muon acts as spectral preconditioning (optimization geometry), and untied readouts expand the realizable model class (factorization freedom).

Significance and Claims

The paper claims that spectral measurements offer a "natural lens" to move beyond the "optimization mirage" of scalar loss. Its primary significance lies in providing practical and operational diagnostics that:

Reveal Hidden Regimes: They expose distinct internal geometries that scalar loss cannot distinguish, specifically regarding batch size effects.
Enable Early Prediction: They allow for the identification of token-optimal batch regimes early in training, potentially saving significant compute resources.
Diagnose Mechanisms: They separate fundamental learning-centric architectural improvements from purely execution-side speedups, offering a clearer understanding of why an intervention works.

The authors maintain a modest stance, acknowledging limitations such as the current focus on pretraining validation loss rather than downstream zero-shot capabilities, and the reliance on a specific cumulative intervention chain. They view the toy models as mechanistic abstractions rather than faithful models of full language dynamics, and note that the informative tail window is currently selected via a stable empirical protocol rather than a fully automatic estimator. Nevertheless, they argue these signals persist across scales and provide a stable protocol for forecasting dynamics as models grow.