🔬 condensed matter

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer

This paper introduces a two-level dynamical mean-field theory to analyze the evolution of hidden-weight spectra in wide neural networks, revealing how outlier dynamics and hyperparameter transfer differ between parameterizations and how spectral behavior shifts from outlier-driven to bulk-restructuring mechanisms as task complexity and output dimensions increase.

Original authors: Clarissa Lauditi, Cengiz Pehlevan, Blake Bordelon

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Clarissa Lauditi, Cengiz Pehlevan, Blake Bordelon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a neural network as a massive, chaotic orchestra of musicians (neurons) trying to learn a song. At the start, everyone is playing random notes. As the conductor (the training algorithm) guides them, they start to find the melody.

This paper is a theoretical study of how the "sound" of this orchestra changes as it learns. Specifically, the authors look at the "spectral dynamics," which is a fancy way of describing the distribution of energy in the network's connections.

Here is the breakdown of their findings using simple analogies:

1. The Two Types of Sound: The Crowd and the Soloists

When the orchestra starts playing randomly, the sound is a uniform, static hum. In math terms, this is called the "Bulk." It's like a crowd of people murmuring in a stadium; you can't hear any individual voice, just a wall of noise.

As the network learns, a few specific "musicians" (or directions in the math) start to stand out and play a clear, distinct melody. These are the "Outliers" or "Spikes."

The Paper's Discovery: The authors developed a new mathematical tool (called Two-Level DMFT) to track exactly when and how these soloists emerge from the crowd.
The Catch: In standard math, we usually assume these soloists are independent of the crowd. But in a learning neural network, the soloists are created by the crowd. They are statistically linked. The authors' new tool is the first to accurately track this specific relationship.

2. The "Magic" of Scaling (µP vs. NTK)

One of the biggest questions in AI is: "If I make my network 10 times bigger, do I have to re-tune all the settings?"

The Old Way (NTK): Imagine a small band. If you suddenly add 100 more musicians, the sound changes completely. The "soloists" behave differently, and you have to adjust the volume knobs (learning rates) entirely. The paper shows that in this "NTK" setup, the behavior of the soloists depends heavily on how big the orchestra is.
The New Way (µP): The authors studied a specific tuning method called µP (Maximal Update Parameterization). They found that with µP, the "soloists" behave consistently regardless of the orchestra size.
- The Analogy: It's like a magic conductor who ensures that whether you have 10 musicians or 10,000, the lead singer always hits the same high note at the same time. This explains why µP allows for "hyperparameter transfer": you can train a small model, find the perfect settings, and apply them to a giant model without re-tuning.

3. When the Whole Orchestra Changes (The "Extensive" Problem)

The paper found that their "Crowd vs. Soloist" theory works perfectly for simple tasks (like recognizing cats vs. dogs in CIFAR-10). In these cases, only a few "soloists" emerge from the noise.

However, for massive tasks like Language Models (GPT) or ImageNet:

The Shift: The output vocabulary is so huge (think 50,000 words) that it's not just a few soloists emerging. Instead, the entire crowd changes its character. The "Bulk" itself shifts and reshapes.
The Analogy: It's no longer just a few people standing up to sing; the entire stadium's atmosphere changes. The "noise" becomes a different kind of noise.
The Solution: The authors built a "toy model" (a simplified simulation) to show that even in this chaotic scenario, if the network is wide enough, the "edge" of the sound (the loudest part) eventually stabilizes. This suggests that even for giant models, there is a predictable limit to how the learning dynamics settle down.

4. The "Edge of Stability"

The paper touches on a concept called the "Edge of Stability" (EoS).

The Analogy: Imagine driving a car. If you go too fast, you crash. If you go too slow, you don't get anywhere. There is a "sweet spot" right at the edge of crashing where you go the fastest.
The Finding: The paper shows that the "soloists" (the outliers) are the ones that determine this speed limit. In the µP setup, this speed limit is stable no matter how big the network is. In the old setup, the speed limit changes depending on the size of the network, making it hard to predict.

Summary

What they did: They created a new mathematical microscope to watch how neural networks learn, specifically tracking how "outlier" signals emerge from random noise.
What they found:
1. µP is special: It keeps the learning dynamics consistent across different network sizes, explaining why it's great for scaling up models.
2. Simple vs. Complex: For small tasks, learning creates a few distinct "soloists." For massive tasks (like LLMs), learning reshapes the entire "crowd" of noise.
3. Predictability: Even in the massive reshaping scenario, the "edge" of the learning dynamics eventually stabilizes if the network is wide enough.

The paper is purely theoretical; it explains why certain training methods work better than others and provides the mathematical "blueprint" for how these networks evolve, without claiming to solve specific real-world problems like curing diseases or building self-driving cars.

Technical Summary: Spectral Dynamics in Deep Networks

Problem Statement

Understanding the evolution of hidden representations in large neural networks remains a central challenge in deep learning theory. While infinite-width limits have provided significant insights, a critical gap exists in regimes where networks exhibit feature learning (the "rich" regime) rather than merely fitting linear models on static features (the "lazy" or kernel regime). Specifically, there is a lack of complete theoretical accounts for:

How the spectral dynamics of hidden weight matrices evolve during training, particularly the emergence of isolated singular values (outliers) from the random bulk.
The mechanisms underlying hyperparameter transfer (e.g., learning rate stability) across different network widths.
The behavior of networks with extensive output dimensions (e.g., language models with large vocabularies), where the standard "bulk + finite outliers" picture may fail.

Current Random Matrix Theory (RMT) models, such as the BBP (Baik-Ben Arous-Péché) transition, typically assume that signal spikes are statistically independent of the random noise bulk. However, in trained neural networks, the learned directions (spikes) are causally dependent on the initial random weights, rendering standard spiked matrix models insufficient.

Methodology

The authors develop a two-level Dynamical Mean Field Theory (DMFT) framework to track the singular value spectra of wide, randomly initialized deep networks trained via (stochastic) gradient descent.

Core Theoretical Framework

Statistically Coupled Spikes: The theory models the weight matrix $W^{(S)}$ as a random initialization $W^{(0)}$ plus a finite-rank update (spikes) where the spike vectors are functionally dependent on the initial random matrix. This captures the causal dependence of learned features on initialization.
Two-Level DMFT:
- Level 1 (Training Dynamics): Describes the evolution of the training fields (forward features $\phi$ and backward gradients $g$ ) generated by gradient descent. These fields determine the structure of the spikes.
- Level 2 (Spectral Probe): Introduces a linear spectral probe to resolve the spectrum of the resulting weight matrix. This level distinguishes between the continuous bulk (governed by the Marchenko-Pastur law) and isolated outliers.
Outlier Detection Criterion: The framework derives a finite-dimensional condition for the emergence of outliers. Specifically, outliers correspond to the zeros of the determinant of a matrix-valued function $A(z)$ , constructed from the correlation and response functions of the DMFT.

Application Regimes

The framework is applied to two specific settings:

Super-Wide Nonlinear Networks: Infinite-width limits with fixed batch size and training time (Mean-field/ $\mu$ P scaling).
Proportional High-Dimensional Linear Networks: Limits where width ( $N$ ), input dimension ( $D$ ), and sample size ( $P$ ) diverge with fixed ratios ( $\nu = N/D$ , $\alpha = P/D$ ).

Key Contributions

1. Theory of Statistically Dependent Spikes

The paper provides an exact theoretical characterization of singular value dynamics for matrix ensembles where spikes are statistically coupled to the random bulk. This extends standard RMT to the context of trained neural networks, yielding a criterion for when and how outliers "escape" the bulk (BBP transition) as a function of training time, width, and initialization.

2. Mechanism of Hyperparameter Transfer

The theory explains why $\mu$ P (maximal update parameterization) enables width-consistent hyperparameter transfer, while NTK parameterization does not:

$\mu$ P: The leading spectral outlier (NTK mode) exhibits width-stable dynamics. The time to reach the "Edge of Stability" (EoS) and the growth of the leading eigenvalue are consistent across different widths.
NTK-P: Despite converging to a stable large-width limit, the outlier dynamics are strongly width-dependent, leading to inconsistent learning rate transfer.

3. Limitations of the Finite-Outlier Picture

The authors demonstrate that the "bulk + finite outliers" picture is descriptive for tasks with small output channels (e.g., CIFAR-10 classification). However, for tasks with extensive output channels (where the number of classes $C$ scales with width $N$ , such as GPT-style language modeling), training restructures the entire spectral bulk rather than producing isolated spikes.

4. Extensive Output Channel Model

To address the extensive regime, the authors propose a toy model for linear networks with $C \sim O(N)$ output channels. They show that in this regime, the spectral bulk shifts and broadens, and the edge of the spectrum converges only when the width is sufficiently large relative to the output dimension.

Results

Empirical and Theoretical Validation

CIFAR-10 (Nonlinear Networks): The theory accurately predicts the dynamics of spectral outliers in wide ResNets trained on CIFAR-10. The observed dynamics show a finite number of spikes detaching from a stable Marchenko-Pastur bulk, consistent with the BBP transition.
Deep Linear Networks:
- The theory predicts that in $\mu$ P, the leading NTK eigenvalue grows toward the Edge of Stability in a width-consistent manner.
- In contrast, NTK parameterization shows strong width dependence in the leading eigenvalue's growth, failing to support consistent learning rate transfer.
ImageNet and GPT (Extensive Outputs):
- ImageNet: Training modifies the bulk of the hidden-weight spectrum (shifting it left and creating a right tail) rather than just creating isolated outliers.
- GPT Pretraining: Similar bulk restructuring is observed in language models where the vocabulary size ( $\sim 50k$ ) exceeds the hidden width. The spectral density cannot be described by a constant bulk plus a finite set of outliers.
- Toy Model: The proposed linear network model with extensive classes recapitulates this bulk restructuring. Theoretical analysis shows that the spectral edge converges to a shifted Marchenko-Pastur plateau, with finite-width corrections scaling as $O(1/\nu)$ .

Significance and Claims

The paper claims to provide a mechanistic diagnostic for:

Feature Learning: The evolution of the spectral bulk and isolated outliers serves as a signature of the transition from lazy to rich regimes.
Hyperparameter Transfer: Width-stable outlier dynamics are identified as a necessary condition for successful learning rate transfer across model sizes. The theory explains why $\mu$ P succeeds where NTK fails in this regard.
Scaling Laws: The work connects spectral dynamics to neural scaling laws, suggesting that the "bulk + outlier" picture is insufficient for modern large-scale tasks with extensive outputs, necessitating a view of training as a restructuring of the entire spectral bulk.

The authors remain modest regarding the scope, acknowledging that the current theory applies strictly to super-wide nonlinear networks (where the bulk+spike picture holds) or proportional linear networks with finite output channels. The analysis of extensive classes is currently restricted to a single gradient step, and the dynamics of the spectrum in fully trained extensive-output networks are left for future work.