🤖 machine learning

Unifying Low Dimensional Spectra in Deep Learning

This paper provides a unifying analytic explanation for the low-dimensional eigenspectra of various deep learning matrices by demonstrating that they arise from Deep Neural Collapse (DNC) within unconstrained feature models, thereby characterizing both eigenvalues and eigenvectors for linear and ReLU networks.

Original authors: Connall Garrod, Jonathan P. Keating

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Connall Garrod, Jonathan P. Keating

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a massive, complex machine (a deep neural network) learns to sort things, like distinguishing cats from dogs. Inside this machine, there are billions of tiny knobs and dials (parameters) that get adjusted during training.

For a long time, scientists have noticed something strange happening when these machines get really good at their job. If you look at the "energy landscape" or the mathematical map of how the machine is changing, it doesn't look like a chaotic mess. Instead, it looks surprisingly organized, almost like a city with a few skyscrapers and a vast, flat plain.

This paper, titled "Unifying Low Dimensional Spectra in Deep Learning," explains why this happens and reveals that the same simple rule is behind the organization of several different parts of the machine.

Here is the breakdown using simple analogies:

1. The Mystery: The "Skyscraper and Plain" Effect

When researchers look at the mathematical "spectrum" (a list of numbers that describe the machine's shape and behavior) of these networks, they see a pattern:

The Bulk: Most of the numbers are tiny, clustered near zero. Imagine a vast, flat plain.
The Outliers: A few numbers are huge and stand far apart. Imagine a few tall skyscrapers rising from that plain.

Scientists have seen these "skyscrapers" in different parts of the machine: in the Hessian (which measures how steep the learning path is), in the gradients (the direction the machine moves to learn), and in the weights (the actual knobs). They also noticed that the number of skyscrapers often matches the number of categories the machine is learning (e.g., 10 skyscrapers for 10 types of digits).

But until now, no one had a single explanation that connected all these different "skyscrapers" together.

2. The Culprit: "Neural Collapse"

The paper identifies the source of this organization as a phenomenon called Deep Neural Collapse (DNC).

Think of the machine's internal memory as a library.

Before training: The books (data points) are scattered randomly on the shelves.
After training (Neural Collapse): The machine gets so good at its job that it organizes the library perfectly. All the books about "cats" are stacked neatly into one single, tight pile. All the books about "dogs" are stacked in another tight pile.
The Result: Instead of millions of individual books, the machine effectively only needs to remember one representative pile for each category. These piles are the "feature means."

The paper argues that this neat stacking (Neural Collapse) is the master key. It is the reason why the "skyscrapers" appear in the math.

3. The Unifying Explanation

The authors used a simplified mathematical model (called the Unconstrained Feature Model) to prove that if the machine achieves this "Neural Collapse," then:

The Hessian (the map of the terrain) will automatically form those specific skyscrapers.
The Gradients (the learning direction) will automatically align with those skyscrapers.
The Weights (the knobs) will automatically take on a low-dimensional shape.

The Analogy:
Imagine a group of dancers (the data) moving on a stage.

The Old View: Scientists looked at the stage lighting (Hessian), the dancers' movements (Gradients), and the choreography notes (Weights) separately. They saw patterns in each but couldn't explain why they matched.
This Paper's View: The paper says, "Look at the dancers themselves." If the dancers all collapse into perfect, tight groups (Neural Collapse), then the lighting, the movements, and the notes must follow a specific, simple pattern. The dancers' formation dictates everything else.

4. What They Actually Proved

The paper provides a mathematical recipe showing exactly how to build these "skyscrapers" using only the "feature means" (the center of those tight piles of data).

The Recipe: If you know where the center of the "cat pile" and the "dog pile" are, you can mathematically construct the entire Hessian matrix, the gradients, and the weights.
The Proof: They proved this works for both simple linear networks and complex networks with "ReLU" activations (a common type of switch used in real AI).
The Validation: They tested this on real datasets (like MNIST handwritten digits) and standard AI architectures. The real machines behaved exactly as their simplified math predicted: the "skyscrapers" appeared, and the "bulk" flattened out.

5. Why This Matters (According to the Paper)

The paper claims this is a unifying explanation. Instead of treating the Hessian, gradients, and weights as separate mysteries, we can now understand them all as different reflections of the same underlying event: Neural Collapse.

It suggests that the "flatness" of the learning landscape (which helps machines generalize and not forget what they learned) is directly caused by this collapse of data into neat, low-dimensional piles.

In short: The paper says, "Stop looking at the complex machinery of deep learning as a black box. If you look at how the data organizes itself into neat piles (Neural Collapse), you can predict exactly how the math of the machine will look, why it has those specific 'skyscrapers,' and why it learns so effectively."

Technical Summary: Unifying Low Dimensional Spectra in Deep Learning

Problem Statement

Deep learning classification networks trained in the overparameterized regime consistently exhibit low-dimensional structures in the eigenspectra of their matrices (e.g., Hessians, gradients, and weights). Empirical observations, notably by Papyan (2020), reveal a "bulk–outlier" structure where most eigenvalues cluster near zero (the bulk), while a small number of outliers emerge. Specifically, the Hessian often displays $K$ large outliers and $K(K-1)$ smaller outliers forming a "mini-bulk," where $K$ is the number of classes. Additionally, gradients have been observed to align with the top- $K$ outlier eigenspace of the Hessian.

Despite the centrality of the Hessian to generalization, optimization, and robustness, theoretical explanations for these specific spectral structures have been fragmented. Existing approaches using random matrix theory, spin-glass analogies, or neural tangent kernels often capture only subsets of the behavior or rely on assumptions that do not hold in practice. Furthermore, the connection between these spectral phenomena and Deep Neural Collapse (DNC)—a phenomenon where penultimate-layer features concentrate around class means forming simplex equiangular tight frames (ETFs)—has remained largely unexplored mathematically.

Methodology

The authors employ the Deep Unconstrained Feature Model (UFM), a theoretical framework where feature vectors are treated as freely optimized variables rather than outputs of a fixed feature map. This model is widely used to study the emergence of DNC in overparameterized networks.

The paper proceeds in two main analytical steps:

Layer-wise Analysis: The authors derive analytic expressions for the eigenvalues and eigenvectors of layer-wise Hessians, gradients, and weight matrices within the deep linear UFM. They assume the network has reached a global optimum satisfying DNC conditions (feature collapse to class means, orthogonal class means, and weights as linear combinations of means).
Global Extension: They extend these results to the full-network Hessian, demonstrating that under minimal assumptions (specifically, that the feature map is expressive enough and the network is sufficiently deep), the global Hessian inherits the spectral structure of the layer-wise Hessians.
ReLU Extension: The analysis is extended to networks with ReLU activations. The authors show that while DNC is not a global optimum in the ReLU case generally, the best-performing DNC solutions (which emerge over long training timescales) effectively behave like linear models with non-negative feature constraints, allowing the linear theory to apply.

Key Contributions

1. Unifying Analytic Explanation for Bulk-Outlier Structure

The paper provides the first mathematical derivation that simultaneously recovers all empirically observed details of the bulk–outlier structure across Hessians, gradients, and weights.

Hessian Spectra: The authors prove that the layer-wise Hessian has rank $K^2$ $K^{2}$ . The non-zero eigenvalues are constructed entirely from the feature means. Specifically, the spectrum decomposes into:
- $K$ dominant outliers associated with the class means ( $G_{class}$ ).
- $K(K-1)$ smaller outliers forming a "mini-bulk" associated with cross-class differences ( $G_{cross}$ ).
- A bulk of zero eigenvalues associated with within-class variance ( $G_{within}$ ), which vanishes under DNC.
Gradient Alignment: The theory proves that the gradient lies entirely within a $K$ -dimensional subspace spanned by the $K$ dominant eigenvectors of the Hessian. This explains the empirical observation that gradients align with the top- $K$ outlier space, despite the Hessian having $K^2$ non-zero eigenvectors.
Weight Spectra: The weight matrices are shown to have rank $K$ , with singular vectors determined by the feature means.

2. Causal Role of Deep Neural Collapse (DNC)

The central claim is that DNC is the source of these low-dimensional spectra. The eigenvalues and eigenvectors of these matrices are explicitly constructed from the feature means ( $\mu_c$ ), which are the defining objects of DNC.

The eigenvectors of the Hessian are Kronecker products of normalized feature means: $\hat{\mu}_{c}^{(l+1)} \otimes \hat{\mu}_{c'}^{(l)}$ .
This establishes a direct causal link: the geometric organization of features (DNC) dictates the curvature (Hessian) and optimization dynamics (gradients) of the network.

3. Extension to ReLU and Global Networks

ReLU Networks: The authors define a specific class of DNC solutions for ReLU networks where intermediate representations are non-negative. They prove that these solutions are global minima of the corresponding linear model and satisfy the same spectral properties as the linear UFM.
Global Hessian: The paper proves that the full Hessian of the network inherits the $K^2$ outlier structure of the layer-wise Hessians in the limit of large depth, provided the feature map is sufficiently expressive.

Results

Theoretical Results

Theorem 3.1 & 3.2: In the deep linear UFM at a global optimum, the layer-wise Hessian has rank $K^2$ . The non-zero eigenvalues are equal (in the MSE case), and the eigenvectors are formed by the tensor products of class means. The decomposition into $G_{class}$ and $G_{cross}$ components yields exactly $K$ and $K(K-1)$ non-zero eigenvalues respectively.
Theorem 3.3: The gradient vector $\tilde{g}^{(l)}$ (excluding regularization) has exactly $K$ equal non-zero coefficients when expanded in the basis of Hessian eigenvectors, confirming it resides in the top- $K$ subspace.
Theorem 3.4: Weight matrices $W_l$ have rank $K$ , with singular vectors given by the class means.
Theorem 4.3: These results hold for ReLU networks under the definition of DNC where intermediate features are non-negative.

Empirical Validation

The authors validate their theory on standard architectures (ResNet-20) with fully connected heads trained on MNIST and CIFAR-10.

Spectral Convergence: Histograms of Hessian eigenvalues show the emergence of $K^2$ outliers from the bulk during training, converging to the predicted values.
Alignment: The squared cosine similarity between predicted eigenvectors (based on feature means) and true Hessian eigenvectors approaches 1.
Gradient Decomposition: Gradient decomposition coefficients confirm that gradients concentrate in the $c=c'$ subspace (size $K$ ) and vanish in the $c \neq c'$ subspace.
ReLU Behavior: Experiments with ReLU networks show that feature matrices evolve to have non-negative entries, and the spectral structure matches the linear theory predictions.

Significance and Claims

The paper claims to provide a unifying analytic explanation for a wide range of spectral phenomena in deep learning that were previously treated as separate empirical observations. By identifying DNC as the unifying mechanism, the work:

Clarifies the relationship between flatness and DNC: It suggests that the local flatness of the loss landscape (governed by the Hessian) is a direct consequence of the geometric collapse of features.
Moves beyond eigenvalues: Unlike prior empirical characterizations that focused primarily on eigenvalues, this work provides detailed analytic expressions for eigenvectors, revealing the specific geometric structure (feature means) that drives the spectra.
Offers a new lens for diagnostics: The authors suggest that because DNC shapes the Hessian, controlling the emergence of DNC could be a practical tool for improving generalization, optimization, and network design, potentially offering a more interpretable target than the Hessian itself.

The authors remain modest regarding the limitations, noting that their results rely on the UFM assumptions (overparameterization, global optima) and that real-world networks exhibit noise due to finite data and non-ideal optimization. However, the strong empirical agreement suggests that the UFM captures the essential geometric mechanisms driving spectral behavior in deep networks.