🤖 machine learning

A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

This paper derives a closed-form depth-width scaling law for neural collapse in deep linear networks, demonstrating that layerwise representation geometry can be predicted analytically from the depth-to-width ratio and data properties without requiring post-training analysis.

Original authors: Pankhil Gawade, Kirsten Fischer, Guy Wolf, Ewa Szczurek

Published 2026-10-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pankhil Gawade, Kirsten Fischer, Guy Wolf, Ewa Szczurek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Deep learning has transformed how machines see, hear, and understand the world, yet the inner workings of these systems often remain a mystery. We know that neural networks are built from layers of simple processing units, stacked to create immense depth, and that each layer contains thousands of these units, creating great width. For years, scientists have wondered how the specific balance between the number of layers and the number of units in each layer changes what the network actually learns. Usually, the only way to find out is to build a network, train it on data, and then look at the results, a process that tells us what happened but not why it happened or how to predict it before the training begins. A key phenomenon observed in these trained networks is called neural collapse. As a network learns, the representations it builds for different categories of data tend to become incredibly distinct from one another, while the representations for items within the same category become nearly identical. It is as if the network organizes its internal map so that every example of a cat clusters tightly together, and every example of a dog clusters tightly together, with the two groups standing far apart.

A team of researchers has now shown that for a specific type of network, this organization is not a random accident of training but a predictable outcome that can be calculated in advance. By focusing on a simplified version of a neural network that uses linear mathematics, the scientists discovered a precise rule that links the network's architecture to the geometry of the data it learns. They found that the degree to which the network compresses similar data points together depends directly on the ratio of the network's depth to its width. If you know how many layers the network has and how many units are in each layer, along with the basic properties of the data it is learning, you can determine exactly how much the network will compress the data, without ever having to run a single training session. This is a significant shift from the usual trial-and-error approach, offering a clear, mathematical law that describes how the structure of a network shapes its understanding of the world.

The researchers approached this problem by treating the network's internal representations not as a collection of individual data points, but as a single, unified map of relationships. They developed a way to measure the "tightness" of these clusters using a concept called the feature kernel, which acts like a blueprint of how the network views the similarity between any two pieces of data. Using this blueprint, they were able to express three different measures of how well the network has organized its data entirely in terms of this blueprint and the labels of the data. This means that instead of needing to see the actual data points inside the network to understand its behavior, one can simply look at the mathematical relationships between them. The team proved that for networks with a specific, symmetric structure, these relationships follow a strict pattern as the data moves from the first layer to the last.

In these symmetric networks, the researchers derived an exact formula showing that the compression of data within each category happens in a geometric progression. With every new layer the data passes through, the spread of similar items shrinks by a constant factor. This factor is determined by the ratio of the network's depth to its width. If the network is very wide compared to its depth, the compression happens slowly. If the network is deep relative to its width, the compression accelerates. Crucially, the rate at which this compression occurs is not a fixed number chosen by the engineer; it is determined by the data itself, the initial settings of the network, and the architecture, all working together in a self-consistent loop. The researchers verified this theory with computer simulations, showing that the predicted rate of compression matched the actual behavior of the network perfectly. They also tested this on slightly more complex, non-linear networks and found that while the exact formula did not apply, the general trend of the data organizing itself in the same direction held true, suggesting the underlying principle is robust.

This work provides a rare glimpse into the "laws of nature" for artificial intelligence. Just as a physicist might predict how a gas will expand based on temperature and pressure without watching every molecule, these researchers have shown that one can predict how a neural network will organize its knowledge based on its size and the data it sees. The findings suggest that the depth-to-width ratio is a fundamental control knob for representation learning. By understanding this relationship, designers might eventually be able to choose the right architecture for a specific task without needing to train dozens of different models to see which one works best. The study confirms that the geometry of learned representations is not chaotic but follows a precise, calculable path, turning the mysterious process of deep learning into something that can be understood and predicted from the very start.

Technical Summary: A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

Problem Statement

The relationship between neural network architecture (specifically depth and width) and the geometry of learned representations is typically analyzed empirically after training. While previous works have identified the depth-to-width ratio ( $L/N$ ) as a controlling parameter for finite-width kernel fluctuations and effective depth, a predictive, closed-form understanding of how architectural choices shape representation geometry during feature learning remains elusive. Specifically, the evolution of Neural Collapse (NC)—a phenomenon characterized by the minimization of within-class variability (NC1), the formation of simplex-structured class means (NC2), and the alignment of classifiers with class means (NC3)—across network layers has not been derived analytically from first principles in a tractable feature-learning setting.

Methodology

The authors develop a theoretical framework that combines Neural Collapse observables with self-consistent feature-learning theory for deep linear networks.

Kernel Formulation of NC Observables:
The paper first extends the kernel formulation of Neural Collapse. While Kothapalli and Tirer [2025] established that NC1 can be expressed via the feature kernel, this work derives closed-form kernel expressions for NC2 (angular deviation from a simplex Equiangular Tight Frame) and NC3 (misalignment between the ridge readout and centered class means). These expressions depend solely on the feature kernel $K$ and the class labels $Y$ , eliminating the need for explicit feature coordinates.
Feature-Learning Theory:
The authors adopt the Bayesian feature-learning framework of Fischer et al. [2024], which derives coupled forward-backward self-consistency equations for the Maximum-A-Posteriori (MAP) layerwise kernels. This theory is exact in the proportional limit where the number of samples $P$ and network width $N$ scale together ( $P/N = \alpha$ ).
Exchangeable Class Geometry:
To obtain an analytic solution, the authors assume an exchangeable class geometry, where the input kernel is invariant under permutations of samples within classes and permutations of classes. Under this symmetry, the matrix-valued kernel equations decouple into three scalar channels:
- Global mean ( $E$ )
- Centered between-class means ( $P_B$ )
- Within-class fluctuations ( $P_W$ )
This reduction allows the derivation of a scalar geometric law governing the evolution of the within-class to between-class variance ratio across layers.

Key Contributions

1. Kernel Identities for NC2 and NC3

The paper provides new identities expressing NC2 and NC3 directly in terms of the feature kernel $K$ and labels $Y$ .

NC2 is formulated as the Frobenius norm distance between the cosine-normalized Gram matrix of centered class means and the Gram matrix of a unit-norm simplex ETF.
NC3 is formulated as the average cosine misalignment between the bias-free ridge readout directions and the centered class means.
These identities allow the computation of all three training-set NC observables without forming the feature matrix $Z$ .

2. Exact Geometric Depth Law

For deep linear networks with exchangeable class structure, the authors derive an exact law for the layerwise evolution of NC1:
$NC^{(l)}_1 = NC^{(0)}_1 \left( \frac{t_W}{t_B} \right)^l$
where $t_W$ and $t_B$ are layer-independent transfer factors determined by the data, prior, and architecture. This demonstrates that within-class compression occurs geometrically with depth, driven by the ratio of the transfer factors for within-class and between-class channels.

3. Depth-Width Scaling Law

In the large-width limit with small regularization ( $\kappa$ ), the accumulated compression is shown to be controlled by the depth-to-width ratio $L/N$ . The terminal NC1 scales as:
$NC^{(L)}_1 \approx NC^{(0)}_1 \exp\left( -\frac{rL}{N} \right)$
Here, $r$ is a rate parameter determined self-consistently by the signal-to-noise ratio (SNR), prior variance, and architecture. This establishes $L/N$ as the primary architectural variable controlling representation compression, with the rate $r$ emerging from the data and model properties rather than being an external hyperparameter.

Results

Analytic Verification: Numerical experiments on the exactly exchangeable population kernel confirm that the closed-form geometric law matches the numerical fixed-point solution across various depths and widths.
Data Dependence: The contraction rate $\log(t_W/t_B)$ is shown to depend explicitly on the input SNR, validating the self-consistent determination of the rate.
Scaling Behavior: Experiments varying $L$ and $N$ while holding $L/N$ constant demonstrate that the terminal NC1 follows the predicted exponential decay, confirming the $L/N$ scaling law.
Finite-Width Trained Networks: When applied to SGLD-trained finite-width linear networks on sampled Gaussian data, the closed-form prediction (using measured input eigenvalues) closely matches the observed geometric decay of NC1, despite deviations from the idealized population setting.
Nonlinear Extensions: On a nonlinear four-class XOR task, the kernel observables (NC1, NC2, NC3) derived from the self-consistent theory evolve in the same qualitative direction as those in trained finite-width networks, suggesting the kernel framework captures the geometry of representation evolution even beyond the solvable linear regime.

Significance and Claims

The paper claims to provide a tractable representation-level scaling law derived from feature-learning theory rather than fitted post-training. Its primary significance lies in:

Predictive Power: It allows the prediction of representation geometry (specifically the degree of neural collapse) directly from the data, prior, and architecture without training candidate networks.
Architectural Insight: It identifies the depth-to-width ratio $L/N$ as the controlling parameter for representation compression in deep linear networks, with the compression rate determined self-consistently by the problem structure.
Theoretical Unification: It unifies the study of Neural Collapse with finite-width kernel theory, extending the known NC1 identity to NC2 and NC3 and providing a closed-form description of their layerwise evolution.

The authors note that while the exact geometric law is derived for exchangeable deep linear networks, the kernel observables remain informative for nonlinear networks, capturing the qualitative evolution of representation geometry. Extending these predictive laws to general nonlinear networks and arbitrary data geometries is identified as a direction for future work.