🤖 machine learning

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

This paper reinterprets Transformer architectural components like skip connections, normalization placement, and width expansion as mechanisms that preserve gradient rank across depth, revealing that successful deep network design hinges on navigating the intrinsic tradeoff between rank collapse, ensemble-like behavior, and parameter efficiency.

Original authors: Katie Everett

Published 2026-07-16

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Katie Everett

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a skyscraper out of thousands of tiny, identical Lego bricks. In the world of artificial intelligence, these "bricks" are mathematical layers that process information, and the "skyscraper" is a deep neural network designed to learn complex tasks like recognizing cats in photos or writing stories. The deeper the building, the more complex the problems it can solve. But there's a catch: as you stack more and more layers, the signal trying to travel from the bottom to the top often gets muffled, distorted, or lost entirely. It's like trying to whisper a secret through a hundred people; by the time it reaches the end, the message is garbled or gone.

To fix this, engineers invented a clever trick called a "skip connection." Think of it as building a super-fast elevator shaft right through the middle of your Lego tower. Instead of forcing the message to crawl through every single brick, the elevator lets it zip past the noisy, messy middle sections and land safely at the top. This keeps the signal strong. However, there's a hidden problem that this paper investigates: even if the volume of the message stays loud, the content might still be getting flattened. Imagine if, as the message travels, all the different colors of the Lego bricks started turning into a single shade of gray. The message is still there, but it's lost its richness and variety. In math terms, this is called "rank collapse," where the network loses its ability to see the world in many different directions, becoming stuck in a narrow, boring perspective.

This paper, titled "Transforming Rank," dives deep into the blueprints of modern AI to figure out exactly how the design of these Lego towers affects this loss of color. The researchers, led by Katie Everett at MIT, treat the network like a detective story, examining how skip connections, normalization layers (which keep the signal from exploding), and the specific arrangement of math operations work together. They discover that the famous "elevator shaft" (skip connection) doesn't just save the signal's volume; it actually saves the signal's variety by routing it around the parts of the network that tend to turn everything gray. But they also find a tricky trade-off: if you rely too much on the elevator, the building stops acting like a deep, thinking tower and starts acting like a pile of separate, shallow rooms that don't talk to each other. The paper maps out exactly how to balance these forces, showing that the placement of the "elevator" and the width of the "hallways" inside the tower are the secret ingredients that keep the AI's mind colorful and capable of learning, even when it gets hundreds of layers deep.

The Great Lego Tower Mystery

So, how do we keep a deep neural network from turning into a boring, gray blob? The authors of this paper decided to look at the "feedforward block," which is basically the standard room in the AI's skyscraper. In a typical room, information comes in, gets stretched out, squeezed through a filter (an activation function), and then squeezed back down. The problem is that this stretching and squeezing process is a bit like a photocopier that loses a little bit of detail every time you use it. If you copy a document a hundred times, the text eventually becomes unreadable. In a neural network, this means the "rank" (a measure of how many different directions the information can take) drops as it goes deeper.

The paper starts by re-examining the "skip connection," that famous elevator shaft. Most people thought the elevator was just there to keep the signal from getting too quiet or too loud. But the authors show that its real superpower is saving the rank. By letting the signal bypass the messy "stretching and squeezing" room and go straight through the elevator, the network preserves its variety. However, there's a catch. If the elevator is too strong and the messy room is too weak, the signal never actually learns anything new from the room; it just skips over it. The network then acts like a crowd of people shouting their own separate thoughts (an "ensemble") rather than a single deep thinker where each layer builds on the last. The authors found that the balance between the elevator and the messy room is controlled by a simple ratio: how big the room's contribution is compared to the elevator's.

The Secret Sauce: Where You Put the Normalizer

One of the paper's biggest discoveries is about "normalization," a step that keeps the numbers in the network from going crazy. For a long time, engineers argued about where to put this step: before the messy room (Pre-Norm) or after the elevator (Post-Norm). The paper reveals that this choice isn't just about keeping numbers stable; it's the master switch for the elevator-vs-room ratio.

If you put the normalizer after the elevator (Post-Norm), it resets the signal's size every single time. This keeps the ratio between the room and the elevator constant. The result? The signal keeps losing its variety layer by layer, and eventually, the whole tower collapses into a gray blob. The authors explicitly rule out the idea that the "projection" part of the normalizer (which removes certain directions) is the main villain. They ran simulations showing that even if you remove that part, the collapse still happens. The real culprit is the constant ratio that prevents the signal from recovering.

On the other hand, if you put the normalizer before the messy room (Pre-Norm), the signal is allowed to grow as it travels up the tower. As the signal gets bigger, the messy room's contribution becomes relatively smaller compared to the elevator. This means the ratio changes as you go deeper. The signal loses some variety in the early layers, but because the ratio keeps shifting, the later layers stop losing as much. The rank doesn't disappear; it hits a "floor" and stays there. This explains why modern giant AI models (like the ones that write code or chat with you) almost always use Pre-Norm: it keeps the network from collapsing, even if it means the deep layers become a bit less "active."

The Two-Matrix Magic Trick

The paper also solves a mystery about why the messy room in these towers has two sets of math operations instead of just one. Usually, the room stretches the signal out to be four times wider, applies a filter, and then squeezes it back down. Why not just do it in one step?

The authors found that if you only have one matrix (one step), the signal develops a "mean spike." Imagine that every time the signal goes through the room, it accidentally adds a tiny bit of "gray noise" in the same direction. If you do this a hundred times, that tiny bit of noise grows into a giant, dominant spike that pushes out all the other colors. The signal becomes a single, boring line.

But when you use two matrices, the second one acts like a magical mixer. It takes that growing spike and scrambles it, spreading the noise out so it doesn't dominate. This keeps the signal colorful and prevents the collapse. The paper shows that this is why modern Transformers use two matrices: it's not just about having more power; it's about decorrelating the noise so the network doesn't get stuck on a single direction.

The Width Expansion Threshold

Finally, the paper looks at how wide the "hallway" inside the messy room should be. They found a specific threshold based on a mathematical law called the Marchenko-Pastur law. If the hallway is too narrow, the filter (the activation function) kills off too many directions, and the signal gets stuck. But if you expand the hallway to a certain width (for example, 4 times wider than the input), you give the signal enough room to survive the filter. The authors calculated that for common filters like ReLU, you need at least a 2x expansion to keep the signal full of variety, but the standard 4x expansion used in real-world models is much safer and keeps the signal in great shape.

The Big Picture: A Three-Way Trade-Off

In the end, the paper paints a picture of architecture design as a delicate balancing act. You have three things fighting for your attention:

Rank Collapse: The signal turning gray and losing its variety.
Ensemble Behavior: The signal skipping the learning process and acting like a pile of shallow rooms.
Parameter Count: The cost of adding more bricks (like the second matrix and wider hallways) to fix the problem.

The authors suggest that the best designs navigate this trade-off. They use skip connections to route the signal around the dangerous parts, but they tune the "branch-to-skip" ratio so the signal still learns. They use two matrices and wide hallways to keep the signal colorful inside the rooms. And they use Pre-Norm to let the signal grow naturally, preventing the ratio from getting stuck in a collapsing pattern.

The paper confirms these ideas through simulations and measurements on networks trained on the CIFAR-10 dataset (a standard test for image recognition). They found that networks where the initial rank collapsed failed to learn, while those that maintained a moderate level of rank succeeded. This suggests that the "secret sauce" of modern AI isn't just about making models bigger; it's about carefully designing the internal plumbing to keep the signal's variety alive as it travels through the deep, dark depths of the network.

Technical Summary: Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

Problem Statement

Deep neural networks, particularly Transformers, face a fundamental challenge in signal propagation across depth: rank collapse. As signals (gradients and representations) pass through successive layers of matrix multiplications and nonlinear activations, the effective rank of the input–output Jacobian and the residual representations tends to decrease monotonically. In plain Multi-Layer Perceptides (MLPs), this is inevitable because every layer acts as a rank-reducing operation. While skip connections and normalization are traditionally understood as mechanisms to prevent vanishing or exploding magnitudes, this paper investigates their role in preserving rank. The authors posit that the architectural design of the modern Transformer feedforward block navigates a critical tradeoff between rank collapse, ensemble-like behavior, and parameter count.

Methodology

The paper analyzes the effective rank of three key objects at initialization in residual networks:

The Input–Output Jacobian ( $\partial h_L / \partial h_0$ ): The gradient of the network output with respect to the input.
The Residual Representation ( $Z_\ell$ ): The matrix of forward activations across the residual stream.
The Branch Jacobian ( $J_f$ ): The Jacobian of the residual branch itself.

The authors employ a path decomposition of the input–output Jacobian, expanding the product of residual blocks into a sum of paths that either traverse the residual branch or skip it. They measure the effective rank (exponential of spectral entropy) normalized by the model width. The study systematically varies architectural hyperparameters, including:

Skip connection scaling ( $\beta$ ) and initialization scale ( $\alpha$ ).
Normalization placement: Pre-Norm (branch input), Output-Norm (branch output), and Post-Norm (after residual addition).
Branch structure: Single-matrix vs. two-matrix branches.
Width expansion: The ratio $m/d$ between the hidden dimension and model dimension.

Experiments are conducted on synthetic residual MLPs and validated on CIFAR-10 trainability, alongside theoretical derivations using random matrix theory (Marchenko–Pastur law) and dynamical systems analysis.

Key Contributions and Results

1. Skip Connections as Rank Restoration Mechanisms

The paper reinterprets skip connections not merely as magnitude stabilizers but as mechanisms that route gradients around rank-reducing operations.

Mechanism: The residual branch reduces rank, while the skip connection (identity path) is trivially full-rank. The skip connection restores rank by allowing the gradient to bypass the branch.
Tradeoff: The relative scale of the branch ( $\beta$ $β$ ) versus the skip determines the "branch-to-skip ratio."
- A high ratio (gradient flows mostly through the branch) leads to rapid rank collapse.
- A low ratio (gradient flows mostly through the skip) preserves rank but causes the network to behave like an ensemble of shallow subnetworks rather than a deep, compositional network.
Path Decomposition: The input–output Jacobian is decomposed into paths of order $k$ (number of branch visits). Small branch scales concentrate magnitude on short paths (low $k$ ), reinforcing ensemble behavior.

2. Normalization Placement Controls Rank via Branch-to-Skip Ratio

The placement of normalization layers dictates how the branch-to-skip ratio evolves with depth, directly controlling rank survival.

Post-Norm: Normalization rescales the residual stream to unit magnitude at every block. This keeps the branch-to-skip ratio constant across depth. Consequently, every layer contributes equally to rank loss, leading to rank collapse (effective rank drops to near zero).
Pre-Norm: Normalization is applied to the branch input. The residual stream grows with depth (as a random walk of fixed-magnitude increments), while the branch output remains roughly constant. This causes the branch-to-skip ratio to decay as $1/\sqrt{\ell}$ . As the ratio decays, later layers contribute less to rank loss, allowing the effective rank to plateau at a moderate level.
Projection Terms: The paper demonstrates that the projection terms in normalization Jacobians (which remove specific directions) are not the primary cause of rank collapse. Experiments removing these projections show that rank collapse persists if the branch-to-skip ratio remains constant (Post-Norm), and plateaus if the ratio decays (Pre-Norm).

3. The Two-Matrix Structure Prevents Coherent Mean Spikes

The standard Transformer feedforward block uses two matrices ( $W_{up}, W_{down}$ ) with an expansion and contraction.

The Problem: With a single matrix and an uncentered activation (e.g., ReLU), the mean of the branch output is non-zero and aligned across layers. This creates a coherent mean spike (a rank-1 vector) that accumulates linearly with depth, dominating the spectrum of the residual representation and collapsing its rank.
The Solution: The second matrix ( $W_{down}$ ) decorrelates the branch means. It rotates the shared "all-ones" component of the mean into random directions for each layer, preventing coherent accumulation. This keeps the mean spike bounded and preserves the effective rank of the forward representation.
Centered Activations: Activations with zero mean (e.g., Tanh) naturally avoid this spike, but the two-matrix structure is necessary for common uncentered activations.

4. Width Expansion Preserves Branch Jacobian Rank

The paper analyzes the rank of the branch Jacobian $J_f = W_{down} D W_{up}$ , where $D$ is a diagonal mask of activation derivatives.

Survival Rate: The activation function masks or attenuates directions. For ReLU, only half the units survive ( $p=0.5$ ).
Full-Rank Threshold: Using the Marchenko–Pastur law, the authors derive that the branch Jacobian becomes full rank only when the width expansion ratio $m/d$ $m / d$ exceeds $1/p(\sigma)$ $1/ p (σ)$ .
- For ReLU ( $p=0.5$ ), the threshold is $m/d = 2$ .
- For smooth activations like GELU, the threshold is higher ( $m/d \approx 2.2$ ).
Conditioning: While the threshold ensures full rank, practical architectures use larger expansions (e.g., $4\times$ ) to ensure the submatrix is well-conditioned (smallest singular value is not near zero).

Significance and Claims

The paper claims to recast deep network architecture design as a navigation of an intrinsic tradeoff among rank collapse, ensemble-like behavior, and parameter count.

Unification: The authors unify disparate literature on normalization placement, depth scaling, and dynamical isometry under the single metric of the branch-to-skip ratio. They show that the success of Pre-Norm over Post-Norm is not just about magnitude stability but about allowing this ratio to decay, thereby preserving rank.
Predictive Power: The initialization effective rank of the input–output Jacobian is shown to predict whether a network will train on CIFAR-10. Configurations where the rank collapses to near zero fail to train, while those that plateau succeed.
Architectural Necessity: The analysis provides a theoretical justification for specific Transformer design choices:
- Pre-Norm is favored because it induces a decaying branch-to-skip ratio.
- Two-matrix branches are necessary to decorrelate means and prevent representation collapse.
- Width expansion (e.g., $4\times$ ) is required to ensure the branch Jacobian remains full rank and well-conditioned.
Ensemble vs. Composition: The paper reframes the "ensemble behavior" observed in deep residual networks not merely as a pathology but as the alternative to rank collapse. A network that routes all gradients through skips preserves rank but loses the ability for layers to compose; a network that routes all gradients through branches composes but collapses rank. The optimal design balances these forces.

The authors conclude that while their analysis focuses on initialization, the spectral properties of gradients and representations provide a first-principles lens for understanding why modern architectures scale and how they might be improved, moving beyond trial-and-error design.