All Routes Lead to Collapse — Plain-Language Explanation

The Big Idea: The "Blind Judge" Problem

Imagine you are a judge in a talent show. Your job is to pick the best performers (tokens) to give a spotlight to. You have a rulebook (the algorithm) that tells you how to score them.

The paper argues that all modern AI models (like Transformers, Mamba, RWKV, etc.) use a rulebook that is "blind" to a specific detail.

The Rulebook: It scores performers based on how well their "content" matches your query.
The Blind Spot: The rulebook completely ignores how loud or big the performer is. It only cares about the shape of their voice, not the volume.

Because the judge is blind to volume, the system gets confused. To make a decision, the system is forced to do something extreme: it stops spreading the spotlight around and instead piles all the attention onto just one or two performers.

The paper calls this "Collapse." It's not a bug in the code; it's a mathematical necessity. If your measuring tape is broken (blind to size), the system has to compensate by squeezing everything into a tiny corner to make sense of it.

The Three Symptoms (The "Signature")

The paper identifies three things that happen whenever this "blind judge" is in charge. Think of these as the symptoms of the same disease:

The "Attention Sink" (The Spotlight Pile-up):
Instead of the spotlight moving smoothly between many actors, it gets stuck on just a few. In AI terms, a tiny number of tokens (like the very first word of a sentence) absorb almost all the attention.
- Analogy: Imagine a classroom where the teacher stops asking questions to the whole class and suddenly starts staring only at the student sitting in the front row, ignoring everyone else.
Rank Collapse (The Flattening):
The AI's internal "thoughts" (representations) stop being diverse. They all start looking the same, shrinking down into a simple, low-dimensional shape.
- Analogy: Imagine a 3D sculpture that slowly gets squashed flat until it looks like a 2D drawing. All the depth and nuance are lost, and everything becomes a flat sheet.
Norm Stratification (The Volume Explosion):
Because the judge ignores volume, the AI tries to "cheat" by making some tokens incredibly loud (high norm) and others quiet, hoping the loud ones will stand out.
- Analogy: Since the judge can't hear the difference in volume, the actors start screaming at the top of their lungs just to be noticed, while others whisper.

The Great Discovery: It's Not Just "Attention"

For a long time, scientists thought these problems only happened in Transformers (the specific type of AI that uses "Attention"). They thought it was a flaw in the "Attention" mechanism itself.

The paper proves this is wrong.

The authors tested this "blind judge" theory on four different types of AI architectures that do not use standard Attention:

Graph Attention: Routing information between nodes in a network.
Mamba: A model that uses "state spaces" (like a memory buffer) instead of attention.
RWKV: A model that mixes information over time like a recurrence.
Attention Residuals: A model that routes information based on depth (layers) rather than time.

The Result: All four of these different systems developed the exact same symptoms (the spotlight pile-up, the flattening, and the volume explosion).

The Conclusion: The problem isn't "Attention." The problem is Content-Based Routing when the measuring tool is "norm-blind." If you route information based on content but ignore the magnitude (size), the system will collapse, no matter what architecture you build.

The "Brake" Analogy: Why Some Models Collapse Faster

If the mechanism is the same, why do some models collapse immediately (like Transformers) while others take a long time (like Mamba)?

The paper introduces the concept of a "Positional Brake."

The Engine: The "Content Score" is the engine trying to push the system toward collapse (picking the best content).
The Brake: Every model has a secondary rule that says, "But wait, remember where this came from?" (e.g., "Don't forget the most recent word," or "Don't forget the first word").

How the brake works:

Strong Brake: If the brake is strong (like in Mamba or RWKV, which have strong time-decay rules), it fights the engine. The system resists collapsing for a long time. It takes a while for the "content" to win out.
Weak Brake: If the brake is weak (like in standard Transformers with "Rotary Positional Encoding"), the engine wins quickly. The system collapses early and hard.

The Experiment:
The authors took a model (RWKV) and manually adjusted the brake.

When they tightened the brake (made the time-decay stronger), the collapse happened later and weaker.
When they removed the brake (let the content run wild), the collapse happened almost instantly.

This proves that the mechanism (the collapse) is always there, but the timing depends on how strong the brake is.

The "Volume" Trick (Norm Stratification)

One of the most interesting findings is about the "Volume Explosion" (Norm Stratification).

The Theory: The AI creates loud tokens to compensate for the judge being blind to volume.
The Test: The authors looked at a model (AttnRes) where the tokens were forced to be the same volume (normalized). You would think this would stop the collapse.
The Result: The collapse still happened. The system just found a different way to cheat. It stopped using volume and started focusing purely on content hubs.

The Lesson: The "volume explosion" is just one way the AI tries to fix the broken ruler. It's a symptom, not the cause. The real cause is the blind ruler itself.

Summary: What This Means

It's a Universal Law: This isn't a bug in one specific AI model. It's a fundamental mathematical consequence of how these routing systems work. If you route based on content but ignore size, you get collapse.
Geometry is a Diagnosis, Not a Cure: The authors use geometric language (distances, manifolds) to describe why this happens, but they aren't saying the AI is doing complex geometry. They are saying the AI is struggling because its "ruler" is flat and blind.
The Brake Controls the Timing: We can't stop the collapse entirely (according to this paper), but we can control when it happens by adjusting the "positional brake" (how much the model cares about position vs. content).

In short: The paper says, "Stop blaming the specific type of AI (Transformer). The problem is the measuring tape. As long as the tape is blind to size, the system will eventually squish everything into a corner to make sense of it."

Technical Summary: "All Routes Lead to Collapse"

Problem Statement
Trained transformer models exhibit three distinct, often-cited pathologies: attention sinks (a few tokens absorbing disproportionate attention mass), rank collapse (hidden representations drifting toward a low-rank subspace with depth), and norm stratification (key and query norms spreading out rather than remaining uniform). These phenomena are typically treated as specific failures of the attention mechanism, diagnosed and patched on a model-by-model basis. This paper argues that these are not pathologies of attention per se, but rather the inevitable behavior of content-based routing operating under a fixed, norm-blind similarity metric. The authors posit that when a router's metric is ill-matched to the representations it processes, the system compensates by concentrating routing weights and collapsing representations.

Methodology and Reframing Identity
The core theoretical contribution is a reframing identity that reinterprets softmax attention. The authors demonstrate that softmax attention is mathematically equivalent to Boltzmann-weighted aggregation over Euclidean distances, provided the keys lie on a hypersphere (constant norm).

The Identity: The standard dot-product score $\langle q, k \rangle$ is derived from a squared-distance metric $D(q, k)^2 = \|q\|^2 - 2\langle q, k \rangle + \|k\|^2$ . The softmax normalization cancels the $\|q\|^2$ term, and the assumption of constant key norms ( $\|k\| = c$ ) cancels the $\|k\|^2$ term.
The Blindness: Consequently, the standard attention score omits the $-\|k\|^2$ term, rendering the router blind to key magnitude.
The Prediction: If the learned representations do not satisfy the constant-norm assumption (i.e., if key norms vary), the router must compensate for this metric inadequacy. The predicted compensation signature is: (1) concentration of routing weights onto a few sources, (2) collapse of routed representations into a low-rank subspace, and (3) stratification of key norms.

Experimental Scope
The authors test this hypothesis across five distinct routing architectures to determine if the signature is universal to content-based routing or specific to transformers:

Standard Transformers: Nine pretrained models (GPT-2 variants and Pythia variants) routing over tokens.
Graph Attention (GAT): Routing over nodes in heterophilic graphs.
Selective State-Space Models (Mamba): Routing over time via a data-controlled operator without explicit attention.
Recurrent Mixers (RWKV): Routing over time with a decayed softmax.
Attention Residuals (AttnRes): Routing over depth (sublayers) rather than tokens.

Key Results

Violation of Assumptions: In all nine transformer models, key norms are not uniform (violating the constant-norm assumption), and keys occupy a low-rank subspace significantly lower than random baselines (Gaussian or shuffled).
Universal Signature: The "compensation signature" (concentration, collapse, and norm stratification) appears in all five architectures, including non-transformer models like Mamba and RWKV.
- Transformers show early onset (9–25% depth) and strong concentration.
- Recurrent models (Mamba, RWKV) show late onset (42–81% depth) and weaker concentration.
- AttnRes (routing over depth) also concentrates, proving the phenomenon is not specific to the token axis.
Causal Ablations:
- Mamba: Freezing the input-dependent gate ( $\Delta$ ) eliminates concentration, proving the selectivity mechanism causes the collapse, not generic recurrence.
- RWKV: Modifying the time-decay parameter (the "positional brake") shifts the onset depth and concentration strength. Killing the carry removes concentration entirely; increasing carry accelerates it.
- AttnRes: Despite using RMSNorm (which forces key norms to be constant by construction, eliminating norm stratification), the router still concentrates. This proves that norm stratification is a compensation strategy, not the cause of the collapse.
Graph Control: A comparison between Graph Attention Networks (GAT) and Graph Convolutional Networks (GCN) shows that GAT collapses more than GCN, isolating the effect to learned attention rather than generic neighborhood smoothing (oversmoothing).

The Hypothesis: Invariant Mechanism vs. Contingent Form
The paper proposes a hypothesis separating the invariant mechanism from the contingent form:

Invariant Mechanism: Content-based routing under a fixed, norm-blind metric inevitably develops a representation attractor (concentration and collapse). This holds regardless of the architecture (transformer, SSM, recurrent) or the routing axis (tokens, nodes, time, depth).
Contingent Form: The timing (onset depth), strength, and specific subspace of the collapse are determined by the "positional brake." This is the positional term (e.g., rotary phase in transformers, time-decay in RWKV, recency bias in AttnRes) that pulls routing toward recent or local sources, counteracting the content-based attractor.
- Weak brakes (e.g., transformers' rotary phases) lead to early, strong collapse.
- Strong brakes (e.g., RWKV's explicit decay) delay and weaken the collapse.

Significance and Claims
The paper does not claim that models implement Riemannian geometry; rather, the geometric view serves as a diagnostic tool to name the inadequacy of the flat, norm-blind metric used in standard routing.

Reframing: The authors shift the interpretation of attention sinks and rank collapse from "bugs" to "features" of a mismatch between a fixed metric and learned representations.
Generality: The mechanism is shown to be architecture-agnostic, appearing in models without softmax, without query-key attention, and without token-based routing.
Intervention: The paper suggests that restoring the dropped $-\|k\|^2$ term could provide a new degree of freedom to correct the metric, though this is left for future work.
Modesty: The authors explicitly state they claim sufficiency, not necessity. They do not claim that all low-rank behavior is caused by this mechanism (e.g., oversmoothing in GNNs is a different route), nor that the attractor is always a pathology (in some graph cases, the collapse without learned attention generalizes better). The hypothesis is falsifiable: a norm-blind router that does not develop this signature would refute the invariant claim.

In summary, the paper argues that "all routes lead to collapse" because the standard metric used for content-based routing is fundamentally blind to magnitude, forcing the system to compensate by concentrating its routing and collapsing its representations, a behavior modulated but not prevented by architectural positional biases.