🤖 machine learning

The Routing and Filtering Structure of Attention

This paper proposes $S$ - $D$ attention, a stable parameterization that disentangles the routing and filtering components of attention to reveal a depth-dependent spectral cascade, enabling significant model compression and linearization of early layers with minimal perplexity loss.

Original authors: Shafayeth Jamil, Rehan Kapadia

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shafayeth Jamil, Rehan Kapadia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Transformer (the AI brain behind models like GPT) as a massive, busy office building with 12 floors. Every floor has the same number of workers, the same amount of desk space, and the same amount of paperwork to do. The architects of these buildings have always assumed that every floor needs to be equally complex, so they build them all the same way.

This paper argues that this assumption is wrong. It turns out that the "workers" (the attention mechanism) are actually doing two very different jobs, but they've been forced to share the same desk, making it impossible to see who is doing what.

Here is the breakdown of what the authors discovered, using simple analogies.

1. The Two Jobs: The "Traffic Cop" and the "Filter"

In a standard AI model, every time it looks at a sentence, it calculates a giant grid of scores. The authors realized this grid is actually a messy mix of two distinct tasks:

The Traffic Cop (Routing): This job is about direction. It decides, "Token A should send information to Token B, but not the other way around." It's like a one-way street. It moves information around without changing its volume.
The Filter (Filtering): This job is about relevance. It decides, "Token A and Token B are very similar, so let's boost their connection," or "They are unrelated, so let's ignore them." It's like a volume knob that turns signals up or down.

The Problem: In standard models, these two jobs are tangled together in one big, messy matrix. The "Filter" job is so loud and dominant that it drowns out the "Traffic Cop." Because they are mixed, the AI wastes a huge amount of energy trying to build a complex "Traffic Cop" system that it barely uses.

2. The Experiment: Untangling the Knot

To see what was really happening, the authors built a new type of attention called S–D Attention. Think of this as building a new office where the "Traffic Cop" and the "Filter" have separate, dedicated desks.

The Traffic Cop (S): Built to be perfectly balanced (mathematically "skew-symmetric"). It can only move information, never create or destroy it.
The Filter (D): Built as a simple list of "volume knobs" (diagonal matrix). It only scales things up or down.

By separating them, they could watch how the AI organized itself without the noise of the "Filter" dominating everything.

3. The Discovery: The "Spectral Cascade"

When they let the AI organize itself in this new, separated way, a beautiful pattern emerged, which they call a Spectral Cascade.

Imagine the 12 floors of the office again.

The Bottom Floors (Layers 0–5): These floors are incredibly simple. They only need two "Traffic Cop" directions to do their job. They are like a simple hallway where everyone just moves left or right. They don't need a complex traffic system.
The Middle Floors: The complexity slowly increases.
The Top Floors (Layers 10–11): These are the only floors that need a massive, complex traffic system with many directions.

The Big Reveal: In standard models, the AI was building a "complex traffic system" (high rank) on the bottom floors where it wasn't needed. It was over-engineering the simple parts. When they separated the jobs, the AI naturally organized itself: simple at the bottom, complex at the top.

4. The "Surgery": Cutting the Fat

Because they could now see exactly how much complexity each floor needed, they performed "surgery" on the models:

Linearizing the Bottom: They replaced the complex attention mechanism on the first 7 floors with a very simple, cheap linear math trick (like using a straight line instead of a curve).
- Result: The model's performance barely dropped (less than 5% worse).
- Standard Model: If you tried this on a normal model, it would collapse immediately. The normal model was relying on that complexity even where it wasn't needed.
The Inversion: In normal models, if you try to simplify the "Filter" (the volume knobs), the model breaks. But in their new separated model, simplifying the "Filter" to just a single number per head didn't hurt much at all. The "Traffic Cop" was the real workhorse.

5. The New Blueprint

The authors used this "Spectral Cascade" map to redesign the AI architecture. Instead of giving every floor the same amount of power, they built a custom model:

Bottom floors: Tiny, narrow, and simple.
Top floors: Wide and complex.

The Result: They created a model that is 47% to 65% smaller in terms of attention parameters but still performs almost as well as the giant, uniform models. They essentially realized that the bottom floors were carrying heavy suitcases they didn't need, and they threw those suitcases away.

Summary

The paper claims that current AI models are inefficient because they treat every layer of the brain as if it needs to be equally complex. By separating the "moving" of information from the "filtering" of information, they discovered that the early layers are actually very simple.

By building models that match this natural "simple-to-complex" structure, they can make AI models significantly smaller and cheaper to run without losing much intelligence. It's like realizing you don't need a Ferrari engine to drive to the grocery store; you only need it for the highway.

Technical Summary: The Routing and Filtering Structure of Attention

Problem Statement

Standard Transformer architectures allocate attention resources uniformly across all layers, assigning identical head dimensions and computational complexity ( $O(N^2)$ ) to every layer regardless of the specific function that layer performs. This uniformity persists because there is no principled measurement to guide non-uniform allocation. Recent hybrid architectures (e.g., Nemotron-H, Jamba, Griffin) attempt to solve this through extensive hyperparameter sweeps to find optimal ratios of quadratic to linear layers, but these ratios are scale-specific and lack transferability.

The core issue identified is that the attention interaction matrix $QK^\top$ entangles two distinct computational functions within a single unconstrained matrix:

Routing: A skew-symmetric component that directs information flow between positions (conservative transport).
Filtering: A symmetric component that scales mutual relevance between tokens (gain control).

In standard attention, these functions compete for the same matrix capacity. Empirical analysis of 1,776 heads across five pretrained models (GPT-2 variants, BERT-base, Pythia-410M) reveals that filtering dominates the spectral budget, while routing operates at a low rank well below the capacity allocated by the weight kernel. Furthermore, all standard attention heads are found to be dynamically unstable (having eigenvalues with positive real parts), relying heavily on Layer Normalization (LN) to contain this instability, which obscures the true spectral structure of the routing mechanism.

Methodology

To disentangle these functions and reveal the underlying spectral structure, the authors introduce S–D Attention, a diagnostic parameterization that separates routing and filtering by construction:

Routing ( $S$ ): Modeled as a learned skew-symmetric matrix ( $S = -S^\top$ ). This ensures eigenvalues are purely imaginary, representing conservative, zero-sum information transport (rotation between modes).
Filtering ( $D$ ): Modeled as a learned positive diagonal matrix ( $D = \text{diag}(d_1, \dots, d_N)$ ). This represents dissipative filtering where each token has a scalar damping coefficient.

The interaction matrix is defined as $L = S - D$ . The authors prove that for any real skew-symmetric $S$ and positive semi-definite diagonal $D$ , the real part of every eigenvalue satisfies $\text{Re}(\lambda) \le 0$ . This guarantees architectural stability without the need for Layer Normalization.

The authors train S–D models at various scales (7M to 355M parameters) on OpenWebText, including configurations with and without Layer Normalization, to observe how routing self-organizes when not constrained by entanglement or normalization-induced equalization.

Key Findings and Results

1. The Spectral Cascade

When routing and filtering are disentangled and normalization is removed, routing self-organizes into a spectral cascade (a rank hierarchy) across depth:

Layer 0: Converges to an effective rank of 2 (a single rotation plane for broad information redistribution).
Intermediate Layers: Maintain a low-rank plateau (effective rank $\approx 2$ –$3$).
Terminal Layers: Expand to higher ranks (up to 10+ independent rotation planes), with the final rank determined by the head dimension rather than depth.
Scale Invariance: This ascending cascade pattern holds across all tested scales (7M to 355M). As model size increases, the fraction of layers operating at minimum spectral rank grows (from 33% at 18M parameters to 92% at 355M).

2. Filtering Collapse

In the disentangled S–D architecture, the filtering component collapses to a single scalar per head. Truncating the filtering component to a per-head mean scalar incurs a negligible perplexity cost (+0.3 PPL), whereas removing routing entirely causes catastrophic failure. This contrasts with standard attention, where filtering dominates and routing is low-rank.

3. Actionable Spectral Surgery

The spectral cascade allows for "surgical" simplification of the architecture:

Linearization: The first 7 layers of a 125M S–D model can be linearized (replaced with ELU+1 causal linear attention) with less than 5% perplexity degradation. In contrast, standard attention collapses under the same intervention by layer 3.
Parameter Reallocation: By matching head dimensions ( $d_{head}$ ) to the measured spectral rank (e.g., $d_{head}=8$ for early layers, $d_{head}=64$ for terminal layers), the authors reduced attention parameters by 47–65% with only a +3.9% to +8.4% increase in perplexity.
Inversion: The paper demonstrates a structural inversion: in standard attention, filtering truncation is prohibitive and routing truncation is cheap; in S–D, routing truncation is prohibitive and filtering truncation is free.

Significance and Claims

The paper claims its primary contribution is measurement, not the architecture itself. The significance lies in making the "spectral budget" of attention legible and actionable:

Diagnosis: The S–D parameterization serves as a diagnostic tool to measure the true routing capacity required at each layer, revealing that standard attention allocates excessive capacity to low-rank layers.
Guidance for Hybrid Architectures: The spectral cascade provides a continuous, measurable metric to determine which layers require full quadratic attention and which can be replaced with linear or low-rank mechanisms. This moves hybrid architecture design from "search" (sweeping ratios) to "measurement" (prescribing based on rank).
Re-evaluation of Linear Attention: The results suggest that the historical failure of simple linear attention (e.g., ELU+1) was due to misallocation (applying it to high-rank layers) rather than intrinsic capacity limits. The paper shows that linear attention works effectively in the low-rank "plateau" identified by the cascade.

The authors conclude that the spectral cascade is the optimizer's natural solution when entanglement and normalization are removed, and that this structure can guide the construction of more efficient, non-uniform transformer architectures.