💬 NLP

Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat

This paper demonstrates that end-to-end sparse attention training suffers from "routing absorption," where the model's Q/K/V projections co-adapt to any imposed mask, rendering learned gating mechanisms ineffective compared to random gates and suggesting that post-hoc sparsification is a superior alternative.

Original authors: Keston Aquino-Michaels

Published 2026-03-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Keston Aquino-Michaels

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why "Smart" Gates Fail in AI

Imagine you are building a super-smart library (the AI model) that can read millions of books. To make it faster, you want to install a Librarian (the "gate") whose only job is to decide which books to pull off the shelf for a specific question and which ones to ignore.

The Goal: You want the Librarian to learn during the training process which books are important, so the library becomes super-efficient.

The Reality: The paper argues that when you train the Librarian and the Library together, the Library learns to ignore the Librarian completely. The Library becomes so good at adapting to any rule the Librarian makes that the Librarian's actual decisions stop mattering.

This phenomenon is called Routing Absorption.

The Core Problem: The "Over-Adapting" Library

The paper uses a 31-million-parameter AI model (a medium-sized brain) to prove this. Here is what happened in their experiments:

The Setup: They gave the AI a "gate" (a small network) to decide which parts of the attention (the "reading") to keep.
The Expectation: They thought the gate would learn to pick the best 10% of the data, and the AI would get faster without losing intelligence.
The Result: The gate learned almost nothing. Whether the gate was a "smart" learned gate or a "dumb" random gate (flipping a coin), the AI performed almost exactly the same.

Why? Because the AI (the Library) is 80 times bigger than the gate (the Librarian).

The Analogy: Imagine a massive, flexible clay sculpture (the AI) and a tiny, rigid stamp (the gate). If you press the stamp into the clay, the clay doesn't just take the shape of the stamp; the clay squishes and reshapes itself around the stamp to make the final shape look perfect anyway.
The AI's internal parts (called Q/K/V projections) are so flexible that they co-adapt to whatever the gate does. If the gate says "ignore these words," the AI just learns to find the meaning in the other words. If the gate is random, the AI just learns to work with the random noise.

The gate's signal gets absorbed by the massive AI, rendering the gate useless.

The Four Proofs (The "Smoking Guns")

The authors ran four specific tests to prove this isn't just a training error, but a fundamental law of how these models work.

1. The "Smart vs. Dumb" Test

Experiment: They trained one AI with a "smart" gate and another with a "random" gate.
Result: Both ended up with the same bad performance. The smart gate didn't get any better than the coin-flip gate.
Takeaway: The AI didn't need the gate to be smart; it just needed a gate to exist so it could adapt to it.

2. The "Zero Signal" Test

Experiment: They tried to make the gate "hard" (making a strict yes/no decision). In math terms, this gives the gate zero "gradients" (no feedback on how to improve).
Result: The gate got zero better.
Takeaway: Even when the gate could learn (in the soft version), it barely improved. When it couldn't learn (hard version), it was just as bad. The bottleneck isn't the math; it's that the AI absorbs the signal too fast.

3. The "Distillation" Test (The Most Important One)

Experiment: They took an AI that was already trained without any gate (a "dense" model). Then, they froze the AI and only trained the gate.
Result: The gate became a genius! It learned to pick the perfect books in just 1,000 steps.
Contrast: When they tried to train the gate while the AI was learning, the gate failed.
Takeaway: The gate can learn, but only if the AI stays still. If the AI moves while the gate tries to learn, the AI changes its shape to hide the problem, and the gate never figures it out.

4. The "Random Noise" Test

Experiment: They tried to prevent the AI from adapting by randomly changing the gate's rules every single second (like "dropout" in other AI methods).
Result: The AI got terrible at its job.
Takeaway: Instead of making the AI robust, the random noise broke the AI's internal structure. The AI learned to "flatten" its attention to survive the chaos, losing all its sharp, smart focus.

Why Does This Happen? (The Size Difference)

The paper compares this to Mixture-of-Experts (MoE) models, where a router sends data to different "expert" sub-networks. In MoE, random routing often works as well as smart routing because the experts are self-contained.

But in Attention (the focus of this paper), the problem is worse.

MoE: Experts are like separate rooms. If the router sends you to the wrong room, that room has to figure it out.
Attention: The "experts" (the Q/K/V parts) are shared across the whole building. If the router messes up, the whole building can shift its weight to compensate. Because the AI has 80x more "muscle" (parameters) than the gate, it always wins the tug-of-war.

The Solution: Don't Train Them Together

The paper concludes that you cannot train the gate and the AI together.

The Wrong Way: Train the gate and the AI at the same time. The AI will absorb the gate's signal, and the gate will learn nothing.
The Right Way (Post-Hoc):
1. Train the AI to be perfect (Dense training).
2. Freeze the AI (stop it from changing).
3. Then train the gate to figure out how to slice the AI up.

Because the AI is frozen, it can't "absorb" the gate. The gate has to actually learn the structure, and it does so very quickly and effectively.

Summary in One Sentence

Trying to teach an AI to be efficient by training a "gate" alongside it is like trying to teach a tiny conductor to lead a massive orchestra; the orchestra is so big and flexible that it will just play whatever the conductor tries to do, making the conductor's actual choices irrelevant. To get a good conductor, you must first let the orchestra learn its music, and then teach the conductor how to cut the performance short.

1. Problem Statement

The paper addresses a critical failure mode in end-to-end sparse attention training. Theoretically, attention mechanisms in Transformers are highly concentrated (a small subset of keys captures most of the attention mass), suggesting that a lightweight learned gate should easily identify and retain the most important tokens.

However, in practice, when sparse attention is trained jointly with the model (end-to-end), the learned gating mechanism fails to outperform frozen random gates. The paper posits that this is not due to poor gate architecture or insufficient training, but rather a phenomenon called Routing Absorption.

Core Hypothesis: The model's massive parameter count (specifically the Query/Key/Value projections) co-adapts to compensate for whatever mask is imposed. The gate's signal is "absorbed" into the Q/K/V representations, rendering the specific routing decisions of the gate irrelevant. Consequently, the model learns to function well with any mask, including random ones, rather than learning to select specific important tokens.

2. Methodology & Experimental Setup

The authors conducted a series of controlled experiments using a 31M-parameter Transformer (6 layers, 256-dim, 4 heads) trained on WikiText-103, with preliminary validation on Qwen3-1.7B (55x larger scale).

Gate Architecture: A lightweight bilinear gate ( $d_{gate}=32$ ) adding ~393K parameters (1.3% of the model).
Training Regimes:
- End-to-End: Gate and Q/K/V projections trained jointly.
- Post-hoc: Q/K/V frozen; only the gate is trained (distillation).
- Gate-Only: Q/K/V frozen; gate trained to convergence.
Key Comparisons: Learned gates vs. Frozen Random gates vs. Oracle masks vs. Stochastic masking.

3. Key Contributions & Evidence

The paper presents four independent lines of evidence demonstrating routing absorption:

A. Learned Gates ≈ Random Gates (Soft Gating)

Experiment: Trained the model end-to-end with differentiable soft gating for 50,000 steps.
Result: The learned gate achieved a perplexity of 48.73, while a frozen random gate achieved 49.83. The gap was only 2.2%.
Implication: Despite 50,000 steps of gradient updates, the gate learned almost nothing. The Q/K/V projections adapted so effectively to the presence of a mask that the content of the mask became irrelevant. The gate captured only 9% of the potential improvement over random routing.

B. Zero Gradient in Hard Top-K

Experiment: Used hard top-k gating (non-differentiable).
Result: Learned gates and random gates performed identically (71.22 vs. 71.24 PPL).
Implication: While hard gating receives zero gradient (a known issue), the fact that soft gating (which does receive gradients) also fails to outperform random suggests the bottleneck is not gradient flow, but the absorption of the signal by the co-adapting Q/K/V layers.

C. The Distillation Contrast (Co-adaptation Proof)

Experiment: Trained a gate on two different checkpoints:
1. Dense-trained Q/K/V: Never saw a mask.
2. Co-adapted Q/K/V: Trained end-to-end with soft gating.
Result:
- On Dense Q/K/V, the gate achieved high F1 and low perplexity (48.6).
- On Co-adapted Q/K/V, the same gate achieved high F1 (predicting the "right" tokens) but catastrophic perplexity (601.6).
Implication: The co-adapted Q/K/V representations became dependent on the functional form of the sigmoid mask, not just the selected tokens. Switching to a hard mask (even with the correct tokens) broke the model. This proves the representations are "mask-agnostic" only when the model is not co-adapted.

D. Stochastic Masking Fails

Experiment: Trained with random mask dropout (randomizing the mask every forward pass) to prevent co-adaptation.
Result: The model degraded significantly (78.2 PPL even without a mask at deployment).
Implication: Unlike feature dropout, randomizing attention masks destroys the attention structure rather than regularizing it. The model flattened its attention distributions to survive the noise, sacrificing the concentrated signal required for sparse attention.

4. Mechanism: Parameter Asymmetry

The paper attributes routing absorption to Parameter Asymmetry:

The Ratio: The model has ~31M parameters, while the gate has ~393K (an 80:1 ratio).
The Dynamic: The massive compute substrate (Q/K/V) has 80x more degrees of freedom to adjust and compensate for the gate's limitations than the gate has to impose a specific structure.
Scale Effect: Experiments on Qwen3-1.7B confirmed this intensifies at scale. Even when only a single layer's Q/K/V were unfrozen, the learned gate provided zero benefit over a random gate. As more layers were unfrozen, the gap between learned and random gates shrank from 31.6 PPL (post-hoc) to 6.9 PPL, demonstrating that co-adaptation capacity drives absorption.

5. Significance & Implications

For Sparse Attention Methods

End-to-End Training is Flawed: Methods relying on learned, per-query token-level gating (e.g., recent sparse attention papers) are likely suffering from routing absorption. The "learned" routing may contribute less than claimed because the model compensates for it.
Recommendation: Sparse attention should be treated as a post-training compression step (post-hoc distillation), not an end-to-end objective. Decoupling the gate from the representation learning prevents absorption.
Ablation Check: Any method claiming learned routing benefits should be ablated against random routing. If random routing performs similarly, absorption is occurring.

For Mixture-of-Experts (MoE)

Analogy: This phenomenon mirrors routing absorption in MoE (where experts co-adapt to random routers).
Distinction: Attention absorption is structurally more severe. In MoE, experts are self-contained modules. In Attention, shared Q/K/V parameters allow for cross-layer compensation, making the absorption effect more robust and harder to break.

Theoretical Insight

Decoupling Principle: When a small auxiliary network (router) controls a much larger compute substrate, they must be decoupled. If trained jointly, the substrate will absorb the router's signal, making the router's specific decisions irrelevant.

Conclusion

The paper concludes that Routing Absorption is the primary reason end-to-end sparse attention fails. The model's Q/K/V projections co-adapt to any imposed mask, rendering learned gates no better than random ones. The solution is decoupling: train the dense model first, then distill a sparse gate onto the frozen representations. This approach achieves near-oracle performance because it prevents the co-adaptation that causes absorption.