🤖 machine learning

Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

This paper presents a comparative study of softmax and four linear attention architectures (DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2) using 350M-parameter models to analyze their mechanisms, performance trade-offs, and the efficacy of a proposed Cross-Layer Value Routing mechanism, finding that Kimi Delta Attention with Muon achieves the lowest validation loss while Gated DeltaNet with AdamW offers the highest throughput.

Original authors: Tommaso Cerruti, Tim Rieder, George Rowlands, Lingfeng Jin, Imanol Schlag

Published 2026-07-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tommaso Cerruti, Tim Rieder, George Rowlands, Lingfeng Jin, Imanol Schlag

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive library where every new book (a "token") needs to check its notes against every single book that came before it. This is how standard AI models work, and it's brilliant but exhausting. As the library grows, the time it takes to check every pair of books explodes, making it incredibly slow and expensive to train.

This paper is like a detective story about finding a smarter, faster way to run that library without losing the ability to remember important details. The authors compared the old, slow way (Softmax Attention) against four new, speedy "linear attention" methods: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2.

The Big Idea: The "Error-Correcting" Notebook

Instead of writing the whole new book into the library's memory every time, these new methods use a clever trick called a delta rule. Imagine you have a notebook that already predicts what the next book will say. Instead of rewriting the whole page, you only write down the difference (the error) between what you predicted and what actually happened. This keeps the memory clean and prevents old notes from clashing with new ones.

The paper then asks: How do we make this notebook even better?

DeltaNet just writes the difference.
Gated DeltaNet adds a "forget button" (a scalar gate) to wipe out old, dusty notes.
Kimi Delta Attention upgrades this to a "channel-wise" forget button, letting the model forget specific types of notes while keeping others.
Gated DeltaNet-2 goes even further, separating the "erase" button from the "write" button, giving the model total control over what to delete and what to keep.

The Race: Speed vs. Smarts

The authors ran a massive experiment with models containing 350 million parameters, training them on 15 billion tokens (a huge amount of text). They didn't just look at who was smartest; they looked at who was fastest.

Here is what they found:

The Speed Champion: A "pure" stack of Gated DeltaNet trained with the AdamW optimizer was the fastest. It processed data so efficiently that it became the baseline for speed (100%). However, it wasn't the smartest; it had a higher "validation loss" (a score where lower is better) of 2.433.
The Smartest Contender: The title for the lowest loss (the smartest model) went to Kimi Delta Attention using a different optimizer called Muon in a "hybrid" stack (mixing the fast linear layers with the slow standard layers). It reached a loss of 2.273.
The Trade-off: The paper shows a clear tug-of-war. If you want the absolute best performance, you often have to mix in the slower, standard attention layers (hybrid stacks), which slows you down. If you want pure speed, you stick to the linear layers, but you might lose a bit of accuracy.

Crucially, the paper rules out one big idea: They found that learning rate (how fast the model learns) is just as important as the architecture itself. A model might look bad not because its design is wrong, but because it was given the wrong learning rate. For example, Muon prefers a lower learning rate (around 3 × 10⁻⁴), while AdamW likes a higher one (around 10⁻³). You can't just swap optimizers and expect the same results; you have to tune them together.

The New Trick: Sharing Secrets Across Layers

Deep neural networks have a problem: as information travels from the bottom layer to the top, it can get "diluted" or lost. The authors tried to fix this by creating a "secret tunnel" between layers, letting lower layers pass a message to higher ones.

They tested two main ideas for what to pass through this tunnel:

The "Mistake" (Error): Pass the error (the difference between what was predicted and what happened).
The "Goal" (Value): Pass the target value (what the model was trying to write).

The Surprise: The paper explicitly argues against the idea that passing the "mistake" is the best way. When they tried passing the error directly into the next layer's target (a method they called CLER), it didn't help at all. It was like trying to fix a leaky pipe by shouting the sound of the leak into the next room; it didn't work.

The Solution: They found that passing the target value (the "Goal") worked slightly better. They called this Cross-Layer Value Routing (CLVR).

In their 350M-parameter experiments, this new method lowered the final validation loss by a tiny amount (about 0.01).
For example, in the Gated DeltaNet model, the loss dropped from 2.8331 to 2.8228 (a difference of -0.0103).
In the DeltaNet model, it dropped from 2.8469 to 2.8350 (a difference of -0.0119).

How sure are they? The authors are careful. They note these results come from single runs (not averaged over many tries), so the gains are small and could be influenced by random chance. However, the pattern was consistent: passing the value was always slightly better than passing the error. They also tested this on a larger 1.3 billion parameter model, where the gain was even smaller (-0.0019), suggesting the benefit might shrink as models get bigger and smarter on their own.

What About the Future?

The paper doesn't claim to have solved everything. They explicitly state that they did not test these routing tricks on the other architectures like Kimi Delta Attention or Gated DeltaNet-2, so we don't know if the "value routing" trick works there yet. They also didn't test how fast these models are at reading (inference), only how fast they are at learning (training).

The Takeaway

This paper doesn't declare one "winner." Instead, it maps out a landscape.

If you need speed and long context, pure linear stacks are your friend.
If you need maximum accuracy and can afford a bit of slowness, hybrid stacks with Kimi Delta Attention and Muon look promising.
If you want to connect layers, don't pass the errors; pass the values.

The authors suggest that while these routing tricks offer a small boost, the real magic lies in understanding how the optimizer, the learning rate, and the architecture all dance together. It's not just about building a faster engine; it's about tuning the whole car.

Technical Summary: Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

Problem Statement
Standard softmax self-attention enables each token to retrieve information from the full context, a mechanism central to the expressivity of Transformer language models. However, this token-to-token exchange incurs a quadratic cost in sequence length ( $O(T^2)$ ) for both training and inference, becoming a dominant bottleneck as context windows expand. While linear attention architectures offer a solution by reformulating attention as a recurrent update over a constant-size memory matrix (yielding linear training time and constant inference time per token), early variants suffered from significant accuracy losses due to interference between stored associations and a lack of mechanisms for forgetting or selective editing. Recent developments, including DeltaNet and its successors, have narrowed this gap by introducing error-correcting delta rules and fine-grained control gates, yet the specific trade-offs between these mechanisms, their scaling behaviors, and the potential for lightweight cross-layer information sharing remain to be systematically mapped.

Methodology and Framework
The authors present a comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2.

Unified Notation: The paper establishes a common recurrent-memory notation to express these mechanisms. This framework isolates the roles of the recurrent memory matrix $W$ , the delta-rule residual $r$ (the error between the current value and the memory's prediction), and the various decay, erase, and write control mechanisms.
- DeltaNet: Replaces naive additive updates with an error-correcting delta rule, writing only the residual $r = v - \bar{v}$ to memory.
- Gated DeltaNet: Adds a learned scalar decay gate $\alpha$ to the memory state before computing the residual, introducing a coarse forgetting mechanism.
- Kimi Delta Attention: Refines forgetting by using a channel-wise decay vector $\alpha$ , allowing different feature dimensions to decay at different rates.
- Gated DeltaNet-2: Further decouples the update by introducing separate channel-wise erase and write gates, allowing independent control over which key channels are removed and which value channels are stored.
Experimental Setup: The study utilizes 350M-parameter decoder-only models trained on 15B tokens (FineWeb-Edu). The experiments compare validation loss, training throughput, optimizer sensitivity (AdamW vs. Muon), and sequence-length scaling. The authors also conduct larger-scale runs (1.3B and 3B parameters) and evaluate downstream tasks (HellaSwag, PIQA, WinoGrande).
Cross-Layer Routing: To address information dilution in deep stacks, the paper introduces and evaluates lightweight routing mechanisms. It explores two axes: what to route (write error vs. write value) and where to inject it (the next layer's value target vs. the shared hidden stream).
- CLER (Cross-Layer Error Residuals): Forwards the delta-rule write error into the next layer's value target.
- CLER-H: Routes the write error into the shared hidden stream.
- CLVR (Cross-Layer Value Routing): Routes the layer's write value (the target before subtraction) into the shared hidden stream via a zero-initialized projection.

Key Results

Architecture and Optimizer Trade-offs: Within the 350M-parameter, 15B-token sweep, Kimi Delta Attention trained with the Muon optimizer in a hybrid stack (interleaved with softmax layers) achieves the lowest final validation loss (2.273). However, pure Gated DeltaNet stacks trained with AdamW demonstrate the highest normalized training throughput (set to 100% baseline). Hybrid stacks generally improve loss at the cost of some throughput, while pure stacks preserve the strongest long-context iteration-time scaling.
Optimizer Sensitivity: The choice of optimizer is inseparable from learning rate tuning. Muon prefers lower learning rates ( ~~$3 \times 10^{-4}$ ) for linear attention variants, whereas AdamW variants prefer higher rates (~~ $10^{-3}$ ). Muon consistently lowers final validation loss relative to AdamW across matched architecture settings.
Scaling Behavior: Pure recurrent stacks exhibit superior scaling with sequence length. At 32k tokens, pure Gated DeltaNet shows a minimal iteration time increase (~1.1x from 4k), whereas softmax attention increases by ~2.9x. Hybrid stacks fall in between, with iteration time increases roughly proportional to their softmax layer density.
Cross-Layer Routing Findings:
- CLER (Error to Value Target): Forwarding the delta-rule write error into the next layer's value target does not improve performance over matched baselines. The authors attribute this to a "space mismatch," where the routed residual competes with the receiver's own learned value geometry.
- CLVR (Value to Hidden Stream): Routing the write value into the shared hidden stream yields a modest but consistent improvement in final validation loss for both DeltaNet and Gated DeltaNet. This suggests that the useful cross-layer signal is the layer's write target, not the delta-rule correction.
- Diminishing Returns: The gains from CLVR are small (reducing loss by ~0.01 at 350M/1B tokens) and diminish as the model scale or training token count increases, suggesting the benefit is most pronounced when the base memory has not yet captured all necessary information.

Significance and Claims
The paper does not declare a single "best" architecture but rather aims to make the design space legible. Its primary contributions are:

A Unified Framework: Providing a common notation that clarifies how recent linear attention variants differ in expressivity, memory decay, and control granularity.
Empirical Trade-off Mapping: Demonstrating that the choice between linear and hybrid stacks, and between optimizers, involves a multi-objective trade-off between validation loss, throughput, and sequence-length scaling.
Routing Insights: Identifying that while naive error-routing (CLER) fails due to basis misalignment, routing the write value into the aligned hidden stream (CLVR) offers a lightweight, linear-time pathway for cross-layer information sharing that improves performance without breaking the recurrent structure.

The authors emphasize that their conclusions are based on single-run comparisons and specific token budgets, urging caution in generalizing small validation loss differences without repeated-seed verification. The work serves as a foundation for understanding the specific mechanisms driving performance in linear attention and highlights open questions regarding the generalization of routing strategies to more complex architectures like Gated DeltaNet-2.