💬 NLP

What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation

This paper presents a comparative study of seven linearized language model architectures, revealing that architectural inductive biases—particularly error-correcting update rules and gated delta formulations—are the primary determinant of performance and long-context capability, as these advantages established early in training persist regardless of parameter scaling or token budgets.

Original authors: Patrick Haller, Jonas Golde, Alan Akbik

Published 2026-02-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Patrick Haller, Jonas Golde, Alan Akbik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class chef (the Transformer) who can cook complex meals using a massive, slow, but incredibly precise kitchen setup. This chef uses a special technique where they look at every single ingredient in the pantry at once to decide what to add next. It's perfect, but it's slow and requires a huge kitchen.

Now, imagine you want to hire a new, faster apprentice (the Linearized Model) who can cook just as well but uses a tiny, efficient kitchen. To do this, you don't teach the apprentice from scratch. Instead, you try to distill the master chef's knowledge into the apprentice's brain. You tell them, "Don't look at the whole pantry; just keep a running list of the most important things you've seen so far."

This paper is a big experiment to see which type of apprentice works best when you try to copy the master chef's skills into this faster, smaller kitchen.

The Big Question

There are many different ways to build this "running list" (called a token mixer). Some apprentices just keep adding new notes to a long scroll (Additive). Some use a gate to decide what to keep and what to throw away (Gated). Others use a "delete and replace" rule where new info overwrites old info if they match (Delta-rule).

The researchers wanted to know: Does it matter which "note-taking style" the apprentice uses? Or can we just make the apprentice bigger and smarter to fix any bad habits?

The Experiment

The researchers took seven different types of apprentices (architectures like xLSTM, Gated DeltaNet, GLA, etc.) and tried to teach them using the same master chef, the same recipes (data), and the same amount of practice time. They tested them at three sizes:

Small (140 million parameters)
Medium (360 million parameters)
Large (1.7 billion parameters)

They also tested them on two other challenges:

The "Needle in a Haystack" Test: Can the apprentice find a specific fact hidden in a very long story?
The "Follow Instructions" Test: Can the apprentice understand and follow complex commands?

What They Found

1. The "Note-Taking Style" Matters More Than Size

The most surprising finding is that the type of apprentice matters more than how big they are.

The Winners: Apprentices that use "Gated Delta-rules" (think of them as having a smart eraser that can precisely delete old, irrelevant notes and replace them with new ones) were the best. They stayed ahead of the pack even as they got bigger.
The Losers: Apprentices that just keep adding notes without deleting anything (Linear Attention) got stuck. No matter how big they grew, they couldn't catch up to the smart ones. They were like someone trying to remember a 10-hour movie by writing every single word on a piece of paper that never gets erased—the paper eventually gets so full of junk that they can't find the important parts.

The Analogy: It's like trying to learn a language. One student writes every word they hear in a notebook (Additive). Another student has a notebook where they can cross out old words and write new ones only when necessary (Gated Delta). The second student learns faster and remembers better, regardless of how many years they study.

2. You Can't Fix Bad Habits with More Data

The researchers wondered: "If we just give the 'bad' apprentices more data (more practice tokens), will they eventually catch up?"

The Answer: No.
Even after training them on a massive amount of data (10 billion tokens), the performance gap remained. The "bad" note-taking styles hit a ceiling. The "good" styles kept improving slightly but mostly stayed ahead.
The Lesson: You can't fix a broken memory system just by feeding it more information. The structure of the memory is the bottleneck.

3. The "Long Memory" Problem

When the story got very long (thousands of words), most apprentices failed the "Needle in a Haystack" test. They forgot the beginning of the story.

The Exception: Only the Gated DeltaNet (the one with the smart eraser) could still find the needle in the haystack.
The Failure: The "Additive" models (the ones that just keep adding notes) completely forgot everything beyond a certain point. Their memory became "saturated" with noise.

4. Teaching Instructions Doesn't Fix the Core Flaw

Finally, they tried to teach these apprentices how to follow instructions (like "write a poem" or "solve a math problem").

The Result: The smart apprentices got even better at following instructions. The weak ones got slightly better, but they were still weak.
The Twist: Trying to teach them instructions during the copying process (distillation) didn't help the weak ones catch up. In fact, it sometimes made their long-term memory even worse.
The Takeaway: You can't teach a student to be a genius at long-term memory if their brain isn't built for it. The architecture (the brain structure) is the primary constraint.

The Bottom Line

If you want to build a fast, efficient AI that can still do complex tasks and remember long stories, you can't just scale up a bad design.

The paper concludes that the way the model updates its memory (specifically, using rules that can selectively delete and overwrite old information) is the most important factor. If you pick the wrong "note-taking style," no amount of training data or model size will make it as good as the right one.

In short: It's not about how big your brain is; it's about how smartly you organize your notes.

Technical Summary: What Matters in Linearizing Language Models?

Problem Statement

Linearization has emerged as a primary strategy for developing efficient language models (LMs) by replacing the $O(L^2)$ self-attention mechanism of Transformers with subquadratic token mixers (e.g., linear attention, gated recurrences). While this approach aims to retain the performance of large-scale Transformers while offering linear-time inference and constant memory overhead, significant gaps remain in understanding the field.

Current literature suffers from heterogeneous evaluation conditions, varying in model size, training budgets, and distillation objectives. Consequently, it is unclear which specific inductive biases (such as gating, decay, or delta-rule updates) are best suited to inherit a Transformer's capabilities. Furthermore, the scaling behavior of these linearized models regarding parameter count and token budget, as well as their viability for downstream tasks like instruction tuning and long-context retrieval, remains an open area of inquiry.

Methodology

To address these gaps, the authors propose a unified empirical framework to compare seven representative subquadratic architectures: xLSTM, GLA, Gated DeltaNet, DeltaNet, RetNet, Kimi (KDA), and Linear Attention.

Experimental Setup

The study isolates the token mixer as the primary variable by employing a three-stage distillation pipeline that aligns student models with a pretrained Transformer teacher (SmolLM2 family) while holding the teacher, data distribution, and training recipe constant.

Token-mixer (Attention-Output) Alignment: The student's token mixer is aligned with the teacher's attention output using Mean Squared Error (MSE) loss. This provides a dense supervision signal across the entire mixing pathway.
Hidden-State Alignment: Intermediate hidden representations of the teacher and student are aligned via MSE loss.
End-to-End Knowledge Distillation: The student is trained end-to-end using standard knowledge distillation (Next-Token Prediction + KL Divergence).

Evaluation Dimensions

The study evaluates the models across three distinct experimental axes:

Scaling Laws: Models are trained at three parameter scales (140M, 360M, and 1.7B) on a fixed budget of 3B tokens to observe performance hierarchies.
Asymptotic Maturity: Models are fixed at ~1.7B parameters and trained on an extended budget of 10B tokens to determine if performance gaps are due to undertraining or fundamental architectural limits.
Downstream Adaptation: The distilled students undergo instruction tuning (Post-Training SFT) and instruction-aware distillation to assess their ability to follow instructions and perform long-context retrieval (Needle-In-A-Haystack tasks).

Key Contributions and Results

1. Stability of Performance Hierarchies and Scaling Exponents

The study finds that performance hierarchies among architectures remain remarkably stable across scales (140M to 1.7B).

Gated Architectures: Models with gating mechanisms (xLSTM, Gated DeltaNet) consistently outperform ungated baselines (Linear Attention, RetNet).
The Crossover Point: A critical finding is a "crossover" at the 1.7B scale. While xLSTM leads at smaller scales, Gated DeltaNet emerges as the top performer at 1.7B (94.2% recovery of teacher performance). This suggests that error-correcting update rules (Delta-rule) possess superior scaling exponents compared to purely additive alternatives as model capacity increases.
Intrinsic Deficiencies: The performance gap between Linear Attention and gated models widens with scale (from ~14% at 140M to ~20% at 1.7B), indicating that the deficiencies of additive linear attention are intrinsic to the architecture and cannot be overcome merely by increasing parameter count.

2. State Resolution as a Fundamental Bottleneck

Extending training to 10B tokens reveals that relative performance gaps are established early and persist even as models approach asymptotic limits.

Maturity: Performance plateaus for gated models after 6–7B tokens, while ungated models saturate earlier at lower levels. This confirms that the gaps are not due to undertraining but represent fundamental limits in how different mixers compress information.
Retrieval Capabilities: In Needle-In-A-Haystack (NIAH) tasks, most architectures suffer from a collapse in state resolution as sequence length increases. Linear Attention and RetNet fail almost entirely beyond 512 tokens.
Superiority of Gated Delta-Rule: Gated DeltaNet is the only architecture to maintain significant retrieval precision at extended contexts (e.g., 81.8% accuracy at 2k tokens), demonstrating that the gated delta-rule provides a more effective mechanism for targeted state updates and preventing state saturation.

3. Instruction Adaptation and Distillation

The study evaluates whether instruction tuning can compensate for architectural weaknesses.

Amplification of Differences: Post-training instruction tuning improves performance across all architectures but amplifies existing differences. Stronger models (Gated DeltaNet, xLSTM) benefit substantially, while weaker models show limited gains.
Distillation Limitations: Instruction-aware distillation (training on an instruction-tuned teacher) does not provide a consistent advantage over standard post-training tuning and, in some cases, degrades long-context retrieval performance.
Conclusion on Adaptation: Instruction adaptation does not alter the fundamental architectural constraints imposed during linearization. The ability to selectively retain and overwrite information remains the dominant factor governing downstream performance.

Significance and Claims

The paper argues that for successful linearization, architectural inductive biases are the primary constraint that cannot be overcome simply by scaling training compute or increasing the distillation token budget.

Design Hierarchy: The study establishes a clear hierarchy for subquadratic design, identifying that error-correcting update rules (Delta-rule) combined with gating (specifically head-wise or scalar gating) are more decisive for success than additive accumulation or channel-wise gating.
State Resolution: The research posits that state resolution is a more fundamental bottleneck than the distillation budget. Models that cannot manage their internal state precision (avoiding saturation) fail to scale effectively or handle long contexts, regardless of training duration.
Reproducibility: To facilitate further research, the authors publicly release all base, instruction-tuned, and instruction-distilled models, ensuring the reproducibility of these findings.

The authors conclude that while subquadratic models can be trained stably, the choice of token mixer dictates the ceiling of performance, with gated delta-rule formulations currently offering the most robust path toward efficient, long-context capable language models.