💬 NLP

Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

This paper presents the first systematic study of massive activations in hybrid linear attention large language models, revealing two distinct, architecture-aligned morphologies—pre-attention spikes and inter-spike plateaus—that emerge early in training, persist across diverse scales and domains, and are mechanistically explained by a shared lifecycle of activation cancellation.

Original authors: Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong, Chuan-Wei Kuo

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong, Chuan-Wei Kuo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a super-smart robot that can read a whole library and remember everything. To do this, engineers use a special kind of brain circuit called a "Transformer." These circuits are amazing because they can look at every word in a sentence at once and understand how they relate. However, there's a catch: as the story gets longer, the brain circuit gets incredibly slow and hungry for memory, like trying to read a book while juggling a thousand balls. To fix this, scientists invented "Hybrid Linear Attention" models. Think of these as a team of two types of workers: some are "Super-Scanners" who can look at the whole book at once (but get tired easily), and others are "Fast-Readers" who read quickly but can only hold a small chunk of the story in their heads at a time. By mixing these two types of workers, the robot stays fast and smart. But here's the mystery: when you mix them, how does the robot's internal "thought process" actually work? Does the mixing create weird glitches, or does it create a new kind of rhythm?

This is exactly what a team of researchers set out to investigate. They looked inside these hybrid robot brains to see how they handle "Massive Activations"—which are basically giant, sudden spikes of electrical energy that happen in the robot's mind. In older robot brains, these spikes were known to happen in a stable, predictable way. But the researchers wanted to know: what happens when you start swapping in the "Fast-Readers"? They discovered that the hybrid brains don't just behave randomly; they develop a very specific, rhythmic pattern of energy spikes that is completely different from the old models.

The researchers found that these massive energy spikes follow a strict schedule. Every time the robot is about to use a "Super-Scanner" (a full attention layer), its energy level shoots up like a rocket just before the launch. They call this a Pre-Attention Spike (PAS). It's like the robot taking a deep breath and flexing its muscles right before it does something heavy. But the story gets even more interesting. When the robot has to read a long stretch of text using only the "Fast-Readers" between two "Super-Scanners," the energy doesn't just drop back to zero. Instead, it stays high, forming a flat, steady plateau of energy. They call this an Inter-Spike Plateau (ISP).

Imagine a rollercoaster. In the old models, the ride was smooth and steady. In these new hybrid models, the ride looks like a series of sharp, jagged peaks (the spikes) connected by long, high, flat bridges (the plateaus). The researchers tested this on many different types of hybrid robots, from small ones with 1.2 billion parameters to massive ones with 397 billion parameters, and they found this "spike-and-plateau" pattern everywhere. It happens whether the robot is reading math problems, coding, or writing stories.

The team also ran experiments to see how these patterns are born. They built robots from scratch and watched them learn. They saw that these spikes and plateaus appear very early in the robot's training, almost as soon as it starts reading. They also tried to "turn off" certain switches in the robot's brain to see what would happen. They found that if they put a special gate on the "Super-Scanners," the spikes got much smaller, but the pattern didn't disappear. However, if they removed the gates from the "Fast-Readers," the spikes actually got a little bigger. This suggests that the "Super-Scanners" are the main bosses organizing this energy rhythm, while the "Fast-Readers" just help carry the energy along.

So, what does all this mean? The researchers propose a simple explanation: these energy spikes are like a "write-and-cancel" dance. The robot writes a huge piece of information into its memory right before it needs to do a heavy calculation, and then it immediately cancels it out to keep things clean. When the robot has to wait a long time between calculations (the plateaus), it just delays the "cancel" part, keeping the energy high until the next big moment. As the robot uses more "Super-Scanners" and fewer "Fast-Readers," these plateaus get longer and longer until they merge back into the smooth, stable ride of the old models.

In short, this paper reveals that hybrid robot brains have a unique, rhythmic way of thinking that looks like a series of sharp jumps connected by high bridges. It's not a bug; it's a feature of how these efficient, mixed-model brains organize their energy. This discovery helps us understand how these powerful, efficient AI models actually work inside, and it suggests that the timing of when they "cancel" their thoughts is the secret sauce behind their behavior.

Technical Summary: Massive Activations in Hybrid Linear Attention Large Language Models

Problem Statement
Hybrid Linear Attention (HLA) Large Language Models (LLMs) have emerged as a solution to the quadratic computational and memory costs of full softmax attention, particularly for long-context modeling. By interleaving efficient linear attention layers with full attention layers, HLA architectures (e.g., Qwen3.5, Kimi Linear, Nemotron-H) aim to balance efficiency with modeling capacity. However, the internal activation dynamics of these hybrid models remain poorly understood. Specifically, the behavior of Massive Activations (MAs)—sparse hidden-state entries that exceed typical values by several orders of magnitude and are closely coupled with attention sinks—is largely unexplored in the context of layer-interleaved hybrids. Prior work has established MAs as a signature of full attention Transformers, but it is unclear how hybridization reshapes their emergence, evolution, and functional role.

Methodology
The authors present the first systematic study of MAs in layer-interleaved HLA LLMs through a multi-faceted approach:

Inference-Time Characterization: The study analyzes MAs across five linear attention architectures (RetNet, HGRN, GLA, DeltaNet, Gated DeltaNet/GDN), six hybridization configurations (varying the ratio of linear to full attention layers), and five data domains (general prose, scientific writing, math, code, multilingual text). This evaluation spans models from 340M to 1.3B parameters using the M-A-P model suite, as well as large-scale open-source models (Kimi Linear, Qwen3.5, Nemotron-H, Zamba2) ranging from 1.2B to 397B parameters.
Attention-Sink-Guided Tracking: To address the fragility of magnitude-based ranking in hybrids (where the maximally activated token switches frequently), the authors develop a method to track MAs using consensus attention sinks. They identify dominant sink tokens by averaging attention matrices across full attention layers and heads, then trace the maximum absolute activation of these specific tokens across model depth.
Controlled Pretraining: To isolate the emergence of MA morphologies from architectural artifacts, the authors train GDN-based hybrids from scratch (340M and 1.3B scales) under controlled conditions. They manipulate the placement of full attention layers and the presence of output gating mechanisms to observe how MAs develop during optimization.
Systematic-Outlier Analysis: The study employs a fixed-coordinate analysis to trace the signed dynamics of specific outlier features. This involves decomposing the residual stream updates to identify "write," "sink," and "cancel" events, linking the evolution of MAs to attention-sink behavior.

Key Contributions and Findings

The paper identifies two distinct, architecture-aligned morphologies of Massive Activations in HLA models:

Pre-Attention Spikes (PAS): MAs consistently intensify immediately before full attention layers. This phenomenon is observed across all linear attention backbones and hybridization ratios. Quantitative analysis shows a high "sink-spike alignment rate," where the consensus sink token reaches its maximum activation in the layer immediately preceding a full attention block.
Inter-Spike Plateaus (ISP): As the density of full attention layers increases (i.e., the hybridization ratio $\rho$ $ρ$ decreases), the activations between successive PAS do not decay to baseline but instead remain elevated, forming sustained regions termed Inter-Spike Plateaus.
- In sparse hybrids (e.g., 12:1 or 24:1), PAS are sharply localized and separated by troughs.
- In denser hybrids (e.g., 3:1), these troughs fill in, creating plateaus that connect successive spikes.
- At the limit of full attention ( $\rho=1$ ), the spikes and plateaus merge into the stable, persistent MA morphology characteristic of conventional full attention LLMs.

Mechanistic Insights
The authors propose a shared lifecycle account governed by the timing of MA cancellation:

PAS Mechanism: Follows a localized write–sink–cancel process. A pre-attention layer writes a large outlier into the residual stream; the subsequent full attention layer acts as a sink, receiving disproportionate attention; and a subsequent opposite-signed update cancels the outlier, causing the spike to dissipate.
ISP Mechanism: Represents a regime of delayed cancellation. The outlier written before a full attention layer persists through intervening linear attention layers because the cancellation event is deferred.
Role of Gating: Controlled pretraining reveals that full attention output gating strongly attenuates the absolute magnitude of both PAS and ISP without eliminating their layerwise organization. Conversely, removing the native output gates of GDN layers results in only modest amplification. This suggests full attention plays a central role in organizing MA dynamics, while GDN gating primarily modulates propagation.

Significance
The paper claims that these findings establish MAs as an informative lens for understanding hybrid attention mechanisms. By demonstrating that PAS and ISP are recurrent, architecture-aligned phenomena that emerge early in training and respond predictably to gating, the work provides a unified explanation for MA dynamics across the spectrum from linear attention to full attention. The study bridges the gap between the distinct, spike-based organization of HLA models and the stable morphology of full attention models, suggesting that the transition between them is a continuum of outlier persistence regulated by cancellation timing.

Limitations and Scope
The authors maintain a modest scope, focusing on characterizing the existence and mechanism of these morphologies rather than proposing immediate applications. The study relies on specific model suites (M-A-P) and representative open-source checkpoints. While the findings are robust across scales and domains, the paper does not claim to solve the underlying causes of why these specific cancellation timings occur, identifying this as a direction for future work. The code and checkpoints are made available to ensure reproducibility.

Technical Summary: Massive Activations in Hybrid Linear Attention Large Language Models

More like this