🤖 machine learning

Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization

This paper reveals that massive activation spikes in Large Language Models are structural vector biases rather than simple scalar anomalies, and leverages this mechanistic insight to propose INSERTQUANT, a post-training quantization framework that eliminates these spikes using template vectors to enable robust, high-fidelity low-bit quantization across text and vision modalities.

Original authors: Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou, Naveen Verma

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou, Naveen Verma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Loud Neighbor" in the Library

Imagine a Large Language Model (LLM) as a massive, high-speed library where books (tokens) are constantly being read and processed. To make this library run faster and use less electricity, engineers want to shrink the books down to tiny, efficient sizes (a process called quantization).

However, there is a problem. In every library, there are a few specific "Loud Neighbors" (specific tokens like newlines or special start symbols) that scream so loudly they drown out everyone else. In the math of the AI, these screams are Massive Spikes—numbers that are thousands of times bigger than normal.

Because these spikes are so huge, the library has to build a giant, expensive storage system just to hold them. This ruins the efficiency. If you try to shrink the books (quantize) without fixing the noise, the library collapses into gibberish.

The Old Theory vs. The New Discovery

The Old Theory: Scientists previously thought these "Loud Neighbors" were just random glitches or simple "volume knobs" (scalar biases) that the AI turned up by accident.

The New Discovery (The Bias Vector Hypothesis): The authors of this paper argue that these spikes aren't accidents. They are actually rigid, structural pillars built into the AI's design.

The Analogy: Imagine a stage play. Most actors (semantic tokens) are moving around, changing costumes, and saying different lines to tell a story. But there is one specific actor (the "Sink Token") who stands perfectly still in the center of the stage, wearing the exact same mask, saying the exact same line, and never moving, no matter what the play is about.
The paper proves that this "still actor" isn't random. It is a fixed vector (a specific direction in math space) that the AI needs to function correctly. It acts as a "do nothing" button that helps the AI ignore distractions and stay focused.

How the AI Uses This "Still Actor"

The paper digs into the mechanics of how this works, revealing a three-step dance:

The Attractor (The Magnet): The AI's "Key" and "Query" mechanisms act like magnets. They pull all the other actors toward this "Still Actor." This creates an Attention Sink, where the AI focuses its attention on this one spot to keep the math stable.
The Silent Void (The Drain): Even though everyone is looking at the "Still Actor," the AI's "Value" mechanism acts like a black hole. It takes the "Still Actor's" message and turns it into zero. This ensures that while the AI looks at this spot for stability, it doesn't actually change the story. It's a "No-Op" (No Operation).
The Shield (The Safety Zone): The AI knows that the "stage" (the input sequence) keeps rotating and changing. To keep this "Still Actor" from wobbling, the AI hides it in a safe zone (low-frequency channels) where the rotation doesn't affect it. It's like building a fortress around the pillar so the wind can't knock it over.

The Solution: INSERTQUANT

Since the authors now know that these "Loud Neighbors" are actually just static, pre-programmed pillars and not random noise, they created a new tool called INSERTQUANT.

Here is how it works, using a "Swapping" analogy:

Identify the Pillar: The system scans the library and finds the "Loud Neighbor" tokens.
Silence the Noise (Clamp): Instead of letting these tokens scream and take up all the storage space, the system simply mutes them (clamps them to zero) before they enter the main processing room. This instantly removes the "spikes" and makes the data easy to shrink (quantize).
Swap in the Blueprint (Insert): Since the AI needs that "Still Actor" to function, the system doesn't just delete it. Instead, it inserts a pre-made blueprint (a template vector) right where the actor should be.
- Analogy: Imagine you are filming a movie. Instead of hiring a real actor to stand still for 10 hours (which is expensive and hard to manage), you just tape a cardboard cutout of that actor to the set. The camera sees the actor, the lighting works, but you saved a lot of money and effort.

The Results

By using this "Cutout" method (INSERTQUANT):

No More Spikes: The data becomes perfectly smooth, allowing the AI to be shrunk down to very small sizes (4-bit quantization) without losing intelligence.
Works Everywhere: Previous methods only worked for text (because they relied on knowing specific words like "newline"). This new method works on the structure of the data, so it also works for Vision Transformers (ViTs) (AI that looks at images), not just text.
High Fidelity: The AI performs just as well as the original, unshrunk version, even with the "cardboard cutouts" in place.

Summary

The paper says: "Stop treating these massive spikes as errors. They are actually rigid, structural tools the AI uses to stay stable. If we recognize them as fixed tools, we can remove the noisy parts of the calculation and replace them with pre-made templates. This lets us shrink the AI down to tiny sizes without breaking it."

Technical Summary: Massive Spikes in LLMs are Bias Vectors

Problem Statement

Large Language Models (LLMs) face severe computational and memory constraints that necessitate efficient quantization. However, the efficacy of activation quantization is critically hindered by Massive Spikes: extreme activation values localized to specific channels and specific token semantics (often low-semantic "sink" tokens). These spikes are orders of magnitude larger than typical values, stretching the dynamic range and causing catastrophic quantization errors, particularly for static per-tensor schemes.

While prior work has characterized these spikes as high-level scalar biases or "outliers" (Sun et al., 2024; An et al., 2025), existing mitigation strategies often treat them as numerical aberrations or rely on heuristics limited to discrete text modalities (e.g., prefixing specific tokens). The fundamental geometric mechanics of these spikes and their role in model behavior remain underexplored, limiting the development of generalized, robust quantization frameworks.

Methodology: The Bias Vector Hypothesis

This work advances the understanding of massive spikes by proposing the Bias Vector Hypothesis. The authors argue that spikes are not merely scalar biases but the scalar intermediates of rigid, structural vector biases inherent to the model's architecture.

1. Geometric Characterization

Through mechanistic analysis, the paper establishes that after RMSNorm, spike-carrying tokens (sink tokens) converge to constant, rigid vectors ( $\hat{x}_{sink} \to b$ ) with negligible variance across samples, distinct from the high-variance semantic tokens.

Attention Sink Mechanism: The projection weights ( $W^K$ and $W^Q$ ) interact to isolate this bias vector. $W^K$ acts as a contrastive amplifier, separating the sink key from semantic keys, while $W^Q$ aligns all semantic queries toward this isolated sink key, creating a global attractor.
Value-State Drain: The value projection matrix ( $W^V$ ) projects the bias vector into the spectral null-space. Consequently, the resulting value vector is near-zero ( $v_{sink} \approx 0$ ), ensuring the sink token contributes no semantic update to the residual stream, effectively enabling a "no-op" operation.
Rotational Stability: The model actively preserves these rigid structures against Rotary Positional Embedding (RoPE) perturbations. This is achieved by localizing the bias in "zones of rotational stability": specifically, utilizing low-frequency bands (high channel indices where rotation angles decay) and coherent channel pairs (indices $i$ and $i + d/2$ ) to maintain magnitude robustness despite rotation.

2. The INSERTQUANT Framework

Leveraging the insight that sink tokens function as static structural biases rather than dynamic semantic content, the authors propose INSERTQUANT, a Post-Training Quantization (PTQ) framework.

Clamp: Sink tokens are identified during a lightweight calibration phase and surgically clamped to zero before entering matrix multiplications. This eliminates the massive spikes from the activation stream, rendering the remaining activations strictly spike-free and quantization-friendly.
Insert: To preserve the model's "no-op" mechanics, the functional role of the clamped tokens is restored in subsequent layers by injecting pre-computed template vectors (mean $K$ and $V$ vectors derived from calibration) directly into the attention mechanism inputs.
Generalization: Unlike prefix-based methods that rely on specific token identities (e.g., <BOS>), INSERTQUANT operates on the embedding space, allowing it to generalize to modalities without discrete token definitions, such as Vision Transformers (ViTs).

Key Contributions

Bias Vector Hypothesis: The paper establishes that massive spikes are the scalar intermediates of rigid bias vectors. It demonstrates that while spike magnitudes fluctuate, their normalized direction is invariant, serving as a structural model component.
Mechanistic Analysis: The authors provide a quantitative geometric analysis showing how bias vectors drive attention sinks and value-state drains. They detail the interaction with projection weights ( $W^K, W^Q, W^V$ ) and identify the localization of these vectors in rotational stability zones (low-frequency bands and coherent pairs) as evidence of active structural preservation.
INSERTQUANT Framework: The work introduces the first PTQ framework to exploit the systematic nature of spikes. By decoupling structural biases from content tokens, it achieves high-fidelity low-bit quantization across both LLMs and ViTs without requiring fine-tuning.

Experimental Results

The authors evaluate INSERTQUANT on multiple LLMs (LLaMA-3.2-3B, LLaMA-2-7B/70B, Mistral-7B-v0.3) and Vision Transformers (CLIP ViT-L, DINOv2 ViT-L/B).

Language Models: In 4-bit weight/activation/KV-cache static settings ($W4A4sKV4$), INSERTQUANT achieves performance parity with state-of-the-art methods like PrefixQuant. For example, on LLaMA-2-7B, it achieves a Common-Sense Reasoning (CSR) accuracy of 60.26% and a WikiText-2 perplexity of 6.31, closely matching the baseline PrefixQuant (60.17% / 6.21) and significantly outperforming RTN and rotation-based baselines.
Vision Transformers: INSERTQUANT successfully generalizes to ViTs. In the $W8A8sKV8$ setting, while the RTN baseline on DINOv2 ViT-B collapses to 6.00% accuracy, INSERTQUANT preserves 74.10% accuracy.
System Overhead: The framework introduces negligible latency overhead (4.69%) and minimal memory overhead (0.007% increase for a 13.5GB model) due to the small size of the stored template vectors.

Significance and Claims

The paper claims that effective quantization requires a mechanistic understanding of how LLMs utilize spikes to enable "no-op" operations. By reframing massive spikes as structural vector-level features rather than numerical outliers, the authors demonstrate that these artifacts can be surgically removed and functionally restored.

The significance of this work lies in:

Deepening Model Understanding: It reveals the rigid, deterministic nature of attention sinks and their geometric coordination with RoPE, moving beyond scalar interpretations.
Unified Quantization Solution: It establishes a framework that transcends text modalities, offering a robust solution for quantizing diverse architectures (LLMs and ViTs) without relying on heuristic token identities.
Hardware Efficiency: By enabling strictly spike-free activation spaces, INSERTQUANT facilitates robust, low-bit inference (e.g., 4-bit) that was previously unattainable with static per-tensor quantization schemes.

The authors remain modest regarding theoretical guarantees, noting that while they provide a mechanistic interpretation and empirical evidence, they do not offer a theoretical proof of why the training process inherently converges to this specific geometric solution. They also acknowledge that their method is unnecessary for models pre-trained with architectural modifications designed to natively suppress these spikes.