🤖 machine learning

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

NSNQuant is a calibration-free vector quantization method for LLM KV caches that employs a unique "Normalize-Shift-Normalize" transformation combined with a Hadamard transform to align token distributions with a standard normal distribution, enabling robust low-bit compression and up to 3x throughput gains without relying on calibration datasets.

Original authors: Donghyun Son, Euntae Choi, Sungjoo Yoo

Published 2026-07-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Donghyun Son, Euntae Choi, Sungjoo Yoo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to carry a massive library of memories in your backpack while walking a very long path. This is what happens when a Large Language Model (LLM)—a super-smart computer brain that writes stories, solves math, and chats with you—tries to process a long conversation. Every time it reads a word, it has to remember everything that came before it to understand the context. This "memory" is called the KV Cache (Key-Value Cache). The problem is, as the conversation gets longer, this backpack gets heavier and heavier, eventually becoming so full that the computer runs out of space and slows to a crawl.

To fix this, scientists have tried to shrink the backpack by compressing the memories, much like packing clothes into a vacuum bag. One popular method is Vector Quantization (VQ). Think of this as grouping similar items together and replacing them with a single label from a "dictionary" or codebook. Instead of remembering the exact shade of blue for every single sock, you just remember "Blue Group 4." However, there's a catch: most existing methods need to study a specific set of clothes (a calibration dataset) to build that dictionary. If you then try to pack a completely different set of clothes (a new type of conversation), the dictionary doesn't fit, and the compression fails. This paper tackles that exact problem: how to shrink the memory without needing to study the clothes first.

The Problem: A Dictionary That Only Works for One Wardrobe

The authors of this paper, from Seoul National University, noticed a frustrating glitch in the current state-of-the-art method for compressing LLM memory, called Coupled Quantization (CQ). Imagine CQ as a tailor who makes a custom suit based on measurements taken from a single person. If that person walks into a room full of people with different body types, the suit fits the first person perfectly but looks ridiculous on everyone else.

In the world of AI, this "person" is the data the model was calibrated on (like a specific text dataset called WikiText-2). When the model tries to process a different type of text (like the C4 dataset, which is a massive collection of web pages), the "suit" doesn't fit. The authors found that this mismatch causes the model to make silly mistakes, especially with punctuation marks. For instance, the model might get confused about commas because the "dictionary" it learned from the training data didn't have the right labels for the way commas appear in the new text. This is a big deal because it means the model becomes unreliable when it steps outside its comfort zone.

The Solution: NSNQuant – The Universal Packing Cube

To solve this, the team introduced NSNQuant, a clever new way to compress the memory that doesn't need to study any specific data first. Instead of trying to learn a custom dictionary for every new wardrobe, NSNQuant forces all the clothes to fit into a standard, pre-made packing cube.

Here is how they do it, using a three-step "magic trick" they call Normalize-Shift-Normalize (NSN):

Normalize (The First Step): Imagine you have a pile of socks where some are tiny and some are giant. The first step is to stretch or shrink every sock so they are all the same size. This prevents the giant socks from hogging all the space and messing up the packing.
Shift (The Middle Step): Now, imagine the socks are all the same size, but they are all leaning to the left. The "Shift" step pushes them all back to the center so they are perfectly balanced.
Normalize (The Final Step): Just to be sure, they check the size one last time to make sure everything is still uniform.

After this three-step dance, the authors add a final twist: a Hadamard Transform. You can think of this as spinning the entire pile of socks in a specific, mathematical way. The magic is that after this spin, the messy, unpredictable shapes of the socks (the data) suddenly look like a perfect, smooth bell curve (a standard normal distribution).

Because the data now looks like this predictable bell curve regardless of what the original text was, the authors can use a single, pre-made "dictionary" (codebook) designed specifically for that bell curve. They don't need to look at the data first; they just know the data will fit the dictionary because they forced it to look that way.

What They Found: A Universal Key

The team tested this idea on several famous AI models, including the LLaMA and Mistral families. They compared NSNQuant against the old methods (like CQ and KIVI) using different types of text, from simple stories to complex code and math problems.

The results were impressive:

Better Generalization: While the old methods (CQ) stumbled when switching from one dataset to another, NSNQuant kept performing strongly. It was like having a universal key that opened every door, whereas the old keys only worked on the doors they were made for.
Low-Bit Success: The team tested compressing the memory down to just 1-bit and 2-bit. In the 1-bit setting (where the memory is shrunk to its absolute smallest), NSNQuant crushed the competition. For example, on a math reasoning task called GSM8K, the old 1-bit method scored around 24, while NSNQuant-1b scored 53.45. That's more than double the performance!
Speed and Space: Because the memory is so much smaller, the computer can process more conversations at once. The authors showed that their method could handle 3 times more data throughput than the standard, uncompressed version, while using significantly less memory.

The Fine Print

The authors are careful to note that while this method is a huge improvement, it isn't perfect magic. They found that in the very first layers of the AI model, the "socks" sometimes still have a few outliers that don't fit the bell curve perfectly. However, even with these small glitches, the overall performance remained very high.

They also emphasize that this method is "calibration-free." Unlike the old methods that required hours of studying specific data to build a dictionary, NSNQuant's dictionary can be built in less than 5 minutes on a single graphics card and then reused for any model. This makes it incredibly practical for real-world use.

In short, NSNQuant is like a universal packing system that forces any messy pile of memories into a neat, predictable shape, allowing AI models to carry their long-term memories in a much smaller backpack without losing their ability to think clearly. It suggests that by standardizing the data before compressing it, we can make AI faster, cheaper, and more reliable, even when it's dealing with completely new and unfamiliar topics.

Technical Summary: NSNQuant

Problem Statement

Large Language Model (LLM) inference is increasingly memory-bound, particularly when handling long sequences and large batch sizes. The primary bottleneck is the Key-Value (KV) cache, which grows linearly with sequence length. While Vector Quantization (VQ) has emerged as a promising compression technique, existing methods like Coupled Quantization (CQ) rely on calibration datasets to learn centroids (codebooks). The authors observe that these calibration-based approaches suffer from severe performance degradation in out-of-distribution (OOD) scenarios. Specifically, when the input distribution differs from the calibration set (e.g., using WikiText-2 for calibration but testing on C4), the learned centroids fail to accurately represent critical tokens, such as punctuation, leading to significant errors in attention weights and degraded model performance (e.g., higher perplexity).

Methodology: NSNQuant

To address the reliance on calibration data and the resulting distribution shift, the authors propose NSNQuant, a calibration-free VQ technique. The core innovation is a transformation pipeline that aligns the KV cache distribution with a standard normal distribution, allowing for the use of a single, reusable codebook without external data.

1. Normalize-Shift-Normalize (NSN)

The method applies a three-step transformation to the KV tensors ( $v \in \mathbb{R}^{l \times d}$ ):

Normalize (Token-wise): Each token is normalized to have a norm of $\sqrt{d}$ . This prevents outlier tokens from dominating subsequent steps and ensures tokens with small magnitudes are not incurring large relative errors.
Shift (Channel-wise): The channel-wise mean is calculated and subtracted to center the distribution around zero.
Normalize (Token-wise): A second normalization is applied to restore the token norm to $\sqrt{d}$ .

This process produces byproducts ( $s_1$ , $o$ , $s_2$ ) used to reconstruct the original tensor. Theoretically, when combined with a Hadamard transform, this process aligns the channel distribution with the standard normal distribution, as supported by the Central Limit Theorem and variance bounds derived in Lemma 1.

2. Hadamard Transform

The NSN process is paired with a Hadamard transform (specifically the Randomized Hadamard Transform or RHT). While the naive Hadamard transform is used in the final implementation for efficiency, the RHT provides theoretical guarantees that the resulting channel variances are close to 1.

3. Vector Quantization and Codebook

Once transformed, the data is quantized using a single global codebook tailored for the standard normal distribution.

Codebook Construction: Instead of learning centroids from a calibration set, the codebook is initialized via K-Means on synthetic standard normal data ( $\text{torch.randn}$ ) and fine-tuned to minimize the cosine distance between original and quantized vectors. This process is fast (minutes) and does not require backpropagation through model weights.
Scale Adjustment: An adaptive scaling strategy is employed where the quantized vector is scaled to preserve components parallel to the original vector, minimizing orthogonal errors.
Double Quantization (DQ): To further reduce overhead, the parameters used for the NSN process (mean $o$ and scale $s_1$ ) and the codebook entries themselves are quantized (4-bit), adding negligible memory overhead (approx. 0.23 bits on average).

4. Implementation Details

Residuals: Following KIVI, the KV cache is split into a quantized part and a full-precision residual. When the residual fills, it is flushed, quantized, and appended.
Attention Computation: For keys, NSN is applied before the Rotary Position Embedding (RoPE). The authors find this ordering yields better quality. The attention score computation is modified to incorporate the NSN byproducts ( $s_1, s_2, o$ ) directly within custom CUDA kernels.
Efficiency: Custom CUDA kernels are implemented for Hadamard transforms, vector quantization, and dequantization, utilizing shared memory to minimize DRAM access.

Key Contributions

Empirical Observation: The authors demonstrate that the strong dependence of KV distributions on input datasets causes existing VQ methods (like CQ) to fail in OOD scenarios, particularly affecting punctuation tokens in models like LLaMA3.
Calibration-Free VQ: NSNQuant introduces a method that eliminates the need for calibration datasets by aligning the KV distribution to a known prior (standard normal) via the NSN process and Hadamard transform.
Performance and Generalization: Extensive experiments show NSNQuant consistently outperforms baselines (KIVI, KVQuant, CQ) in both 1-bit and 2-bit settings across diverse datasets (WikiText-2, C4, LongBench, GSM8K, MMLU). It exhibits superior generalization where calibration-based methods degrade.
Efficiency: The implementation achieves up to a 3× throughput gain over full-precision baselines and supports larger batch sizes (up to 4×) due to reduced memory usage.

Results

Perplexity (PPL): On the C4 dataset, NSNQuant significantly outperforms CQ. For instance, with LLaMA3-8B, CQ-4c9b yields a PPL of 13.97, while NSNQuant-2b achieves 9.08. In 1-bit settings, NSNQuant-1b (PPL 16.69) vastly outperforms CQ-8c10b (PPL 43.78).
LongBench: NSNQuant-1b outperforms other 1-bit baselines by a large margin. In 2-bit settings, it matches or slightly exceeds other methods, though the authors note metric noise in certain tasks.
Reasoning and Generation: On GSM8K (math reasoning) and MMLU (multi-task understanding), NSNQuant excels, particularly in few-shot Chain-of-Thought (CoT) settings where strict reasoning paths are required.
Latency: While the prefill stage incurs higher latency due to NSN and Hadamard overhead, the decoding stage is faster than full-precision baselines due to the alleviation of memory-bound attention computations.

Significance and Claims

The paper claims that NSNQuant represents a shift from data-dependent calibration to distribution-aligned quantization. By leveraging the property that the Hadamard transform combined with NSN produces a standard normal distribution, the method enables the use of a universal, model-agnostic codebook. This makes the approach robust to distribution shifts and eliminates the computational cost and data requirements of calibration. The authors position NSNQuant as a practical solution for enabling long-context LLM inference on memory-constrained hardware, offering a balance of high compression (1-bit/2-bit), strong generalization, and efficient runtime performance.

The authors modestly acknowledge limitations, noting that the standardization is not perfect in early layers due to outlier channels with high variances, and that the method does not explicitly model inter-channel correlations, unlike CQ. However, they argue that the trade-off favors robustness and generalization over the potential gains from overfitting to specific calibration distributions.