🤖 machine learning

Quantize What Counts: More for Keys, Less for Values

This paper establishes a theoretical foundation for mixed-precision KV-cache quantization in Large Language Models by proving that prioritizing higher precision for keys over values strictly reduces quantization error and preserves accuracy, thereby transforming bit allocation from heuristic tuning into a geometry-driven design principle.

Original authors: Mohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han, Vipin Chaudhary

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han, Vipin Chaudhary

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Memory Fridge" is Too Full

Imagine a Large Language Model (LLM) like a brilliant chef who is cooking a very long meal (a conversation or a story). To keep the meal going, the chef needs to remember every ingredient they've added so far. In computer terms, this memory is called the KV Cache (Key-Value Cache).

As the meal gets longer (more tokens), the chef needs a bigger and bigger fridge to store these ingredients. Eventually, the fridge becomes so full that the kitchen runs out of space, slowing everything down or stopping the cooking entirely. This is the "memory bottleneck" the paper addresses.

The Current Fix: Shrink the Ingredients

To fix the full fridge, engineers use quantization. Think of this as compressing the ingredients. Instead of storing a whole, heavy watermelon (high precision), you store a smaller, lighter slice (lower precision). This saves space.

However, there's a catch: if you shrink the ingredients too much, the chef might forget the recipe or make a mistake. The big question has always been: "How much can we shrink the Key ingredients versus the Value ingredients without ruining the meal?"

Until now, people guessed (heuristics) or tried random combinations to see what worked. This paper says: "Stop guessing. Let's look at the math."

The Discovery: Keys are the "Heavy Lifters"

The authors discovered a fundamental difference between the two types of ingredients: Keys and Values.

The Analogy: Imagine the "Key" is the label on a box, and the "Value" is the stuff inside the box.
The Finding: The paper proves that the "labels" (Keys) are mathematically "heavier" and more complex than the "stuff inside" (Values). In technical terms, the Key matrices have larger "norms" (a measure of size and energy) than the Value matrices.

Because the Keys are heavier, they carry more "information density." If you squish a heavy object too much, it breaks. If you squish a light object, it barely changes.

The Solution: "More for Keys, Less for Values"

Based on this discovery, the authors propose a new rule for shrinking the ingredients:

Give the Keys more bits (keep them larger): Since the Keys are the heavy, complex labels, they need to stay relatively precise.
Give the Values fewer bits (shrink them more): Since the Values are lighter and less sensitive, you can compress them significantly without losing much accuracy.

The Creative Metaphor:
Imagine you are packing a suitcase for a trip.

The Keys are your passport and tickets. If you scribble on them or fold them too small, you can't use them, and you get stuck. They need to be kept crisp and clear (High Precision).
The Values are your socks and t-shirts. You can fold them tightly, roll them up, or even stuff them in the corners. They take up space, but if they get a little wrinkled, you can still wear them (Low Precision).

The paper's strategy is: Pack the passport carefully (4 bits), but stuff the socks tightly (2 bits).

The Results: Saving Space Without Losing Quality

The researchers tested this "Passport vs. Socks" strategy on many different models (like Llama, Mistral, and Qwen) and tasks (math, conversation, writing).

The Old Way: Treat everything the same (e.g., 4 bits for passports, 4 bits for socks). This wastes space on the socks.
The New Way: 4 bits for passports, 2 bits for socks.

The Outcome:

Space Saved: They saved about 25% of the memory needed for the fridge.
Accuracy Kept: The model still performed at 98.3% of its original accuracy.
The "K4V2" Sweet Spot: Specifically, allocating 4 bits to Keys and 2 bits to Values worked almost as well as keeping everything at 4 bits, but used half the memory for the values.

Why This Matters

This paper changes the game from "trial and error" to "scientific design."

Before: Engineers would tweak settings randomly until the model didn't crash.
Now: We have a rule based on the geometry of the model itself: Prioritize the Keys.

They also showed that this rule works perfectly with other tricks (like "rotation," which is like rearranging the suitcase to fit things better). When you combine "More for Keys" with these other tricks, the results get even better.

Summary

The paper argues that in the memory of AI models, Keys are the critical, heavy-lifting parts, while Values are the flexible, compressible parts. By giving the Keys more attention (bits) and the Values less, we can shrink the model's memory footprint significantly without making the AI "forget" how to think. It's a simple, math-backed rule: More for Keys, Less for Values.

Technical Summary: Quantize What Counts: More for Keys, Less for Values

Problem Statement

Large Language Models (LLMs) face severe inference-time memory bottlenecks driven by the Key-Value (KV) cache, which scales linearly with model size and context length. While KV-cache quantization (reducing precision from BF16 to INT4 or lower) offers a pathway to memory savings, existing methods often rely on heuristic, ad-hoc tuning for bit allocation between keys and values. Current approaches typically treat the key-value split as a hyperparameter to be grid-searched or derived from inference-time activation statistics, lacking a theoretical foundation rooted in intrinsic model properties. This raises a fundamental question: How can bit allocation be determined in a principled, generalizable manner that minimizes quantization error while preserving model accuracy?

Methodology

The paper proposes a geometry-driven approach to mixed-precision KV quantization, grounded in the intrinsic spectral and Frobenius norms of Transformer weight matrices. The methodology proceeds in two theoretical and empirical steps:

Theoretical Analysis of Norm Disparity: The authors analyze the training dynamics of key ( $W_K$ ) and value ( $W_V$ ) projection matrices. They demonstrate that $W_K$ systematically acquires larger spectral and Frobenius norms than $W_V$ due to the architectural asymmetry in attention mechanisms. Specifically, $W_K$ shapes the attention map and interacts multiplicatively with queries, leading to amplified gradient signals during training, whereas $W_V$ only influences post-attention representations. This results in $W_K$ having higher information density and magnitude.
Derivation of Quantization Theorems: Based on the observation that quantization error scales with signal energy (norm magnitude), the authors derive two theorems:
- Key-Value Norm Disparity Theorem: Proves that the expected spectral and Frobenius norms of $W_K$ exceed those of $W_V$ across prominent LLM families.
- Key-Prioritized Quantization Theorem: Establishes that for any fixed memory budget, allocating higher precision to keys than to values strictly reduces the total quantization error. This is because the error introduced in the high-norm key path is amplified more significantly through the network than errors in the value path.

The authors validate these theorems across diverse models (Llama-3, Mistral, Phi, Qwen, DeepSeek), datasets (C4, MMLU, GSM8K, CoQA, EQ-Bench, LongBench), and quantization backends (Optimum Quanto and HQQ). They also investigate the integration of this strategy with rotation-based outlier redistribution methods (e.g., QuaRot).

Key Contributions

Theoretical Grounding: The paper provides the first theoretical justification for asymmetric bit allocation in KV caches, moving beyond heuristic tuning to a design principle based on the linear algebraic properties of Transformer weights.
Norm Disparity Theorem: It proves that key projections consistently exhibit larger norms than value projections, a property preserved in the resulting caches.
Key-Prioritized Quantization Theorem: It formally demonstrates that prioritizing precision for keys over values minimizes quantization error and preserves inference accuracy more effectively than symmetric or inverted allocations.
Empirical Validation: Extensive experiments show that key-favored allocations (e.g., 4-bit keys, 2-bit values) retain up to 98.3% of the accuracy of uniform 4-bit allocations (K4V4) while reducing KV-cache memory by 25%.
Orthogonality and Integration: The study shows that this geometry-driven strategy is orthogonal to existing outlier redistribution techniques. Combining key-prioritized quantization with key-only rotation yields synergistic gains, outperforming uniform baselines by 4.4–18% in accuracy across tasks.

Results

Reconstruction Error: Across seven models and multiple datasets, key caches consistently incur larger reconstruction errors than value caches at identical bit-widths (e.g., K2 vs. V2), confirming their higher sensitivity to quantization.
Downstream Accuracy:
- In GSM8K evaluations using Optimum Quanto, the K4V2 configuration (4-bit keys, 2-bit values) significantly outperforms K2V4, recovering approximately 94% of the full-precision baseline accuracy, whereas value-favored allocations incur losses of up to 30 percentage points.
- Using HQQ across a broader range of bit-widths (1–8 bits) and models (0.6B–32B), allocating even a single extra bit to keys at ultra-low precisions yields massive gains (e.g., +48 percentage points for Qwen3-8B on GSM8K when moving from 1-bit keys to 2-bit keys).
- On average, K4V2 retains 98.3% of the accuracy of K4V4 across CoQA, EQ-Bench, and GSM8K.
Rotation Integration: Applying rotation to keys alone in a K4V2 setting achieves accuracy nearly matching the full K4V4 baseline, indicating that the primary benefit of rotation stems from mitigating key outliers. Rotating values alone provides marginal or negative returns.
Group Size Sensitivity: Finer group sizes (e.g., 32) for keys improve accuracy due to higher information density, while coarser group sizes (64 or 128) for values suffice with negligible accuracy loss.

Significance

The paper claims to transform bit allocation from an ad-hoc design choice into a theoretically grounded, geometry-driven design principle for efficient LLM inference. By anchoring mixed-precision strategies in the intrinsic norms of model weights, the authors provide a generalizable rule: "More for keys, less for values." This approach offers a lightweight, one-off analytical cost per model without requiring inference-time introspection or retraining. The findings suggest that future KV quantization frameworks and hardware-algorithm co-design should prioritize key precision to achieve maximum memory efficiency with minimal accuracy degradation. The work also highlights that this strategy complements existing outlier redistribution techniques, enabling further performance gains without increasing complexity.