💬 NLP

LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation

LogQuant introduces a novel log-distributed 2-bit quantization technique for KV caches that significantly improves inference throughput, batch size, and task accuracy for large language models while maintaining a minimal memory footprint.

Original authors: Han Chen, Zicong Jiang, Zining Zhang, Bingsheng He, Pingyi Luo, Mian Lu, Yuqiang Chen

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Han Chen, Zicong Jiang, Zining Zhang, Bingsheng He, Pingyi Luo, Mian Lu, Yuqiang Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to remember a very long story to tell a joke at the end. To do this, your brain (the AI model) keeps a "scratchpad" (the KV Cache) where it writes down every word and sentence it has heard so far.

The problem is that as the story gets longer, this scratchpad gets huge. It takes up so much space that you can't tell many jokes at once, or the story takes forever to process.

The Old Way: Throwing Things Away or Guessing
Previous methods tried to solve this in two ways:

The "Trash Can" approach: They would look at the story and guess which parts were unimportant, then throw them away completely. The problem? They often threw away the wrong things, or missed a crucial detail hidden deep in the past.
The "Blurry Photo" approach: They tried to keep everything but write it down in a blurrier, less precise way (quantization). But if they made everything too blurry, the story made no sense.

The New Way: LogQuant (The "Logarithmic Library")
The authors of this paper, LogQuant, realized something clever about how our brains (and AI models) actually pay attention.

The Insight: The "Logarithmic" Pattern
They discovered that when the AI looks back at the story, it doesn't look at every word equally.

It looks very closely at the most recent words (the last few sentences).
It looks less closely at words from a while ago.
It looks very sparsely at words from the very beginning.

Crucially, this pattern isn't random; it follows a specific mathematical curve called a logarithm. Think of it like a library where the books on the front shelves (recent events) are packed tight, but as you go deeper into the library, the shelves get wider and the books get spaced further apart.

How LogQuant Works
Instead of guessing what to keep or throwing things away, LogQuant uses this "spaced-out" pattern to compress the memory:

The "Recent" Zone: It keeps the very last part of the story in high definition (full precision) because that's where the action is.
The "Middle" Zone: As it goes further back, it starts skipping words. It keeps one word, skips one, keeps one, skips one.
The "Deep" Zone: Even further back, it skips more. It keeps one word, skips three, keeps one, skips three.

By doing this, it can shrink the memory down to just 2 bits (a tiny amount of space, like turning a high-definition movie into a tiny text file) without losing the important plot points. Because it follows the natural "logarithmic" way the AI pays attention, it doesn't need to guess which parts are important; the math tells it exactly where to look.

The Results: More Jokes, Better Memory
The paper claims that by using this method:

Speed: The AI can process information 25% faster.
Capacity: You can run 60% more conversations at the same time on the same computer because the memory footprint is so much smaller.
Accuracy: For hard tasks like solving math problems or writing code, LogQuant is 40% to 200% more accurate than other compression methods. It's like keeping the story clear enough to solve a puzzle, even when the memory is tiny.

Why It's Better Than "Throwing Things Away"
The authors also proved that "throwing things away" (eviction) is bad for the AI's attention. If you delete a word, the AI has to recalculate how the remaining words relate to each other, which distorts the story. LogQuant keeps all the words but writes them down in a smaller, compressed format. It's like keeping every page of a book but printing them in a smaller font, rather than ripping out half the pages.

In Summary
LogQuant is a smart way to shrink an AI's memory by realizing that the AI naturally pays attention to recent events intensely and older events loosely. By compressing the memory to match this natural pattern, it saves massive amounts of space and speed without making the AI "forget" the important parts of the story.

Technical Summary: LogQuant

Problem Statement

The rapid expansion of context windows in Large Language Models (LLMs), ranging from 4k to 128k tokens, has created a critical bottleneck in Key-Value (KV) cache management. Recent studies indicate that KV cache memory usage grows linearly with context length, often exceeding the memory footprint of the model weights themselves during long-context inference and batch processing.

Existing compression strategies generally fall into two categories:

Eviction-based methods (e.g., H2O, Keyformer, SnapKV): These selectively remove tokens deemed unimportant. However, they rely on historical attention patterns or fixed windows, leading to frequent mispredictions and the loss of distant but critical tokens.
Quantization-based methods (e.g., QAQ, KiVi): These reduce the precision of less important tokens. While they retain more data, they struggle with accurately identifying which tokens are crucial for maintaining performance, particularly at aggressive compression ratios like 2-bit.

Current approaches face a trade-off between memory efficiency and accuracy preservation, often suffering from performance bottlenecks or significant accuracy degradation in complex tasks like mathematics and code completion.

Methodology

LogQuant introduces a novel 2-bit quantization technique for KV Cache that leverages a specific observation regarding attention score distributions.

1. Log-Distributed Attention Spikes

Through analysis of various models (e.g., Llama3, Qwen) and tasks, the authors observed that the positions of high-attention spikes follow a logarithmic distribution. As tokens move further from the current position, the density of important tokens decreases logarithmically. This contrasts with the assumption that importance is strictly local or predictable via historical scores.

2. Log-Based Filtering Strategy

Instead of predicting importance or using fixed windows, LogQuant applies a log-based filtering mechanism to compress the KV cache across the entire context:

Window Configuration: A base window size $W$ is defined.
Selection Scheme: The method retains the most recent $2W$ tokens in full precision. Following this, it selects tokens in subsequent windows with decreasing density: a window of size $W/2$ selects tokens spaced one apart, $W/4$ selects with further spacing, and so on.
Full Precision Reserve: A final window of $3W$ tokens is reserved in full precision.
Dynamic Adjustment: As new tokens arrive and the length limit is reached, the density in the earlier windows is halved (retaining tokens at regular intervals), effectively reducing the retained full-precision count from $2W$ to $W$ before adding new tokens to restore the window size. This creates a stable log-sparsity selection.

3. Quantization vs. Eviction

The paper demonstrates that quantization is superior to eviction for this strategy. Eviction removes tokens entirely, altering the softmax normalization and causing larger deviations in the attention distribution. Quantization retains all tokens with reduced numerical precision, preserving the attention distribution's shape more effectively. Experimental results show eviction-based methods produce twice the attention error compared to quantization.

4. Position-Agnostic Optimization

A key efficiency gain is achieved by recognizing that the specific ordering of tokens in the Key and Value matrices does not affect the final attention output ( $A \cdot V = A_P \cdot V_P$ ). LogQuant ignores the absolute positions of KV cache entries during the decoding phase, allowing the system to concatenate high-precision and quantized tokens arbitrarily. This enhances memory locality and accelerates the quantization/dequantization process without affecting accuracy.

5. Implementation

LogQuant is implemented as a derived class of the Cache class in Hugging Face's transformers library, ensuring seamless integration with existing inference pipelines. It utilizes the Quanto backend for 2-bit quantization with a Key-per-channel strategy.

Key Contributions

Observation of Log-Distributed Attention: The paper identifies that high-attention spike positions follow a log distribution, becoming sparser with distance from the current token. This insight forms the basis for estimating token importance.
Design of LogQuant: A 2-bit quantization technique that utilizes this log-distribution to selectively compress the KV cache. It outperforms existing methods (like KiVi and H2O) by better preserving important tokens.
Throughput Optimization: By disregarding absolute positions, the method optimizes the speed of the quantization process, achieving significant throughput gains.

Experimental Results

The authors evaluated LogQuant against KiVi and a 16-bit baseline across multiple model families (Llama3/3.1, Qwen1.5/2, Phi-3) and datasets (GSM8K, LongBench).

Accuracy Preservation:
- On complex tasks like Math (GSM8K) and Code Completion, LogQuant improves accuracy by 40% to 200% compared to KiVi at the same compression ratio (2-bit).
- For example, on the GSM8K dataset with Llama3.1-8B, LogQuant (2-bit) achieved an accuracy of 40.41 compared to KiVi's 18.04.
- Smaller models (e.g., Phi-3-mini, Qwen2-7B) typically suffer severe accuracy drops with 2-bit quantization; LogQuant provides a notable improvement for these architectures.
Efficiency Gains:
- Throughput: LogQuant achieves a 25% increase in throughput compared to the baseline.
- Batch Size: It supports a 60% increase in batch size within the same memory constraints (tested on NVIDIA H100).
- Memory: It delivers substantial memory savings while maintaining superior performance compared to existing 2-bit methods.

Significance and Claims

The paper claims that LogQuant offers a "groundbreaking" approach to KV cache management by shifting the paradigm from predicting token importance to leveraging the inherent log-distribution of attention spikes.

Superior Accuracy: The method claims to preserve "superior performance" and "superior accuracy" compared to prior art, particularly in challenging reasoning tasks where previous 2-bit methods failed.
Seamless Integration: The authors emphasize that LogQuant integrates effortlessly with popular inference frameworks (specifically Python's transformers library), making it immediately applicable for efficient LLM inference.
Resource Efficiency: The technique enables the deployment of LLMs in resource-constrained environments by significantly reducing memory footprints and increasing batch processing capabilities without compromising the quality of the output.

The authors conclude that while LogQuant currently achieves significant gains, future work will focus on refining the quantization approach and exploring operator fusion to further maximize efficiency and speed.