💬 NLP

DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity

DeltaKV is a residual-based KV cache compression framework that leverages long-range token similarity to reduce memory usage to 29% of the original, paired with a high-performance Sparse-vLLM engine to achieve up to 2 $\times$ throughput improvement in long-context scenarios.

Original authors: Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang, Jun Yu

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang, Jun Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a massive, thousand-page epic novel. To keep your story consistent, you need to remember everything you’ve written so far—every character's eye color, every plot twist, and every setting.

In the world of Artificial Intelligence (LLMs), this "memory" is called the KV Cache. The problem is that as the story gets longer, the memory required becomes so huge that it eventually crashes the computer's "brain" (the GPU).

Currently, AI researchers try to solve this in two ways:

The Eraser Method (Eviction): They just throw away old pages to save space. But if you throw away the page where you introduced the villain, the story stops making sense.
The Highlighter Method (Sparsity): They keep everything but only "look" at certain highlighted parts. This saves time, but it doesn't actually save much physical space in the library.

DeltaKV is a third, much smarter way. Here is how it works using three simple analogies:

1. The "Reference Photo" Strategy (Long-Range Similarity)

Imagine you are a detective looking at thousands of crime scene photos. Instead of memorizing every single pixel of every photo, you realize that many photos look almost identical.

Instead of storing 1,000 full photos, you pick 10 "Master Reference Photos." For every other photo, you don't save the whole image; you only write down a tiny note: "This photo is exactly like Reference Photo #3, but the suspect is wearing a red hat instead of a blue one."

DeltaKV does exactly this. It realizes that many parts of an AI's memory are repetitive. It saves a few "Reference Tokens" and, for everything else, it only stores the "Delta" (the difference). This is why it can shrink the memory usage to just 29% of its original size without losing the plot.

2. The "Sketch vs. Oil Painting" Concept (Highly Shared Components)

Think about a landscape painting. Most of the canvas is filled with "shared" things: the blue of the sky, the green of the grass, and the texture of the clouds. You don't need to describe the blue sky every time you mention a tree.

DeltaKV noticed that KV caches have "High-Norm Latent Components"—basically, a lot of "blue sky" that is common across all tokens. DeltaKV strips away all that common "background noise" and only stores the unique "details" (the specific tree or the specific bird). Because the "details" are much smaller and simpler than the "whole painting," they are incredibly easy to compress.

3. The "Express Lane" (Sparse-vLLM)

Even if you have a tiny, compressed notebook, you still need a fast way to read it. If you have to spend ten minutes decoding every tiny note, you'll be too slow to be useful.

The researchers built a specialized engine called Sparse-vLLM. Think of this as a high-speed scanner designed specifically to read "shorthand notes." Because it knows exactly how the DeltaKV notes are structured, it can skip the "full books" and jump straight to the "shorthand," allowing the AI to work up to 2x faster than before.

The Bottom Line

DeltaKV turns a massive, heavy library of full books into a slim, efficient folder of "Master References" and "Quick Shorthand Notes."

The Result: You can give the AI much longer "stories" (longer contexts) to read and write, it uses much less expensive hardware, and it does it all without forgetting who the villain is.

Technical Summary: DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity

1. Problem Statement

The deployment of Large Language Models (LLMs) for long-context applications (e.g., autonomous agents, complex reasoning, and long-document analysis) is severely limited by the KV cache memory bottleneck. As sequence length increases, the KV cache grows linearly, often exceeding the capacity of single GPU accelerators.

Existing solutions generally fall into two categories, both with significant drawbacks:

Token Eviction/Selection (Static/Dynamic): Methods like SnapKV or H2O discard tokens to save memory. However, static methods often discard tokens that become critical later in multi-turn dialogues, while dynamic methods (e.g., Quest) retain the full cache and rely on offloading, which introduces high PCIe latency.
Compression/Quantization: Methods like CacheGen or Palu often assume locality (similarity between nearby tokens) or require complex, GPU-unfriendly reconstruction pipelines that reduce throughput.

2. Key Empirical Observations

The authors identify two fundamental properties of KV caches that motivate their approach:

Long-Range Inter-Token Similarity: Semantic similarity is not just local. Over 60% of highly similar tokens are separated by more than 16 positions, meaning effective compression must leverage global, rather than local, context.
Highly Shared Latent Components: KV caches exhibit strong anisotropy. A small number of high-norm latent directions capture common linguistic patterns. When these shared components are subtracted, the remaining residuals are low-magnitude, noise-like, and much easier to compress.

3. Methodology: The DeltaKV Framework

DeltaKV is a residual-based compression framework that encodes only the token-specific deviations from a set of retrieved historical references.

Strided Reference Selection: To avoid the cost of searching the entire history, DeltaKV maintains a set of reference tokens $T$ selected at a fixed interval (stride $s$ ). For a new token, it retrieves the $k$ nearest neighbors from this strided set using L2 distance.
Residual Encoding: Instead of compressing the raw KV state, DeltaKV computes the mean of the retrieved references ( $KV^R$ ) and calculates the residual: $z_\Delta = f_c(KV) - f_c(KV^R)$ . A lightweight MLP/linear compressor ( $f_c$ ) encodes this residual into a low-dimensional space.
Reconstruction: During inference, the KV state is reconstructed via a decompressor ( $f_d$ ): $\widehat{KV} = f_d(z_\Delta) + KV^R$ .
Hybrid Training Objective: To ensure the compressed cache preserves generative capabilities, the modules are trained using a combination of Mean Squared Error (MSE) (for numerical fidelity) and Next-Token Prediction (NTP) loss (for semantic/task fidelity).
Sparse-vLLM Integration: To translate compression into speed, the authors developed Sparse-vLLM, an inference engine that supports irregular, sparse KV layouts. It allows for selective decompression, where only the tokens required by the sparse attention mask are reconstructed on demand.

4. Key Contributions

Discovery of Global Redundancy: Proving that KV cache redundancy is a global phenomenon and that residuals are highly compressible.
DeltaKV Framework: A GPU-friendly, residual-based compression method that achieves high compression ratios with minimal accuracy loss.
Sparse-vLLM Engine: A high-performance inference framework that decouples memory management from execution, enabling efficient handling of sparse and compressed KV caches.

5. Experimental Results

Memory Efficiency: DeltaKV reduces KV cache memory to 29% of the original size (when combined with 4-bit quantization) while maintaining near-lossless accuracy.
Accuracy: On LongBench, SCBench (multi-turn), and AIME (complex reasoning), DeltaKV matches or exceeds the performance of state-of-the-art dynamic selection methods like OmniKV.
Throughput: When integrated with Sparse-vLLM, it achieves up to 2× throughput improvement over standard vLLM in long-context scenarios (e.g., at 512k context length).
Generalization: The compressor, trained on short sequences (8k), generalizes effectively to extremely long contexts (up to 1M tokens).

6. Significance

DeltaKV provides a practical and scalable path for deploying long-context LLMs on existing hardware. By shifting the focus from "which tokens to keep" (eviction) to "how to represent the difference between tokens" (residual compression), it overcomes the fundamental trade-off between memory footprint and reasoning accuracy. This approach is particularly vital for the next generation of "infinite context" models used in complex, multi-step reasoning and agentic workflows.