💬 NLP

xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction

The paper introduces xKV, a post-training method that compresses KV-Cache memory by 8x and accelerates inference by up to 4.23x through joint factorization of aligned singular vectors across layers and selective reconstruction, offering a plug-and-play solution for efficient long-context LLM inference without requiring pretraining.

Original authors: Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Hung-Yueh Chiang, Yash Akhauri, Xilai Dai, Huiqiang Jiang, Yucheng Li, Luis Ceze, Kai-Chiang Wu, Mohamed S. Abdelfattah

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Hung-Yueh Chiang, Yash Akhauri, Xilai Dai, Huiqiang Jiang, Yucheng Li, Luis Ceze, Kai-Chiang Wu, Mohamed S. Abdelfattah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Memory Overload"

Imagine a Large Language Model (LLM) as a very smart librarian who is reading a massive book to answer your questions.

The Context: If the book is short, the librarian can easily remember the whole story.
The Long Context: If the book is 1 million pages long (like a whole encyclopedia), the librarian needs to keep a massive "cheat sheet" (called the KV-Cache) in their head to remember what they've read so far.
The Issue: As the book gets longer, this cheat sheet becomes so huge that it fills up the librarian's entire brain (the computer's memory). Once the brain is full, the librarian can't read more books at the same time, and the whole system slows down or crashes.

The Old Solutions: Why They Didn't Work Perfectly

Researchers tried to shrink this cheat sheet before, but they had two main problems:

The "Copy-Paste" Method (Token Eviction): They tried to throw away pages they thought were unimportant. Problem: Sometimes the "unimportant" page holds the key to the answer, so the librarian starts making mistakes.
The "Merge" Method (Cross-Layer Merging): They tried to combine the notes from different chapters because they looked similar. Problem: They only looked at the surface level (like comparing the first word of a sentence). They missed the deeper structure, so the merged notes became messy and inaccurate.

The New Discovery: The "Hidden Blueprint"

The authors of this paper discovered something surprising about how the librarian's brain works.

The Observation: Even though the specific words (tokens) in Chapter 1 look different from the words in Chapter 2, the underlying structure of the notes is actually almost identical.
The Analogy: Imagine two different architects (layers) designing two different rooms. If you look at the furniture (the specific words), they look totally different. But if you look at the blueprint (the dominant singular vectors), both architects are using the exact same structural grid. They are just painting the grid different colors.
The Tool: The authors used a mathematical tool called CKA (Centered Kernel Alignment) to prove that these "blueprints" are perfectly aligned across different layers of the model.

The Solution: xKV (The "Shared Blueprint" System)

Instead of making the librarian write out a full cheat sheet for every single chapter, xKV does this:

Find the Shared Blueprint (Cross-Layer Factorization):
The system looks at a group of 4 chapters at a time. It realizes they all share the same "skeleton" or "blueprint." It extracts this one shared blueprint and stores it once.
- Analogy: Instead of writing out the full recipe for 4 different cakes, you just write down the shared "flour and sugar base" once, and then only write down the tiny list of unique toppings for each cake.
Reconstruct Only What You Need (Selective Reconstruction):
When the librarian needs to answer a question, they don't need to rebuild the entire cheat sheet. They only need to rebuild the specific parts relevant to the current question.
- Analogy: If you ask, "What was the color of the car in Chapter 5?", the system doesn't rebuild the whole book. It just quickly reconstructs the specific sentence about the car using the shared blueprint.

The Results: Faster, Smaller, and Smarter

By using this "Shared Blueprint" approach, the paper claims:

Massive Memory Savings: They can shrink the cheat sheet by up to 8 times (8x) without losing accuracy.
Speed Boost: Because the memory is smaller, the librarian can work much faster. They achieved up to 4.23 times faster generation speeds compared to the standard method.
Plug-and-Play: You don't need to retrain the librarian from scratch. You can just apply this method to existing models (like Llama-3 or Qwen) and it works immediately.

Summary

Think of xKV as a smart filing system. Instead of keeping a separate, full file for every single page of a massive book, it realizes that many pages share the same underlying structure. It keeps just one master template for a group of pages and only fills in the specific details when asked. This saves huge amounts of space and makes the whole process much faster, all while keeping the answers accurate.

Technical Summary: xKV - Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction

1. Problem Statement

Long-context Large Language Models (LLMs) enable advanced applications such as large-scale information retrieval and codebase debugging but face a critical bottleneck: the memory consumption of Key-Value (KV) caches during inference. As context lengths extend to hundreds of thousands or millions of tokens, the KV-Cache memory footprint becomes prohibitive, severely limiting concurrent inference requests and throughput.

Existing mitigation strategies generally fall into two categories:

Intra-layer methods: Techniques like quantization, token eviction, and single-layer low-rank decomposition compress the KV-Cache of each layer independently. While effective to a degree, they fail to exploit potential redundancy across layers.
Cross-layer methods: Approaches like Cross-Layer Attention (CLA) and YOCO require expensive architectural modifications and pretraining. Others, like MiniCache, rely on post-hoc merging of adjacent layers based on token-wise cosine similarity. However, empirical analysis shows that token-wise similarity between layers is often too weak to support robust compression without significant accuracy degradation.

2. Methodology: xKV

The authors propose xKV, a post-training, plug-and-play compression method that exploits deep structural redundancy across layers without requiring model retraining or architectural changes.

2.1. Core Observation: Aligned Singular Vectors

The paper challenges the assumption that layers are distinct based on token-wise cosine similarity. Using Centered Kernel Alignment (CKA), the authors demonstrate that while individual token embeddings may diverge, the dominant singular vectors (the basis defining the principal variations in the token space) are highly aligned across adjacent layers. This high CKA score indicates that the geometric structure of the KV-Cache subspaces is shared, even if point-wise similarities are low.

2.2. Cross-Layer Factorization (CLF)

Leveraging this alignment, xKV partitions the model's $N$ layers into contiguous windows of size $W$ . For each window, it performs a Cross-Layer Factorization:

Concatenation: The KV-Caches of all layers within a window are horizontally concatenated.
SVD: A rank- $r$ Singular Value Decomposition (SVD) is computed on the concatenated cache.
Decomposition: The result yields a shared token basis ( $A_k$ ) for the window and layer-specific reconstruction matrices ( $B_\ell$ ).
Storage: Instead of storing $W$ full-rank caches, the system stores one shared basis and $W$ smaller reconstruction matrices, reducing memory complexity from $O(W \cdot L \cdot d)$ to $O(L \cdot r + W \cdot r \cdot d)$ .

2.3. Selective Reconstruction (SR)

To address the computational bottleneck of reconstructing all tokens during the decode phase (which scales linearly with sequence length $L$ ), xKV introduces **Selective Reconstruction (SR).

Mechanism: Instead of dense reconstruction, SR identifies a sparse subset of tokens ( $S$ ) most relevant to the current query using landmark-guided chunk selection (similar to ShadowKV).
Execution: Only the rows corresponding to these critical tokens are reconstructed using the shared basis and layer-specific matrices.
Benefit: This reduces the reconstruction cost to be constant relative to sequence length, enabling end-to-end speedups.

3. Key Contributions

Discovery of Cross-Layer Redundancy: The paper reveals that KV-Caches share highly aligned dominant singular vectors across layers, a form of redundancy obscured by low token-wise similarity.
xKV Algorithm: A training-free method that extracts a shared basis via Cross-Layer Factorization (CLF), enabling effective information sharing across $W$ layers ( $W > 2$ ).
Selective Reconstruction: A decoding-time optimization that reconstructs only query-relevant tokens, eliminating the linear computational overhead of dense reconstruction.
Plug-and-Play Compatibility: The method requires no fine-tuning or architectural modifications, making it applicable to existing pretrained models (e.g., Llama-3.1, Qwen2.5, DeepSeek-V2).

4. Experimental Results

The authors evaluated xKV on Llama-3.1-8B-Instruct, Qwen2.5-7B/14B-Instruct, and DeepSeek-Coder-V2-Lite using the RULER, LongBench, and Multi-turn NIAH benchmarks.

Compression vs. Accuracy:
- xKV achieves up to 8× KV-Cache compression with minimal accuracy loss (≤3% on average) compared to full attention baselines.
- On Llama-3.1-8B, xKV at 8× compression achieves 88.50% average accuracy on RULER, significantly outperforming MiniCache (45.04% at 1.3×) and matching or exceeding token eviction methods like SnapKV and PyramidKV.
- In multi-turn NIAH tasks, xKV maintains stable performance across conversation turns, whereas token eviction methods suffer steep declines after the first turn.
Efficiency and Throughput:
- Memory: xKV enables larger batch sizes under the same GPU memory budget.
- Latency: Dense reconstruction reduces attention latency by up to 3.6×.
- Throughput: Combined with Selective Reconstruction (xKV-SR), the system achieves up to 4.23× end-to-end speedup over full attention baselines on NVIDIA A100 GPUs.
- Comparison: xKV-SR surpasses ShadowKV by 30% in throughput under similar accuracy levels while keeping both Keys and Values on the GPU, avoiding PCIe transfer bottlenecks.
Robustness: The method remains effective across different model architectures, including Grouped-Query Attention (GQA) and Multi-head Latent Attention (MLA) in DeepSeek-V2.

5. Significance and Claims

The paper claims that xKV provides a practical, plug-and-play approach to simultaneously reduce both memory footprint and latency for long-context LLM inference. By shifting the focus from token-wise similarity to singular vector alignment, xKV unlocks unexploited inter-layer redundancy that previous methods missed.

The authors emphasize that their approach:

Avoids the high cost of pretraining required by architectural changes.
Overcomes the accuracy limitations of similarity-based merging (MiniCache).
Solves the computational bottleneck of dense low-rank reconstruction via Selective Reconstruction.
Enables fully on-device (GPU) inference for compressed caches, bypassing the PCIe bandwidth limits that constrain offloading-based methods like ShadowKV.

The work is presented as a foundational step toward efficient long-context inference, with code made publicly available to facilitate adoption.