💬 NLP

FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration

FastKV is a novel KV cache compression framework that decouples prefill compute reduction from decoding KV budget by leveraging token importance stabilization in later layers, thereby achieving significant speedups in both prefill and decoding stages without sacrificing accuracy.

Original authors: Dongwon Jo, Jiwon Song, Yulhwa Kim, Jae-Joon Kim

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongwon Jo, Jiwon Song, Yulhwa Kim, Jae-Joon Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a brilliant detective (the AI) trying to solve a massive mystery. You are handed a stack of evidence that is 128,000 pages long.

To solve the case, you have two main jobs:

The Briefing (Prefill): You must read the entire stack of evidence to understand the story.
The Interrogation (Decoding): You ask questions one by one, looking back at your notes to find the answers.

The Problem: The "Too Much Stuff" Bottleneck

In the past, AI models tried to solve this by reading every single page of the 128,000-page stack.

The Briefing Problem: Reading 128,000 pages takes forever. The time it takes grows exponentially (if you double the pages, it takes four times as long to read).
The Interrogation Problem: After reading, you have to keep a giant notebook (the KV Cache) with notes on every single page. When you ask a question, you have to flip through this massive notebook every time. It's slow, and your desk (computer memory) gets cluttered.

Previous solutions tried to fix one side of the problem but broke the other:

The "Skip the Reading" approach: Some tried to only read the first and last few pages. This made the briefing fast, but the detective missed crucial clues in the middle, leading to wrong answers.
The "Shrink the Notebook" approach: Others read everything but threw away most of the notes immediately. This made the interrogation fast, but the briefing was still painfully slow because they still had to read all 128,000 pages first.

The Solution: FastKV (The Smart Detective)

The paper introduces FastKV, a new way to handle this that fixes both problems without losing accuracy. It uses two clever tricks based on how human brains (and AI layers) actually work.

Trick 1: The "Deep Dive" vs. "The Highlight Reel" (Token-Selective Propagation)

Imagine you are reading a 128,000-page novel.

Early Layers (The First 15 Chapters): When you first start reading, you don't know what's important yet. You need to read the whole story to understand the context. If you skip pages now, you might miss the villain's introduction.
Later Layers (The Final Chapters): Once you've read the first half, you realize the story is mostly about the detective and the butler. The other 100,000 pages of background scenery aren't needed anymore.

FastKV's Move: It lets the AI read the entire story for the first half of the process (so it doesn't miss anything). But once it hits a specific "switch point" (around the middle), it stops passing the whole story to the next stage. Instead, it passes only a Highlight Reel of the most important characters and events.

Analogy: Think of it like a movie director. The first assistant director reads the whole script to understand the plot. The second assistant director only gets a list of the 50 most important scenes to focus on. The second director saves time, but because the first director did the full reading, the story is still perfect.

Trick 2: The "Two-Step Cleanup" (Decoupling)

This is the paper's biggest breakthrough. Previous methods were like a strict teacher who said: "If you only read 50% of the book, you can only keep notes on 50% of the pages." This forced a trade-off: read less to save time, but lose accuracy.

FastKV's Move: It breaks the rule. It says: "You can read the whole book (or most of it) to get the context right, but you are allowed to throw away 90% of your notes before you start the interrogation."

Analogy: Imagine you are packing for a trip.
- Old Way: You only pack what you can carry. If you have a small suitcase, you can't bring your favorite jacket, even if you saw it in the store.
- FastKV Way: You go to the store and look at everything (the full context) to make sure you know what exists. But when you pack your bag for the trip (the decoding stage), you only pack the absolute essentials. You don't have to carry the whole store with you.

The Result: Why It Matters

By using these two tricks, FastKV achieves the "Holy Grail" of AI efficiency:

Faster Briefing: It skips reading the boring parts of the middle and end of the story, making the initial setup 1.8x faster.
Faster Interrogation: It keeps a tiny, super-efficient notebook, making the answering process 2.8x faster.
No Accuracy Loss: Because it read the whole story at the beginning, it didn't miss any clues. It answers just as accurately as if it had read and remembered everything.

In a Nutshell

FastKV is like a smart librarian who realizes that while you need to scan the whole library to find the right books, you don't need to carry the entire library with you to the reading table. By separating "finding the info" from "storing the info," it makes long conversations with AI faster, cheaper, and just as smart as before.

1. Problem Statement

Large Language Models (LLMs) with extended context windows (e.g., 128K+ tokens) face severe computational and memory bottlenecks in two distinct stages:

Prefill Stage: Attention computation scales quadratically ( $O(N^2)$ ) with input length, making the processing of long prompts computationally expensive.
Decoding Stage: The Key-Value (KV) cache grows linearly with context length. Every generated token must access this expanding cache, leading to high memory bandwidth usage and reduced throughput.

Limitations of Existing Solutions:

Decoding-Only Methods (e.g., StreamingLLM, SnapKV): These prune the KV cache after the full context is processed. While they accelerate decoding, they do not reduce the quadratic prefill cost.
Prefill-Aware Methods (e.g., GemFilter, PyramidInfer): These attempt to reduce the context during prefill to generate a smaller KV cache. However, they suffer from a fundamental trade-off:
- They tightly couple the amount of prefill compute reduction with the final KV cache size.
- Aggressive early pruning (to save prefill time) often discards tokens that early layers need but later layers might find critical, leading to significant accuracy degradation.
- They often fail to account for the layer-dependent dynamics of token importance.

2. Methodology: FastKV

FastKV introduces a framework that decouples context reduction (prefill compute) from KV cache retention (decoding memory). It is built on two key observations:

Layer-Dependent Context Dynamics: In early layers, the set of "critical" tokens is unstable and shifts rapidly. In later layers, the set of critical tokens stabilizes and overlaps significantly across layers.
Sparse Context Utilization: During decoding, attention is highly sparse; only a small fraction of tokens (e.g., top 0.38%) dominate the attention scores, even for very long contexts.

Based on these insights, FastKV employs two core techniques:

A. Token-Selective Propagation (TSP)

Instead of pruning tokens immediately, FastKV uses a two-stage prefill strategy:

Stage 1 (Early Layers): The model processes the full context through the initial layers. This ensures that early layers, which have unstable attention patterns, can access all tokens to build robust representations.
Stage 2 (TSP Layer & Beyond): At a specific TSP layer (typically in the middle of the model), the system evaluates token saliency.
- Saliency is calculated by averaging attention weights from a recent "window" of tokens (queries) to all context tokens.
- Only the top-ranked tokens (plus the window tokens) are propagated to subsequent layers.
- This reduces the computational load for the remaining layers without discarding information that early layers needed.

B. Decoupled KV Retention

FastKV introduces independent control over two hyperparameters:

TSP Rate: Controls how many tokens are propagated after the TSP layer (affecting prefill compute).
KV Retention Rate: Controls how many KV entries are kept at each layer for the decoding stage.

Key Innovation: Unlike previous methods where reducing the prefill context automatically shrinks the KV cache, FastKV allows the KV cache to be compressed independently. For example, one can propagate a moderate amount of context to preserve accuracy (TSP) while aggressively pruning the KV cache for decoding efficiency (KV Retention).

3. Key Contributions

Decoupling Mechanism: The first framework to separate the decision of which tokens to process (prefill) from which tokens to cache (decoding), breaking the rigid trade-off found in prior art.
Layer-Aware Strategy: It leverages the observation that token importance stabilizes in deeper layers, allowing for safe pruning only after a specific "TSP layer" where the model has converged on critical tokens.
Two-Stage Prefill: A novel workflow that retains full-context computation in early layers to prevent information loss, switching to selective propagation only when attention patterns stabilize.
Efficiency-Accuracy Balance: Achieves significant speedups in both stages without the accuracy degradation typically seen in aggressive prefill-aware methods.

4. Experimental Results

The authors evaluated FastKV on LLaMA-3.1-8B-Instruct and Ministral-8B-Instruct using LongBench, RULER, and Needle-in-a-Haystack benchmarks.

Speedup:
- Prefill: Up to 1.82× faster compared to full-context baselines.
- Decoding: Up to 2.87× faster compared to full-context baselines.
- At 128K context length, FastKV achieves >2× end-to-end speedup over the baseline.
Accuracy:
- FastKV maintains accuracy comparable to full-context baselines (drop < 1% on LongBench).
- It significantly outperforms GemFilter and PyramidInfer, which suffer from large accuracy drops (up to 11.58% for GemFilter) due to their coupled pruning strategies.
- In Needle-in-a-Haystack, FastKV achieved a score of 99.9 (vs. 99.0 for full-context), suggesting that selective propagation can sometimes help the model focus on critical information.
Comparison:
- vs. Decoding-only (SnapKV): FastKV is faster in prefill while matching accuracy.
- vs. Prefill-aware (GemFilter): FastKV is slightly slower in prefill (due to processing full context in early layers) but vastly superior in accuracy.

5. Significance

FastKV addresses a critical bottleneck in the deployment of long-context LLMs. By recognizing that early layers need full context while later layers and decoding only need sparse context, it provides a flexible optimization knob for system designers.

Scalability: It enables efficient inference as context windows grow beyond 128K, where memory bandwidth and quadratic compute become prohibitive.
Flexibility: Users can tune the TSP rate and KV retention rate independently based on their specific constraints (e.g., prioritizing low latency vs. low memory usage).
Orthogonality: The method is compatible with other optimization techniques like quantization and sparse attention kernels, offering a path toward universally efficient LLM serving.

In summary, FastKV demonstrates that timing is as crucial as selection in KV cache compression. By delaying context reduction until token importance stabilizes, it achieves the "holy grail" of long-context inference: fast prefill, fast decoding, and high accuracy simultaneously.