🤖 machine learning

CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM

CONF-KV is a novel KV cache eviction strategy that dynamically adjusts cache budgets based on the model's per-step prediction confidence and employs mixed-precision storage to significantly reduce memory usage while maintaining high retrieval accuracy and task performance for long-horizon LLM inference.

Original authors: Yubo Li, Yidi Miao

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yubo Li, Yidi Miao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to read a very long book, but your brain (the computer's memory) can only hold a few pages at a time. As you read, you need to remember what happened earlier to understand the current sentence. If you forget too much, you get confused. If you try to remember everything, your brain gets so full it slows down to a crawl.

This is the exact problem CONF-KV solves for AI models (Large Language Models) when they are writing long stories or having long conversations.

Here is how the paper explains it, using simple analogies:

The Problem: The "Overfilled Backpack"

When an AI generates text, it keeps a "backpack" of information called the KV Cache. This backpack holds the context of everything the AI has said so far.

The Issue: As the conversation gets longer, the backpack gets heavier. Eventually, it becomes so heavy that the AI runs out of memory (the backpack rips) or gets so slow it takes forever to think.
The Old Way: Most AI systems use a "Sliding Window." Imagine a window on a train. As the train moves, the view outside changes. The AI only remembers the last 512 words (the view right outside the window) and forgets everything before that.
- The Flaw: If the answer to a question was mentioned 1,000 words ago, the sliding window forgets it completely, and the AI fails.
The Other Old Way: Some systems try to remember "important" words based on how often they were looked at in the past. But this is like looking at a map of where you were yesterday, not knowing how you feel right now.

The Solution: The "Confidence Meter"

The authors of this paper, CONF-KV, introduced a new way to manage the backpack. Instead of using a fixed rule, they give the AI a Confidence Meter.

Think of the AI as a student taking a test:

When the student is confident: They know the answer easily. They don't need to look back at their notes. So, the system says, "Great! You're sure. Let's throw away some old notes to save space."
When the student is confused: The student is hesitating. They need to check their history to figure it out. The system says, "Wait, you seem unsure. Keep all the notes! Don't throw anything away yet."

How it works in practice:

The Signal: Before the AI picks the next word, it looks at how sure it is. If the next word is obvious (high confidence), it shrinks the memory. If the next word is tricky (low confidence), it expands the memory.
The Sorting: Even when it needs to shrink, it doesn't just delete random things. It keeps the most recent words (because they are usually important) and the words the AI has looked at the most in the past. It deletes the "boring" old stuff that nobody cares about.

The "Mixed-Precision" Trick

The paper also mentions a clever storage trick.

Imagine your notes are written on paper.
Recent notes are written in high-quality, thick paper (FP16) so they are crystal clear.
Older notes are written on thin, recycled paper (INT8). They take up much less space, but you can still read them well enough to get the gist.
This allows the AI to fit a lot more history into the same amount of backpack space without losing too much quality.

What the Results Show

The authors tested this on four different AI models and found:

Memory Savings: It uses about the same amount of memory as a simple "sliding window" (forgetting everything old), but it remembers way more important details.
Better Accuracy: In a "Needle in a Haystack" test (finding one specific fact hidden in a huge text), CONF-KV found the needle 91.4% of the time. The old sliding window only found it 53.8% of the time.
Real-World Tasks: When used as a web-browsing agent (trying to buy things or fill out forms online), it succeeded 95.3% as often as the full-memory version, but used 2.8 times less memory.
Speed: Because the backpack is lighter, the AI thinks faster (lower latency) and can handle more users at once (higher throughput).

The Bottom Line

CONF-KV is like a smart librarian who doesn't just throw away old books because they are "old." Instead, the librarian watches the reader. If the reader is struggling, the librarian keeps the whole library open. If the reader is cruising along, the librarian clears out the clutter to make the room faster.

This allows AI to have longer, smarter conversations without running out of memory or slowing down, simply by listening to how "sure" the AI feels at every single step.

Technical Summary: CONF-KV

Problem Statement
Long-horizon Large Language Model (LLM) inference faces a critical bottleneck where the Key-Value (KV) cache becomes the dominant consumer of GPU memory, causing per-token attention latency to scale linearly with sequence length. Existing eviction policies, such as sliding windows or historical attention-based methods (e.g., H2O, SCISSORHANDS), rely on static recency or past signals to determine which tokens to retain. These approaches often fail to utilize a signal available at every decoding step: the model's current uncertainty. Consequently, static policies may prematurely evict context during difficult generation spans or retain unnecessary tokens when the model is confident, leading to a suboptimal trade-off between memory usage and generation quality.

Methodology: CONF-KV
The authors propose CONF-KV, a confidence-aware KV cache manager that dynamically adjusts the cache budget based on the model's current next-token distribution. The system operates through the following mechanisms:

Confidence Estimation: At each decoding step, the method converts the logits into a scalar confidence score ( $c$ ). This score is a weighted combination of normalized entropy, the log-probability margin between the top two tokens, and the probability of the top token.
$c = w_H(1 - \hat{H}) + w_m\sigma(m) + w_p p_{(1)}$
where $\hat{H}$ is normalized entropy, $m$ is the log-margin, and weights are set to $(0.4, 0.3, 0.3)$ .
Adaptive Budget Selection: A threshold $\tau$ determines the cache budget ( $N$ ) for the current step.
- If $c \ge \tau$ (high confidence), a tight budget ( $N_{high}$ ) is selected, allowing for aggressive eviction.
- If $c < \tau$ (low confidence/uncertainty), a looser budget ( $N_{low}$ ) is selected to retain more context.
  This creates a "sawtooth" memory profile where the cache expands during uncertain spans and contracts during confident ones.
Token Ranking and Eviction: Within the selected budget, tokens are ranked by a composite score combining Exponential Moving Average (EMA) attention mass and recency. A hard protected window ( $P$ ) ensures the most recent tokens are never evicted to preserve local coherence.
System Integration:
- Mixed-Precision Storage: The most recent $W$ tokens are stored in FP16, while older retained tokens are quantized to INT8 per head and channel. This reduces memory footprint while minimizing perplexity degradation compared to lower-bit quantization.
- Pyramidal Budgets (CONF-KV-L): An optional variant allocates budgets non-uniformly across layers, assigning smaller budgets to deeper layers based on the observation that deeper layers often concentrate information into fewer tokens.
- Implementation: The manager uses contiguous compaction to maintain a dense layout for attention kernels, avoiding indirection tables, and fuses dequantization into blockwise attention reads.

Key Contributions

Confidence-Aware Policy: A novel eviction strategy that uses the current output distribution to drive per-step cache budget decisions, rather than relying solely on historical signals.
Systems Design: A complete implementation combining adaptive eviction with blockwise online-softmax attention, mixed FP16/INT8 storage, and optional pyramidal layer budgets.
Mechanistic Validation: A test demonstrating that the confidence score strongly anticorrelates (Pearson $r = -0.77$ ) with the KL shift induced by ablating recent context, validating the assumption that low confidence indicates a need for more retained context.
Comprehensive Evaluation: Extensive benchmarks across four model families (GPT-2, Qwen-14B/32B, gpt-oss-20b) and diverse workloads, including perplexity, long-context retrieval, and web-agent tasks.

Experimental Results

Memory-Quality Trade-off: On GPT-2 with a 2048-token generation length, CONF-KV+INT8 matches the memory footprint of a fixed 512-token sliding window (~38.7 MB) but reduces perplexity by 3.11 points compared to the sliding window. CONF-KV-L (pyramidal variant) further reduces memory to 34.2 MB while improving perplexity by 3.89 points, closing 74% of the quality gap between a sliding window and full KV.
Long-Context Retrieval: In Needle-in-a-Haystack (NIAH) tests up to 32K tokens, CONF-KV achieves 91.4% retrieval accuracy, significantly outperforming sliding windows (53.8%) and H2O (80.6%).
Agent Tasks: On 75 VisualWebArena tasks, CONF-KV retains 95.3% of the full-KV success rate while reducing peak memory by 2.8×.
Throughput and Latency: CONF-KV reduces per-step latency by 1.8× compared to full KV on GPT-2 and Qwen-32B. Throughput scales with batch size; at batch size 8, CONF-KV achieves 2.06× the throughput of full KV.
Ablation: Isolating the confidence signal shows that random eviction at the same rate degrades performance to 36.54 PPL, whereas full CONF-KV achieves 30.92 PPL, confirming that both the confidence gating and the ranking mechanism are essential.

Significance and Claims
The paper claims that CONF-KV demonstrates that current uncertainty is a useful systems metadata for improving the memory-quality Pareto frontier. The method is distinct because it is forward-looking: it cannot recover evicted tokens but prevents premature eviction during difficult spans. The approach is designed to be training-free, does not alter model weights, and is orthogonal to existing techniques like head-wise allocation, layer-wise allocation, and precision selection, allowing it to compose with them.

The authors note limitations, including that the method is a "no-op" on short contexts that never exceed the eviction threshold and that contiguous compaction incurs a gather cost (though this is outweighed by attention savings). They also acknowledge that the confidence signal becomes less informative under high-temperature sampling. Finally, the paper highlights a dual-use consideration: while lower inference costs improve efficiency, they may also lower barriers to undesirable long-horizon automation, suggesting that safety controls for the underlying models remain necessary.