🤖 AI

KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models

This paper introduces KVDiagnosis, a comprehensive diagnostic benchmark and dataset that systematically evaluates KV-cache compression methods by categorizing them into a taxonomy, isolating specific failure cases through controlled comparisons, and providing detailed measurements to identify why compressed models fail while maintaining high coverage and enabling targeted interventions.

Original authors: Chen Qiu, Ziwu Liu, Chao Fei, Guozhong Li, Panos Kalnis

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Chen Qiu, Ziwu Liu, Chao Fei, Guozhong Li, Panos Kalnis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to remember a massive, thousand-page story so you can answer a question about it. Your brain (or in this case, a super-smart computer program called a "Large Language Model") has to keep the whole story in its short-term memory while it reads. This memory is called the KV-cache. Think of it like a giant whiteboard where the computer writes down every single word it has read so far, along with notes on how important each word is. The problem is, as the story gets longer, this whiteboard gets so huge that it runs out of space, causing the computer to crash or slow down to a crawl.

To fix this, scientists have invented "compressors." These are like magical erasers that try to wipe away parts of the whiteboard that seem unimportant, keeping only the most critical notes. But here's the tricky part: sometimes, the computer erases the wrong thing. It might delete a crucial clue needed to solve a mystery, and then give you a completely wrong answer. Until now, we mostly just knew that the answer was wrong, but we didn't know why or which part of the memory was erased to cause the mistake. It was like knowing your car broke down, but having no idea if it was the engine, the tires, or the fuel pump.

This paper introduces a new tool called KVDiagnosis, which acts like a super-detailed mechanic's report for these AI memory compressors. Instead of just saying "the car is broken," KVDiagnosis looks under the hood to see exactly which piece of the memory was lost, how much the computer's confidence dropped, and whether the error happened because the computer forgot a fact or because it got confused while writing the answer. The researchers tested this on a popular AI model and found that while some compressors are great at saving space, they often fail in very specific, surprising ways that total scores hide. They discovered that simply looking at the final grade isn't enough; you need to know which memories were sacrificed to understand the failure.

The Big Idea: Why "Good Enough" Isn't Good Enough

Imagine you have a library of 2,600 different stories, and you ask an AI to answer questions about them. If you tell the AI to use a "compressed" memory (a smaller, eraser-filled whiteboard), it might get the answer right 90% of the time. That sounds great, right? But what if it gets the wrong answer on the exact same story that a different compressor got right?

The authors of this paper argue that just looking at the average score (like a class average) is misleading. It's like saying two students both got a B on a test, but one failed because they forgot the dates of history, while the other failed because they couldn't spell the names. If you only look at the grade, you don't know how to help them improve.

The paper builds on the idea that KV-cache compression is necessary to let AI handle long stories, but current ways of testing it are too blunt. They don't tell us what changed when the AI made a mistake. Did the AI forget the evidence? Did it keep the evidence but misunderstand it? Or did it keep everything but get confused while writing the final sentence?

The Solution: A Diagnostic Toolkit

The researchers created KVDiagnosis, a massive dataset and testing framework designed to be a "detective's kit" for AI failures. Here is how they built it:

The "Full Memory" Baseline: First, they let the AI read the entire story without any compression (the "FullCache" mode) and recorded the correct answers. This is the gold standard.
The Compression Test: Then, they ran the same stories through 25 different types of memory compressors (the "magic erasers").
The "C→W" Filter: They specifically looked for the cases where the AI got the answer Correct with full memory but Wrong with compression. They call these C→W rows.
The Deep Dive: For every single one of these mistakes, they didn't just record the wrong answer. They recorded everything:
- Cache Retention: Did the AI actually keep the specific words needed for the answer?
- Likelihood Drift: Did the AI's confidence in the right answer drop significantly?
- Attention: Did the AI's "eyes" (attention) look at the right parts of the story?
- Decoding: Did the AI get confused while writing the final words?

What They Found: The Surprises

The team ran 59,800 compressed tests on 2,600 different sources. Here is what the "mechanic's report" revealed:

1. Most failures are due to "Lost Evidence"
The most common reason for a mistake was that the compressor simply deleted the part of the story that held the answer. About 40.3% of the failures happened because the AI had "low mapped coverage"—it literally didn't have the evidence anymore. Another 22.9% had "partial coverage," meaning it kept some of the clues but not enough to solve the puzzle.

2. The "Ghost" Failures
Some failures were weirder. In 17.0% of the cases, the AI kept the positions of the words (it knew where the evidence was), but the content of the memory was garbled or changed. It's like having a map that points to the right street, but the street name on the sign has been painted over. The AI knew where to look, but the information it found was wrong. This happened with methods that change the quality of the memory (like turning high-definition notes into low-resolution sketches) rather than just deleting them.

3. Similar Scores, Different Failures
Two compressors might both get a score of 85%, but they fail on completely different stories. The researchers found that when comparing two popular compressors (SnapKV and TOVA), their failures only overlapped by about 38% at most. This means you can't just pick the one with the highest average score; you have to know which stories it is bad at.

4. A Targeted Fix Works
The most exciting finding came from a specific type of failure called "Low-EAR" (Low Evidence Attention Retention). This is when the AI kept the evidence but didn't "look" at it hard enough. The researchers tried a simple fix: they artificially boosted the AI's attention to the evidence.

The Result: This fix repaired 29.2% of these specific failures.
The Control: When they tried the same boost on random, non-evidence words (a "sham" intervention), it only fixed 6.3% of the problems.
This proves that for this specific type of error, the problem wasn't that the memory was gone, but that the AI wasn't paying attention to it.

The Takeaway

The paper concludes that we cannot just rank compressors by their average scores. A "good" compressor for one type of task might be terrible for another. The KVDiagnosis benchmark shows us that:

63.2% of failures are due to low or partial memory coverage.
Only 0.2% of failures were cases where the memory was perfect, but the AI's confidence drifted wildly (a very rare "high mapped-coverage drift").
19 specific rows showed that even when the AI kept the evidence perfectly, it still got the answer wrong due to other reasons.

By using this diagnostic approach, researchers can stop guessing why AI fails and start fixing the specific mechanism that broke. It's the difference between saying "the car is broken" and saying "the fuel pump is clogged, so we need to clean that specific part." The authors made all their data and code available so others can use this same "diagnostic toolkit" to build better, more reliable AI for long stories.

Technical Summary: KVDiagnosis

1. Problem Statement

Long-context language models face a critical bottleneck: the memory required for KV-cache storage grows linearly with sequence length and concurrent requests, limiting practical inference. While KV-cache compression techniques (e.g., token eviction, quantization, channel pruning) offer a solution, they introduce a significant evaluation challenge. When a compressed model fails to produce a correct answer that a full-cache model would have generated, standard aggregate task scores and memory-footprint statistics fail to identify the specific cause of the failure.

The paper argues that different compression mechanisms transform different objects (e.g., token positions vs. key channels vs. semantic chunks), meaning a single metric like "50% compression" has distinct physical meanings across methods. Furthermore, existing benchmarks often lack:

Per-source paired controls: Comparing compressed outputs against a FullCache baseline for the exact same input.
Method-specific failure sets: Selecting failure cases (Correct FullCache $\to$ Wrong Compressed, or C $\to$ W) independently for each method to avoid distribution shifts.
Valid diagnostic traces: Linking failures to specific measurements (cache retention, likelihood drift, attention access) that are applicable to the specific mechanism used.

Without these, it is impossible to distinguish whether a failure stems from deleted evidence, damaged representations, weakened access, or decoding artifacts.

2. Methodology: KVDiagnosis Benchmark

The authors introduce KVDiagnosis, a diagnostic dataset and benchmark designed to isolate and analyze compression failures.

Benchmark Design

Taxonomy and Scope: The work synthesizes a taxonomy of 25 representative methods grouped into five mechanism families: Token Eviction/Position Selection, Budget Allocation, Query-aware Access/Channel Pruning, Tensor/Representation Compression, and Chunk/Semantic Compression. From these, eight verified implementations spanning all five families are selected for evaluation.
Paired Execution Matrix: The benchmark executes a FullCache control once per source. Every supported method-setting cell then runs on the same fixed split of sources using the same model, prompt, tokenizer, and decoder.
Failure Selection (C $\to$ W): Only after all runs are complete does the benchmark extract C $\to$ W rows (where FullCache was correct and compression was wrong) for each method-setting independently. This ensures no compressor defines another's test set.
Record Format: A common format links outputs and metadata to specific measurements:
- Cache Retention: Measured token coverage (ERR, ECov) or projected chunk-to-token coverage. For methods like ThinK and QuantizedCache where positions are structurally addressable but representation fidelity is unknown, this is marked as "structural position addressability."
- Likelihood Drift: Changes in Negative Log-Likelihood (NLL), Generalized Probability Ratio (GPR), and Top-50 overlap between FullCache and compressed teacher-forced distributions.
- Access and Decoding: Evidence Attention Mass (EAM), Evidence Attention Retention (EAR), and decoding output metrics.

Experimental Setup

Model: Primary experiments use Qwen3-8B. Cross-model validation uses Falcon3-7B and Mistral-Small-24B.
Datasets: Four evidence-aware workloads: RULER-8K, RULER-16K (controlled retrieval/tracking), Qasper, and HotpotQA (evidence-annotated QA).
Settings: Methods are evaluated at three compression tiers (e.g., 75%, 50%, 25% retention or 8-bit/4-bit/2-bit quantization).

3. Key Contributions

Mechanism Coverage: A clear mapping from five mechanism families to eight verified implementations, defining the transformed objects and valid diagnostics for each.
Evaluation on All Sources: A paired design where every source is evaluated against FullCache, with C $\to$ W failures selected separately per method-setting to preserve the evaluated source set without bias.
Paired Empirical Resource: A dataset containing 59,800 supported compressed runs and 12,520 C $\to$ W failure rows across four workloads, complete with valid cache, likelihood, attention, and decoding measurements.

4. Key Results

Failure Frequency and Identity

Aggregate Scores Mask Differences: While aggregate task scores for methods like TOVA, AdaKV, and SnapKV may appear similar at light compression tiers, their C $\to$ W failure rates vary significantly (e.g., 0.8% to 2.0%, a 2.5 $\times$ difference).
Method-Specific Failure Sets: Similar aggregate quality does not imply interchangeable failures. The Jaccard overlap between failure sets of different methods (e.g., SnapKV vs. TOVA) is often low ( $\le$ 0.38), indicating they fail on different subsets of data.
Workload Sensitivity: Failure rates increase from light to aggressive compression tiers across all workloads, but the composition of failures (e.g., single-key vs. multi-key retrieval) varies by method.

Diagnostic Analysis of Failures

Coverage is Often Low: Among 12,520 failure rows, 63.2% exhibit low or partial measured/projected coverage (mapped ECov < 0.90). This suggests that for many failures, the evidence was simply not retained.
Structural Addressability vs. Fidelity: A significant portion of failures (17.0%) occur in methods like ThinK and QuantizedCache where positions remain "structurally addressable" but representation fidelity is unknown. These show high likelihood drift ( $\Delta$ NLL $\ge$ 1) despite positional retention, proving that addressability does not guarantee correctness.
Diagnostic Separation: All ten proposed diagnostics (e.g., ERR loss, $\Delta$ NLL, EAR loss, Top-50 disagreement) successfully separate failed (C $\to$ W) from successful (C $\to$ C) compressions, with stratified AUROC scores ranging from 0.684 to 0.871.
Targeted Repair: For 96 reproducible low-EAR (low evidence attention retention) failures, a controlled intervention boosting attention logits on retained evidence positions repaired 29.2% of cases. This significantly outperformed a sham intervention (6.3%) and degraded only 3.3% of matched successful controls, validating low EAR as a specific signature for access/routing deficits.

Cross-Model Generalization

Coverage Trends: Failure frequency and coverage diagnoses (e.g., slot-ECov trends) generalize across Qwen, Falcon, and Mistral architectures.
Model-Specific Drift: Quantization-induced drift remains model-specific. For instance, ThinK causes severe drift on Qwen but less so on Falcon/Mistral, while QuantizedCache shows distinct behavior across models.

5. Significance and Claims

The paper claims that KVDiagnosis provides the first reusable diagnostic dataset that jointly offers complete per-source results, method-specific failure selection, and valid mechanism-aware measurements.

The authors emphasize that aggregate scores are insufficient for diagnosing compression failures. By separating failure sets and applying mechanism-specific diagnostics, KVDiagnosis reveals that:

Different compressors fail on different data subsets, making direct ranking without context misleading.
Many failures are due to simple evidence loss (low coverage), while others stem from representation degradation or access deficits even when positions are retained.
Specific failure signatures (like low EAR) can be used to guide targeted repairs, suggesting a path toward more robust compression strategies.

The work concludes that explicit denominators, native geometry, and missing-value semantics are essential for explaining why a compressor fails, rather than merely ranking them.