💻 computer science

Still: Amortized KV Cache Compaction in a Single Forward Pass

The paper introduces Still, a lightweight, reusable per-layer Perceiver that performs amortized KV cache compaction in a single forward pass, achieving superior speed-quality trade-offs and enabling iterative long-horizon inference across extreme compression ratios and context lengths.

Original authors: Charles O'Neill, Alex Sandomirsky, Harry Partridge, Mudith Jayasekara, Max Kirkby

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Charles O'Neill, Alex Sandomirsky, Harry Partridge, Mudith Jayasekara, Max Kirkby

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Infinite" Notebook

Imagine a large language model (like a super-smart AI assistant) is reading a very long book to answer a question. To remember what it read, the AI keeps a "notebook" (called the KV Cache) next to its brain. Every time it reads a new word, it writes a note in this notebook.

The Issue: If the book is 100 pages long, the notebook is small. But if the book is 1,000,000 pages long, the notebook becomes huge. Eventually, the notebook gets so big that it doesn't fit in the computer's memory (RAM).
The Current Solutions:
1. Throwing things away: The AI looks at its notes and deletes the ones it thinks are boring. This is fast, but it might accidentally throw away the one sentence that holds the answer.
2. Rewriting the whole thing: The AI tries to summarize the whole book into a short paragraph. This is smart, but it takes a long time to do every single time it reads a new page.

The Solution: "Still" (The Smart Summarizer)

The authors introduce a new tool called Still. Think of Still as a super-efficient, instant editor that sits between the AI and its notebook.

Instead of just deleting notes or rewriting the whole book, Still does something clever: it looks at the entire notebook and instantly compresses it into a tiny, high-quality summary card.

How it Works (The Analogy)

Imagine you are a museum curator with a massive library of artifacts (the full notebook). You need to fit them all into a tiny display case (the compressed cache) so you can move them easily.

The Old Way (Selection): You pick 10 random artifacts and throw the rest away. You might lose the most important one.
The Old Way (Synthesis/Per-Context): You sit down and carefully craft a perfect summary for this specific library. It's great, but it takes hours. If a new library arrives, you have to start over.
The "Still" Way: You have a magic camera (the Perceiver module). You snap a photo of the entire library, and the camera instantly processes the image into a single, perfect 4x6 card that contains the essence of everything in the library.
- Crucial Point: You only have to learn how to use this camera once. After that, you can use it on any library, instantly, without stopping to think or calculate.

Why "Still" is Special

1. It's Fast (The "One-Shot" Trick)
Most smart summarizers have to do a lot of math every time they see new text. Still is different. It was trained once to be a "fast forward pass."

Analogy: Imagine a chef who has to chop vegetables for a soup.
- Old methods: The chef stops to measure every single carrot and onion perfectly before chopping (slow).
- Still: The chef has a pre-trained muscle memory. They chop the whole pile in one smooth motion. It happens in a single step.

2. It's Smart (Synthesis vs. Selection)
Still doesn't just pick the "best" notes to keep; it mixes them together to create new, super-dense notes.

Analogy: If you have a library of 1,000 books, a "selector" might keep 50 of the best books. Still takes the ideas from all 1,000 books and blends them into 50 new "super-books" that contain the DNA of the originals. This means it can keep the answer even if the answer was hidden in a sentence that a simple selector would have deleted.

3. It Works for Long Stories (Iterative Compaction)
The paper shows that Still can be used over and over again. As the AI reads a story chapter by chapter, Still can compress the memory of Chapter 1, then compress the memory of Chapter 2, and so on.

Analogy: Imagine you are writing a diary. Every night, instead of keeping every single page, you write a one-page summary of the day. The next day, you read your one-page summary, add today's events, and write a new one-page summary. You can keep doing this forever without your diary getting too heavy.

The Results (What the Paper Found)

The authors tested Still on different models (Qwen and Gemma) and different lengths of text (from 8,000 words to 128,000 words).

Speed vs. Quality: They found that Still sits in the "sweet spot." It is much faster than the smartest summarizers and much more accurate than the simple "delete the boring stuff" methods.
Extreme Compression: Even when they compressed the memory by 200 times (making a 128k document fit into the space of a 600-word document), Still kept the AI smart enough to answer questions correctly.
Summarization: It works not just for answering multiple-choice questions, but for writing free-form summaries too.

The Catch (Limitations)

The paper is honest about what Still can't do yet:

It's not magic: If you compress something too much (like 200x) on a very long text, you will lose some details. It's not perfect.
Training is needed: You have to train a specific "Still" module for each specific AI model you use. You can't just plug it into any model without a little setup.
Exact Recall: If you need the AI to recite a specific sentence word-for-word from 100,000 words ago, Still might struggle. It's great for understanding the meaning, but not for being a perfect photocopier.

Summary

Still is a new tool that lets AI assistants remember massive amounts of text without running out of memory. It works by instantly "distilling" the entire memory into a tiny, smart summary card. It's fast enough to use in real-time, smart enough to keep the important details, and flexible enough to handle stories that go on for hours.

Technical Summary: Still: Amortized KV Cache Compaction in a Single Forward Pass

1. Problem Statement

The KV (Key-Value) cache is the primary memory bottleneck for deploying large language models (LLMs) over long horizons, such as multi-day coding agents or repository-scale reasoning. Current deployment strategies face an "all-or-nothing" choice:

Lossless Caching: The cache grows linearly with context length, eventually hitting memory limits.
Lossy Alternatives: Methods like fine-tuning, retrieval-augmented generation (RAG), or document summaries abandon the model's internal representation structure and fidelity.

Existing KV cache compaction methods fall into two categories, each with significant limitations:

Selection Methods: These score tokens and evict a subset (e.g., H2O, SnapKV). They are lightweight but structurally bound to the original token subset, meaning they cannot synthesize new information and degrade rapidly under tight compression budgets.
Synthesis Methods: These combine content to create new compact representations (e.g., Attention Matching, Cartridges). While expressive, they typically require per-context optimization (fitting) at inference time, making them computationally expensive and impractical for iterative, long-horizon trajectories where compaction must happen repeatedly.

The paper identifies a gap for a method that is amortized (trained once, applied repeatedly without per-context optimization), synthetic (capable of blending information beyond simple selection), and lightweight enough to run during inference.

2. Methodology: Still

The authors introduce Still, a learned primitive for KV cache compaction designed for frozen pre-trained models.

Architecture

Still consists of a small, per-layer Perceiver module trained against a frozen base model.

Input: For each transformer layer, the compactor receives the full per-layer KV cache ( $K, V$ ) of the prefix. The keys are first "un-rotated" to remove Rotary Position Embeddings (RoPE), placing them in a position-free frame.
Latent Queries: The compactor maintains a bank of $t$ learned latent queries ( $Z$ ) per layer. These queries cross-attend to the full KV cache.
Processing: The latents pass through pre-norm blocks containing:
1. Cross-Attention: From latents to the full KV cache.
2. Self-Attention: Among the latents.
3. Feed-Forward (FFN): Optional sublayer.
Output: The final latent state is projected via linear heads to produce compact keys ( $C_k$ ) and values ( $C_v$ ) of size $t \times d$ , where $t \ll T$ (the original sequence length).
Position Handling: The resulting compact keys are re-rotated at chosen output positions before being written back to the cache. Values remain unrotated.
Bias Channel: An optional bias head ( $\beta$ ) is supported but disabled ( $\beta=0$ ) in all reported results to simplify deployment.

Training

Objective: The compactor is trained using forward KL divergence between a "teacher" (the frozen base model with the full uncompressed cache) and a "student" (the same model using the compact cache). The loss is masked to answer-side tokens only.
Data: Training uses a four-domain extractive multiple-choice QA dataset (Financial, Project Gutenberg, Legal, Code).
Efficiency: The base model weights remain frozen; only the compactor parameters are trained. The compactor adds roughly 1% to the base model's parameter count (e.g., ~50M parameters for Qwen3-4B).

Operational Modes

Single-Pass Compaction: The model runs a full prefill to generate the full cache, which is then compressed by Still. The generation continues using the compact cache.
Iterative Chunked Compaction: For long-horizon contexts arriving over time, Still supports a recurrent schedule. The model processes chunks of tokens, and after each chunk, the accumulated cache is compressed to a fixed size. This allows the system to handle contexts far exceeding the training horizon, provided the compactor was trained on sufficiently long iterative horizons.

3. Key Contributions

Amortized Synthesis: Still demonstrates that amortized synthesis (learning an encoder once) is superior to amortized selection (learning a scorer once) for long-context compaction. It avoids the "subset-bound" limitation of selection methods and the "per-context optimization" cost of synthesis methods.
Single Forward Pass: Unlike per-context synthesis methods that require optimization steps for every new context, Still performs compaction in a single forward pass through a learned module.
Iterative Long-Horizon Capability: Because compaction is a forward pass, Still can be applied iteratively as new context arrives, enabling a long-horizon regime that is unavailable to per-context fitting methods.
Generalization: The method transfers across model scales (4B to 32B), architectures (dense and MoE), and attention mechanisms (global and sliding-window).

4. Results

The paper evaluates Still across compression ratios from 8× to 200× and context lengths from 8k to 128k.

Speed-Quality Frontier: On Qwen and Gemma models, Still occupies the favorable side of the speed-quality frontier. It remains accurate and fast across the entire matrix, whereas selection-based methods degrade under tight budgets, and per-context synthesis methods are too slow.
RULER Benchmark: In matched-training settings on the RULER benchmark, Still exceeds the strongest amortized selection baseline (KV-Distill) by 8–22 accuracy points across 16 of 18 cells.
Transfer Performance: In zero-shot transfer settings (training on mixed-domain MCQ, testing on RULER), Still maintains a significant lead over KV-Distill, though margins are smaller due to train-test mismatch.
Summarization: Still supports free-form summarization. On HELMET (multi_lexsum), it preserves 74–95% of the full-context gain across 8k–64k contexts, outperforming Attention Matching and KV-Distill. On LongBench v1, it wins 60% of pairwise comparisons against KV-Distill.
Iterative Compaction: Still enables iterative chunked compaction. While performance degrades if the training horizon is too short relative to the deployment horizon (e.g., an 8k-trained compactor collapsing at 128k), training on longer iterative horizons (16k, 32k) allows the model to absorb context many times its physical capacity.

5. Significance and Claims

The paper claims that amortization makes long-context cache compaction tractable, and synthesis makes the compact state useful at extreme compression.

Tractability: By learning the optimizer (the compactor) once, the system avoids the compounding fitting costs of per-context methods, making iterative long-horizon deployment feasible.
Expressivity: Synthesis allows the compact cache to blend information from arbitrarily many positions, overcoming the information density ceiling of token selection.
Practicality: Still is lightweight enough to be called repeatedly during inference, supporting both answer selection and free-form generation without modifying the base model's weights.

Limitations Acknowledged by the Authors

Not Lossless: Full-context inference remains superior in the hardest 128k settings.
Training Horizon Constraint: Iterative compaction is not free extrapolation; the compactor's performance is bounded by the training horizon. Supporting 1M tokens requires training at proportionally longer horizons, which is expensive.
Exact Retrieval: The current recurrence is weak on exact-retrieval and "needle-in-a-haystack" tasks, even when it preserves semantic evidence for broader MCQ tasks.
Memory vs. Latency: In single-pass mode, Still reduces the retained cache size but does not eliminate the initial full-context prefill latency.
Deployment Requirements: Each base checkpoint requires its own trained compactor, adding parameter and training data overhead.