💬 NLP

Practical Online KV Cache Compaction for LLM Agents: An Empirical Study

This empirical study demonstrates that practical online KV cache compaction for LLM agents can achieve significant memory reduction and throughput gains by delaying compaction until future agent queries are available and utilizing token eviction with robust proxy sources, rather than relying on immediate or static context assumptions.

Original authors: Yujian Liu, Jiabao Ji, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yujian Liu, Jiabao Ji, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, multi-step mystery. You have a brilliant detective (an AI) who can ask questions, check clues, and talk to witnesses. But there's a catch: the detective's brain has a strict memory limit. Every time they take a step, write a note, or hear a witness's story, that information piles up. If the case goes on for too long, the detective's brain gets so full of old notes that they can't think clearly anymore, or they simply run out of space to write new ones. This is the world of "LLM Agents"—smart computer programs that solve complex tasks by talking to tools and the internet. The "KV cache" is just the technical name for that growing pile of notes in the detective's brain. Keeping this pile small enough to fit in memory, without losing the clues needed to solve the case, is the big challenge this paper tackles.

The researchers at UC Santa Barbara and LinkedIn decided to test a clever trick called "KV cache compaction." Think of it like a detective summarizing a long, boring police report into a single sticky note. Instead of keeping every single word of a past conversation, the AI tries to compress it into a shorter version that still holds the most important meaning. But here's the twist: in a normal story, you know the ending before you start summarizing. In an AI agent's life, the story is being written as it happens. The AI doesn't know what question it will ask next, so it has to summarize the past before it knows what the future will need. The paper asks: How do you summarize a chapter of a story when you haven't read the next chapter yet?

The team tested two main ways to do this summarizing. The first method, called Token Eviction (TE), is like a strict editor who reads the current page and decides, "These 80% of words are boring; let's throw them away and keep only the top 20%." The second method, Attention Matching (AM), is more like a fancy artist who not only picks the best words but also tries to paint a new, shorter version that feels exactly like the original long version when read later.

To figure out the best way to summarize, the researchers had to decide when to do it and what to use as a guide. They tested three different "guide" strategies:

The "Right Now" Guide: Summarize immediately using only the words just spoken.
The "Repeat" Guide: Ask the AI to pretend to re-read the last part and use that to decide what's important.
The "Future" Guide: Wait a little bit. Let the AI write the next few steps of the story, then use those new questions to decide what to keep from the old steps.

The results were surprising and practical. First, they found that summarizing immediately (using the "Right Now" guide) often made the AI dumber. It was like summarizing a mystery novel's first chapter before knowing who the villain is; you might throw away a clue that turns out to be vital later. However, if they waited just one turn—letting the AI ask its next question first—the summarization became much smarter. By using the "Future" guide, the AI could see what information was actually needed and keep only that.

They also discovered that the simpler method, Token Eviction (TE), was often more reliable than the fancy, complex Attention Matching (AM). Even when the "guide" wasn't perfect, the simple "keep the best 20%" approach held up better. It turns out that trying to be too clever with the math (like AM does) doesn't always help when you're guessing about the future.

The most exciting part is what this means for speed and cost. When they tested this on larger, more powerful AI models, the results were a game-changer. By compressing the memory to just 20% of its original size (keeping 1 out of every 5 tokens), they didn't just save space; they made the AI run 4.2 times faster on one model and 1.7 times faster on another. How? Because the AI's "brain" was so much smaller, the computer could run four times as many detective cases at the same time without crashing.

Interestingly, the paper also noticed that when the AI's memory was compressed, the detective sometimes got a bit "anxious." It tended to ask more questions and take more steps to solve the same puzzle, perhaps trying to double-check facts it felt it had lost. This suggests that while the AI still got the right answers, its behavior changed slightly to compensate for the tighter memory.

In short, this paper suggests that if you want to run smart, long-running AI agents without breaking the bank or the computer's memory, you shouldn't rush to summarize. Instead, let the AI take a few more steps, peek at what it's going to do next, and then compress the past. And surprisingly, you don't need a super-complex algorithm to do it; a simple, smart selection of the most important words works just as well, if not better. This approach could make running these advanced AI agents much cheaper and faster for everyone.

Technical Summary: Practical Online KV Cache Compaction for LLM Agents

Problem Statement
Large Language Model (LLM) agents accumulate long trajectories of reasoning steps, tool calls, and environment feedback. This growing context creates a significant inference bottleneck, as the Key-Value (KV) cache required to retain this history dominates memory usage and increases decoding costs. While KV cache compaction is a known solution, existing methods primarily assume a static setting: the full context is available before generation, allowing offline compaction using known future queries or synthetic data.

LLM agents, however, operate in an online regime. Context is constructed incrementally through interaction with an environment (e.g., search results, file diffs). Consequently, future relevance is not observable at the time a turn is completed, and the compaction procedure must fit within a strict runtime budget without adding expensive rollouts or optimization steps. This creates a design space where compaction signals must be derived from cheap, available proxies, and the trade-off between signal quality and latency is critical.

Methodology
The authors conduct a systematic empirical evaluation of adapting two established sequence-level compaction families to the online agent setting:

Token Eviction (TE): Uses proxy queries to score cached positions by induced attention mass, retaining only the original Keys and Values (KVs) at the highest-scoring positions.
Attention Matching (AM): Uses the same proxy-based key selection but additionally fits an additive attention bias and reconstructed values to better match the attention outputs of the full cache.

The study focuses on how these methods perform when adapted to online agent trajectories, specifically investigating the selection of proxy queries (signals used to guide compaction). The authors compare three categories of proxy sources:

Current-turn proxies: Available immediately upon turn completion.
- Boundary queries: Extracted from structural closing/transition tokens (e.g., end-of-thinking tokens).
- Repeat-prefill queries: Generated by teacher-forcing the model to repeat the previous turn's content.
Future-turn proxies: Derived from the agent's own subsequent generations, requiring a delay in compaction.
- Assistant generation queries: Query vectors from the next $k$ turns.
- Tool response queries: Query vectors from tool-response prefill tokens.

Experiments were conducted on two benchmarks: BROWSECOMP-PLUS (deep search requiring iterative evidence retrieval) and WIDESEARCH (broad information seeking requiring table completion). Models tested include Qwen3.5 and Gemma-4 across multiple scales (4B, 27B, and 31B).

Key Results

The Necessity of Delay: Immediate compaction using current-turn proxies often degrades task performance. Delaying compaction to utilize queries from the agent's own future generations (even by just one turn) consistently recovers most of the performance gap compared to no compaction.
Robustness of Token Eviction: Despite its simpler "selection-only" design, TE proves surprisingly robust to imperfect proxy signals. It often outperforms or matches the more complex AM method, particularly when proxy sources are suboptimal. AM's additional optimization (fitting bias and values) provides limited benefit when the proxy queries do not perfectly represent future usage.
Proxy Selection:
- For TE, boundary queries serve as strong immediate signals on specific benchmarks (e.g., BROWSECOMP-PLUS), but future-turn queries are generally superior.
- Combining multiple proxy sources (e.g., adding tool-response queries to future-turn queries) does not reliably improve performance and can sometimes hurt it.
Scalability and Efficiency:
- On larger models (Qwen3.5-27B and Gemma-4-31B), delayed compaction with a ratio of 0.2 preserves most of the no-compaction accuracy.
- Compaction significantly reduces peak KV footprint (up to 3.5× reduction for Qwen3.5-27B and 2.24× for Gemma-4-31B with TE), enabling larger serving batches.
- This leads to substantial throughput improvements (up to 4.2× for Qwen3.5-27B and 1.49× for Gemma-4-31B with TE), offsetting the increased number of agent turns required by the compaction process.
Behavioral Adaptation: Compaction alters agent behavior. Models like Qwen3.5-4B tend to lengthen their trajectories and issue more repeated search queries to recover information weakened by compaction. This suggests that the agent adapts to the compressed context by interacting more with the environment.

Significance and Claims
The paper positions proxy-query selection as a core design choice for practical online KV compaction. The authors claim that:

Online compaction is distinct from static compaction due to the unobservability of future relevance and strict latency constraints.
Delaying compaction to leverage the agent's own future queries is a critical strategy for maintaining performance.
Token Eviction (TE) is a highly effective, robust baseline that often outperforms complex optimization methods (AM) in the online setting due to its resilience to imperfect proxies.
Practical memory systems for agents must evaluate not only compression ratios and final accuracy but also downstream agent behavior, as compaction can induce redundant tool use and longer trajectories that are necessary to maintain task success.

Limitations
The authors note that their study is limited to search and document-reading workloads, excluding code editing or GUI control. They also restrict their scope to cheap, online proxy sources, acknowledging that more expensive offline optimization or learned compressors might yield better results if latency budgets were relaxed. Finally, they used fixed compaction ratios and delays rather than adaptive policies, which remains a direction for future work.

Technical Summary: Practical Online KV Cache Compaction for LLM Agents

More like this