🤖 machine learning

IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference

IntentKV is a learned, cross-turn intent-aware KV cache pruning method that maintains a frozen base LLM while using session-level memory and a zero-initialized residual head to dynamically score and redirect tokens, achieving significant reductions in peak memory and KV read bandwidth for long-horizon agent inference with minimal accuracy loss.

Original authors: Junjie Li, Jiong Lou, Jie Li

Published 2026-06-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Junjie Li, Jiong Lou, Jie Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a highly skilled detective (the AI Agent) trying to solve a complex mystery. You don't just ask one question and get an answer; you have to go on a long journey. You search the library, interview witnesses, check old files, and write down notes. Each step adds more and more paper to your desk.

Eventually, your desk becomes so cluttered with papers that you can't find anything, and you run out of space to write new notes. This is exactly the problem IntentKV solves for AI agents.

Here is the breakdown of the paper using simple analogies:

The Problem: The "Cluttered Desk"

When an AI agent works on a multi-step task (like researching a flight and then booking it), it generates a massive trail of information:

The User's Request: "Find me a flight."
The Search: The AI looks up flights.
The Results: It reads a list of 50 flights.
The Follow-up: "Which one is cheapest?"
The New Search: It searches again.

Every time the AI thinks, it has to look back at all the previous papers on its desk to understand the context. As the conversation gets longer, the "desk" (memory) gets huge. The paper argues that the biggest bottleneck isn't the AI's brainpower (computing power); it's the memory space and the speed of reading that desk. If the desk is too full, the AI slows down or crashes.

The Old Solutions: "Shuffling the Papers"

Previous methods tried to fix this by throwing away old papers.

The Problem: They usually looked at the current question and decided what to throw away.
The Flaw: In a long detective story, a clue from page 1 might be crucial on page 50. Old methods often threw away that page 1 clue because it didn't seem important right now.
The Second Flaw: Some methods physically moved the remaining papers to a new, smaller pile. This broke the "index" or "map" that the system uses to quickly find shared information between different users. It's like reorganizing a library so that "Harry Potter" is now on a different shelf than before; the librarian (the system) gets confused and can't reuse the books efficiently.

The New Solution: IntentKV

The authors created IntentKV, which acts like a smart, organized assistant who manages the desk without moving the books.

1. The "Session Memory" (The Detective's Notebook)

Instead of just looking at the current question, IntentKV keeps a running summary of the whole investigation.

How it works: It maintains a "QueryMemory" that updates as the conversation goes on. It knows that even though the user is currently asking about "prices," they might need the "flight schedule" from 10 minutes ago to make a decision.
The Analogy: Imagine a detective who keeps a sticky note on the wall that says, "Remember: The suspect was wearing a red hat." Even if the current conversation is about the weather, the detective knows the red hat is still relevant. IntentKV keeps the "red hat" papers on the desk, even if they aren't being looked at right this second.

2. The "Zero-Init Residual" (The Safety Net)

The system uses a simple rule to decide what to keep (based on the Session Memory). But sometimes, the rule misses something subtle.

The Fix: They added a tiny, learnable "residual head." Think of this as a junior intern who double-checks the senior detective's list.
The Magic: This intern starts with zero knowledge (zero-initialized) and learns only to correct the senior's mistakes. It doesn't change the detective's brain (the main AI model); it just adds a small layer of smart correction.

3. The "Dead Slot" Trick (The Magic Map)

This is the most clever part. When the desk is full, IntentKV has to remove some papers.

Old Way: Take the papers out, stack the remaining ones tightly, and renumber them. (This breaks the library's index).
IntentKV Way: It takes the paper out, but instead of moving the others, it puts a "Do Not Read" sign (a sentinel dead slot) on the empty spot.
The Result: The papers stay exactly where they were. The "map" of the desk remains perfect. If another detective comes in with a similar case, the system can instantly recognize the shared papers because they haven't moved. This allows the system to reuse memory efficiently, saving huge amounts of time and space.

The Results: What Did They Find?

The paper tested this on two specific AI models (Qwen3-8B and Qwen2.5-14B) using a difficult "deep research" benchmark called BrowseComp-Plus.

Accuracy: IntentKV kept the AI just as smart as if it had infinite memory. It didn't lose any "detective skills."
Efficiency:
- It reduced the amount of memory needed by about 24% to 31% for standard tasks.
- For the hardest, longest tasks, it reduced the memory usage by up to 78% and the reading speed requirements by up to 92%.
Comparison: It beat all other "cleaning up" methods. While other methods crashed or gave wrong answers when the memory got tight, IntentKV kept working smoothly.

Important Limitations (What the Paper Doesn't Say)

It's not a magic wand for all AI: The paper notes that they only tested this on specific "Qwen" models. Other popular open-source models (like Llama or Mistral) failed to even perform the multi-step tasks correctly in their tests, so IntentKV couldn't be tested on them. The issue wasn't the memory; the models just couldn't "act" like agents in that specific test environment.
It doesn't choose its own budget: The system needs a human to set a fixed limit (e.g., "keep 8,000 tokens"). It doesn't automatically decide, "Oh, this question is short, I'll use less memory."
It's for Agents, not Chatbots: This is specifically designed for AI that uses tools (searches, code, etc.) over many turns, not for a simple one-off chat.

Summary

IntentKV is a smart memory manager for AI agents. It keeps the most important clues from the past by tracking the intent of the whole conversation, not just the current sentence. It removes old information without shuffling the remaining papers, allowing the AI to stay fast, accurate, and efficient even during very long, complex investigations.

Technical Summary: IntentKV

Problem Statement

Modern Large Language Model (LLM) agents increasingly execute multi-turn workflows involving browsing, deep research, and tool usage. In these scenarios, a short initial user query expands into a long trajectory of retrieved documents, tool outputs, intermediate decisions, and generated arguments. This creates an asymmetric serving profile where the agent state grows to tens of thousands of tokens, while the original query remains short. Consequently, Key-Value (KV) memory capacity and KV read bandwidth, rather than parameter computation, become the dominant bottlenecks for long-horizon agent sessions.

Existing KV-cache pruning methods (e.g., StreamingLLM, SnapKV, H2O) are primarily designed for single-prompt compression. When applied to multi-turn agents, they face two critical failure modes:

Stale Retention Signals: Token importance shifts across turns. Evidence useful for a later turn (e.g., a specific search result) may be discarded by a scorer trained only on the initial prompt, as the "importance" of tokens is not static but evolves with new intents.
Loss of Prefix Identity: Standard pruning often compacts the cache by relocating surviving KV rows to contiguous slots. This changes the logical positions and RoPE phases of the tokens, invalidating the prefix identity required by systems like SGLang and vLLM to reuse cached states via radix trees across turns.

Methodology: IntentKV

IntentKV addresses these issues by decoupling the retention policy (which tokens to keep) from the layout policy (how the cache is represented). The method keeps the base LLM frozen and introduces a learned, session-aware pruning mechanism.

1. Cross-Turn Intent-Aware Retention

Instead of scoring tokens against a single prompt, IntentKV maintains a QueryMemory ( $M_t$ ) that aggregates query-like signals across the entire session.

QueryMemory Update: $M_t$ is updated via a geometric recurrence: $M_t = e^{-\lambda} M_{t-1} + \text{Enc}(q_t)$ , where $\text{Enc}(q_t)$ represents the actionable query span of the current turn. This allows the system to track evolving intents (e.g., shifting from "search flights" to "find the cheapest option").
Rule Scoring: Tokens are scored based on their attention alignment with the accumulated $M_t$ , replacing the window-based tail attention of methods like SnapKV.
Learned Residual: To handle cases where the rule scorer misses future-use evidence, a zero-initialized residual head is added. This head uses cross-attention over the current query's K-vectors and per-token features (including the rule score and QueryMemory state) to predict a correction. The final score is a combination of the rule score and the learned residual.

2. Layout-Preserving Eviction

To maintain compatibility with prefix caching, IntentKV employs a slot-map redirection strategy rather than compaction.

Sentinel Dead Slot: When a token is evicted, its position in the slot map is redirected to a pre-allocated "sentinel" dead slot. The K/V values for this slot are set to specific constants (e.g., $K=-10^4$ , $V=0$ ) that effectively zero out its attention contribution without altering the physical layout of surviving tokens.
Preserved Identity: Surviving K/V rows, their RoPE phases, and their logical slot identities remain unchanged. This allows the radix tree to continue matching prefixes across turns, enabling efficient reuse of cached states even after pruning.

Key Contributions

Multi-Query Retention Formulation: The paper formulates agent KV pruning as a multi-query retention problem, instantiated with QueryMemory, a per-session state that aggregates queries, tool calls, and search intents across turns.
Frozen-Model Residual Pruner: A learned residual pruner that keeps the session-aware heuristic as the primary signal and adds a correction grounded in future actions, all without updating the base LLM weights.
Composable Eviction Recipe: An eviction strategy for paged KV caches (SGLang/vLLM) that redirects dropped positions to a sentinel slot. This preserves radix-prefix reuse, achieving a 20.7% prefix-hit rate at an 8k budget where compaction baselines drop to 0–3%.
Performance Gains: On the BrowseComp-Plus (BCP) benchmark, IntentKV matches the full-cache baseline with minimal accuracy loss while significantly reducing resource usage.

Experimental Results

The method was evaluated on Qwen3-8B and Qwen2.5-14B using the BrowseComp-Plus (BCP) benchmark (830 deep-research queries).

Accuracy: At an 8k KV budget, IntentKV-Phase-2 achieves a True Accuracy of 14.10 on Qwen3-8B (surpassing the strongest heuristic, StreamingLLM, by 2.29 points) and 18.55 on Qwen2.5-14B (surpassing StreamingLLM by 10.36 points). At a 16k budget, it matches the full-cache ceiling within 0.96 points on Qwen3-8B.
Efficiency: IntentKV reduces the worst-case peak request tokens by 77.8%–81.7% and worst-case raw KV reads by 36.9%–92.6% compared to the uncompressed ceiling on the 100 most demanding queries.
System Overhead: It achieves up to 43% reduction in wall-clock latency and 59% reduction in raw KV reads compared to compaction baselines, largely due to preserved prefix reuse.
Ablation: The cross-turn memory component is identified as the dominant signal; removing it causes a significant drop in accuracy. The learned residual provides marginal gains at tight budgets but improves performance when capacity allows.

Significance and Claims

The paper claims that IntentKV successfully solves the tension between KV compression and multi-turn agent serving by separating retention logic from cache layout. By maintaining a session-level memory of intent and using a non-compacting eviction strategy, it enables agents to operate under tight memory budgets without sacrificing the ability to reuse cached prefixes.

The authors note that their evaluation is restricted to Qwen models because other open-source backbones (e.g., Llama-3.1, Mistral) failed to exhibit reliable autonomous multi-step tool-calling behavior on the BCP benchmark under their serving harness, preventing a fair comparison of compression effects. The method is designed to be composable with existing serving stacks (SGLang, vLLM) and future prompt-local scorers, though per-query dynamic budget selection remains a direction for future work.