💻 computer science

Break Later or Never: Prefix-Preserving Context Compaction for Coding Agents

This paper addresses the conflict between context compaction and prefix caching in coding agents by introducing "batched observation masking" and "ObsDrop," a KV-cache-level technique that releases stale tool-observation data while preserving the logical token prefix, thereby significantly improving throughput and latency without compromising task quality.

Original authors: Musa Cim, Burak Topcu, Mahmut Kandemir

Published 2026-11-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Musa Cim, Burak Topcu, Mahmut Kandemir

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer program designed to write code, fix software bugs, or solve complex engineering problems. These "coding agents" do not work in a single burst; they operate in a long, looping conversation with a large language model. In each step, the agent thinks, issues a command to a tool (like opening a file or running a test), and then reads the result. It then repeats the process, adding the new result to its history. Over time, this history grows into a massive document containing tens of thousands of words. To keep the system running, the computer must remember every single word of this conversation to understand the current step. This memory requirement creates a heavy burden, slowing down the system and making it expensive to run.

To manage this growing pile of text, engineers have developed ways to shrink the history, a process called context compaction. The most common method is to simply delete old parts of the conversation or replace them with a short summary. However, there is a hidden trap in this approach. Modern computers are incredibly efficient at reusing work they have already done. If a new request starts with the exact same words as a previous one, the computer can skip the heavy lifting and jump straight to the new part. But if you delete or rewrite the beginning of the conversation, even slightly, the computer loses its memory of the past and must start the heavy calculation over from scratch. The researchers behind this study discovered that the very act of trying to save space by editing the text often forces the computer to do far more work, canceling out any savings and sometimes making the system slower and more expensive.

The team, based at Pennsylvania State University, set out to understand this conflict between saving space and saving time. They observed that coding agents naturally repeat almost the entire history of their conversation in every single step. Because the history is mostly just being added to, like a diary where you only write new entries at the bottom, the computer can reuse its previous calculations almost perfectly, achieving a reuse rate of 98 to 99 percent. The biggest chunk of this memory cost comes not from the agent's thoughts, but from the raw data it receives, such as the contents of files or the logs from software tests. The researchers found that when they used standard methods to delete or summarize these old files, they broke the chain of reuse. The computer was forced to re-read and re-calculate everything that came after the deletion, turning a quick task into a slow, expensive one.

To solve this, the researchers first tested a simple adjustment to how the deletion happens. Instead of deleting old information after every single step, they tried deleting it only once every few steps. This small change, which they call "batched masking," allowed the computer to keep its memory intact for longer stretches. It reduced the amount of text the agent had to carry by up to half, while still keeping the computer's memory reuse rate high, between 95 and 98 percent. This approach lowered the cost of running the agent by up to 18 percent and made the tasks finish up to 21 percent faster. It was a practical fix that required no changes to the underlying computer software, simply a smarter way to schedule the deletions.

However, the team realized that even this improvement still involved rewriting the text, which inevitably broke the memory chain at the moment of deletion. They then developed a more advanced solution called ObsDrop. Instead of deleting words from the text that the agent sees, ObsDrop works inside the computer's memory itself. It keeps the text exactly as it is, so the computer can still recognize the pattern and reuse its previous calculations. But in the background, it quietly releases the memory slots holding the old, irrelevant data, such as the contents of files that are no longer needed. This allows the computer to keep the logical history intact for speed, while physically freeing up the space needed to run more agents at once.

The researchers tested this new system on powerful servers running multiple agents simultaneously. They found that as the number of agents increased, the standard methods began to struggle, with the system slowing down dramatically as it ran out of memory or had to constantly re-calculate. In contrast, the ObsDrop system maintained its speed and efficiency. Under heavy load, it allowed the servers to process between 2.45 and 2.85 times more work than before, and in some specific configurations, the improvement reached more than five times the original speed. The system handled the pressure without losing the quality of the work the agents produced.

This work highlights a crucial lesson for the future of artificial intelligence: managing the memory of a smart agent is not just about how much text you keep, but how you keep it. By understanding that the computer's ability to reuse past work is fragile, the researchers showed that effective management requires preserving the structure of the conversation even while discarding the data. Their new method, ObsDrop, offers a way to have both: a leaner memory footprint and a faster, more responsive system, ensuring that these powerful coding tools can scale up to handle the complex problems of tomorrow without getting bogged down by their own history.

Technical Summary: Break Later or Never: Prefix-Preserving Context Compaction for Coding Agents

Problem Statement

Coding agents operate by repeatedly invoking Large Language Models (LLMs) over growing, append-only histories consisting of reasoning, tool calls, and tool observations. This workload is uniquely friendly to prefix caching, as each new request typically reuses the vast majority of the previous context, allowing serving engines to skip the expensive "prefill" phase for repeated tokens.

However, a critical conflict exists between context compaction (necessary to manage memory and cost as trajectories grow) and prefix caching. Traditional compaction methods (e.g., truncation, masking, summarization) rewrite the prompt history. Because prefix caching relies on exact token matching, any edit to the prompt invalidates the Key-Value (KV) cache for that prefix and all subsequent tokens. This forces the system to recompute (re-prefill) the entire suffix, negating the computational savings of caching. The paper identifies that while tool observations (file views, test logs) constitute the majority of context cost, removing them via prompt-level editing breaks the prefix cache, potentially making a shorter prompt more expensive to serve than a longer, unmodified one.

Methodology

The authors characterize this interaction and propose solutions at two levels: prompt-level and KV-cache level.

1. Characterization and Prompt-Level Mitigation

Anatomy of Cost: The study analyzes coding agent trajectories on SWE-bench (Verified and Multilingual) using Qwen3-Coder and Devstral-Small models. It finds that without compaction, prefix-cache hit rates are 98–99%. The cost is dominated by re-reading tool observations, not recomputation.
Evaluation of Existing Policies:
- Sliding Window: Discards oldest turns. While it reduces prompt length, it shifts the prefix start point, causing a 100% re-prefill of the window at every step. This results in high serving costs and low task success rates.
- Observation Masking: Replaces old observations with placeholders. While it preserves task success better than sliding windows, it still edits the prompt at every step, invalidating the prefix cache and forcing re-prefills.
- LLM Summarization: Replaces history with a summary. This introduces decode overhead (generating the summary) and can lose critical information, reducing task success on multilingual benchmarks.
Batched Observation Masking: The authors propose a simple mitigation where observation masking is applied only once every $k'$ turns (e.g., every 8 steps) rather than every step. This amortizes the prefix invalidation cost, preserving 95–98% prefix reuse while reducing context length.

2. ObsDrop: KV-Level Compaction

To resolve the conflict between compaction and caching entirely, the authors introduce ObsDrop, a mechanism that decouples the logical token sequence (the prompt sent to the model) from the physical KV residency (what is stored in GPU memory).

Mechanism: ObsDrop leaves the prompt and token positions unchanged to ensure prefix matching remains valid. However, it releases the KV entries corresponding to stale tool observations from the GPU memory.
Implementation:
- SGLang: Implemented within the RadixAttention cache. ObsDrop identifies observation spans, replaces their KV-slot indices with a reserved "dead" slot (index 0), and returns the physical slots to the allocator. The attention kernel is then instructed to skip these dead slots.
- Sparse-vLLM: Implemented as an independent layer that releases KV entries for old observations while maintaining the logical prefix structure.
Optimization: To avoid runtime overhead, dead slot tracking is handled on the CPU, and attention index lists are pruned in a single GPU pass before kernel execution, preserving CUDA graphs.

Key Contributions

Characterization: The paper demonstrates that prompt edits are a previously overlooked component of compaction cost. In coding agents, the high rate of prefix reuse means that breaking the prefix cache is often more expensive than the memory savings gained from compaction.
Prefix-Aware Prompt Compaction: The authors show that batching observation masking (masking every $k'$ steps) is a practical, engine-agnostic solution that preserves 95–98% prefix reuse while reducing context length by up to 50%.
ObsDrop: A novel KV-level compaction mechanism that reclaims memory for stale observations without altering the logical prompt. This allows the system to maintain prefix matching while reducing KV residency and attention work.
End-to-End Evaluation: Comprehensive evaluation across SGLang and Sparse-vLLM, using FlashInfer and Triton backends, under varying concurrency levels.

Results

Prompt-Level (Batched Masking):
- Reduces mean context length by up to 50%.
- Reduces serving cost by up to 18% and task completion time (TCT) by up to 21%.
- Maintains task success rates comparable to no compaction.
KV-Level (ObsDrop):
- Throughput: Under high concurrency (16 agents), ObsDrop achieves 2.45× to 2.85× higher throughput with FlashInfer and up to 5.37× with the Triton backend compared to no compaction.
- Latency: ObsDrop significantly reduces tail latency (p95 TTFT) under memory pressure by preventing cache eviction and the subsequent re-prefill cycles that plague "no compaction" scenarios.
- Efficiency: ObsDrop reduces the volume of attended tokens by 19–44% across different models and engines, directly lowering compute costs.
- Quality: Task success rates remain close to the "no compaction" baseline, unlike aggressive summarization or sliding window approaches which degrade performance.

Significance and Claims

The paper argues that effective context management for coding agents must consider not only which tokens are retained but how compaction interacts with the serving system's caching mechanisms.

The Trade-off: The authors claim that the conflict between compaction and prefix caching is fundamental. Prompt-level edits inevitably break cache reuse, whereas ObsDrop resolves this by operating at the KV level.
Systemic Impact: ObsDrop provides a primitive that allows serving engines to reclaim memory without breaking the logical history required for prefix caching. This is particularly critical under concurrent workloads where memory pressure causes cache evictions, leading to sharp increases in tail latency.
Modest Claims: The authors acknowledge that the optimal batching interval ( $k'$ ) may vary by trajectory and that their implementation in vLLM requires further decoupling of block identity from KV residency. They position ObsDrop not as a universal replacement for all compression techniques, but as a specific solution for the unique "append-only, observation-heavy" workload of coding agents.

In summary, the paper demonstrates that by preserving the logical prefix while physically dropping stale observation data, agents can achieve significant performance and cost improvements without sacrificing task quality.