🤖 AI

Parallel Context Compaction for Long-Horizon LLM Agent Serving

This paper introduces **parallel context compaction**, a method that replaces lossy and unpredictable sequential summarization with a parallelized approach to provide fine-grained control over summary volume while significantly reducing wall time and improving throughput for long-horizon LLM agents.

Original authors: Musa Cim, Burak Topcu, Chita Das, Mahmut Taylan Kandemir

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Musa Cim, Burak Topcu, Chita Das, Mahmut Taylan Kandemir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a brilliant detective (the AI Agent) solving a massive, multi-part mystery. Every time you ask a question or get a clue, you write it down in a giant notebook. As the case goes on, the notebook gets thicker and thicker.

Eventually, the notebook becomes so huge that:

It's too heavy to carry: The detective gets tired just reading the beginning of the book before they get to the new clues (this is called "context rot").
It doesn't fit in the briefcase: The notebook is bigger than the size limit of the briefcase the detective is allowed to use (the "context window").

To fix this, the detective usually stops and asks a scribe (the LLM) to summarize the whole notebook into a tiny, 1-page cheat sheet. But the paper you provided points out three big problems with this traditional method, and then offers a clever new solution called Parallel Compaction.

Here is the breakdown in simple terms:

The Problem: The "One-Size-Fits-All" Summary

The authors found that the traditional way of summarizing is broken in three specific ways:

The "Black Box" of Length: You might tell the scribe, "Make this summary super detailed!" or "Make it super short!" But the scribe ignores you. No matter how long the original notebook is (2 pages or 200 pages), the scribe always writes a summary that is roughly the same size (about half a page). They have a "mental rule" from their training that says, "A summary is always this long," and they won't change it.
The "Wait Time" Bottleneck: Because the scribe has to read the entire giant notebook before writing a single word, the detective has to stand still and wait. In a fast-paced investigation, this waiting time adds up to minutes or even hours of lost productivity.
The "Roll of the Dice" Instability: If you ask the scribe to summarize the same notebook twice, they might give you two completely different summaries. One time they keep the clue about the red car; the next time, they forget it and keep the clue about the blue hat. This makes the detective's performance unpredictable.

The Solution: The "Assembly Line" (Parallel Compaction)

Instead of asking one scribe to read the whole book and write one summary, the authors suggest hiring a team of scribes and splitting the work.

Imagine the giant notebook is a long train.

Old Way: One person walks the whole length of the train, reads every car, and writes one report.
New Way (Parallel Compaction): You cut the train into smaller, equal-sized cars (blocks). You hand one car to Scribe A, the next to Scribe B, and so on.

Here is the clever twist:
When Scribe A summarizes their car, they also get to see the previous cars (the history) so they understand the context. Scribe B sees cars 1 and 2, Scribe C sees cars 1, 2, and 3, and so on. They all work at the same time (in parallel).

Why This Works Better

The paper claims this "Assembly Line" approach solves the three problems above:

Total Control (The Volume Knob): Since you decide how many cars (blocks) to cut the train into, you control the final size. If you want a huge summary, you use tiny blocks (more scribes, more detail). If you want a tiny summary, you use big blocks (fewer scribes, less detail). You don't have to beg the AI to follow instructions; you just change the number of workers.
Speed (The Traffic Jam): Because all the scribes are working at the same time, the total time to finish the summary is much faster. It's like having 10 people wash a car at once instead of one person doing it alone.
Stability (The Consistent Recipe): Because each scribe is only responsible for a small, focused chunk of the story, they are less likely to get confused or forget things. The summary becomes much more consistent, run after run.

The Bottom Line

The paper shows that for long, complex AI tasks, you shouldn't rely on a single AI to summarize a massive history. Instead, you should chop that history into small pieces, summarize them all at once using a smart layout that keeps the context connected, and stitch the results together.

This gives the operator (the human in charge) a precise "volume knob" to decide exactly how much information to keep, while making the whole process faster and more reliable.

Technical Summary: Parallel Context Compaction for Long-Horizon LLM Agent Serving

1. Problem Statement

Long-horizon Large Language Model (LLM) agents operate in sessions where conversation histories grow continuously, eventually exceeding model context windows. While context compaction via LLM-based summarization is the standard remedy, the authors identify three critical limitations in the current sequential synchronous approach:

Lossy and Unpredictable Output: Summarization is inherently lossy. Crucially, the volume of the summary is largely invariant to input length and prompt instructions. As input context grows from 2k to 96k tokens, the model self-bounds its output, retaining only a tiny fraction (dropping from ~48% to ~3% of tokens) of the original information.
Run-to-Run Instability: Even with identical inputs and prompts, the length and semantic content of summaries fluctuate significantly across runs. This instability makes agent behavior non-reproducible and complicates memory budgeting.
Latency Bottlenecks: Sequential compaction requires a blocking call that stalls agent inference. In long-horizon flows, this overhead can consume up to 62% of the total end-to-end (E2E) wall time, particularly when compaction triggers frequently at lower thresholds.

Furthermore, the paper notes that prompt engineering (e.g., asking for "concise" vs. "detailed" summaries) fails to provide fine-grained control over summary volume, as models tend to revert to internalized priors on summary length derived from training data.

2. Methodology

Experimental Setup

The authors evaluated their approach across four backbones spanning different scales and architectures:

Llama-3.1-8B (Small, Dense)
gpt-oss-20B (Medium, MoE, Reasoning)
Llama-3.3-70B (Large, Dense)
gpt-oss-120B (Large, MoE, Reasoning)

Evaluation was conducted on two benchmarks:

HotpotQA: Multi-hop question answering over accumulated evidence.
LoCoMo: Long-context multi-session dialogue with multiple QAs per session.

Proposed Solution: Parallel Compaction

The paper introduces Parallel Compaction, a block-based design that replaces the single blocking summarization call with concurrent processing. The workflow consists of three phases:

Snapshot & Partition: The conversation history is copied and partitioned into $N$ contiguous blocks of size $B$ tokens.
Dispatch: Each block $k$ $k$ is processed by a separate worker. Crucially, the prompt for worker $k$ $k$ includes the full conversation history up to block $k$ , with block $k$ $k$ itself wrapped in a <TARGET_BLOCK> marker.
- Prefix-Aware Target-at-End Layout: This design ensures each worker sees the causal history preceding its target block (preserving cross-block context) while the target block sits at the end of the prompt where attention is strongest. This layout also enables prefix cache reuse across workers in the vLLM server.
Merge: Once all workers complete, the $N$ per-block summaries are concatenated to form the compacted history.

3. Key Contributions and Characterization

Characterization of Sequential Compaction

The authors empirically demonstrated that sequential compaction suffers from:

Input/Prompt Invariance: Output token counts remain nearly flat (growing only ~3x) despite a 48x increase in input length. Prompt instructions regarding length have negligible impact.
Instability: As context length increases, the Coefficient of Variation (CV) in output length and the drop in cosine similarity (semantic consistency) increase, making outcomes unpredictable.
Optimal Block Size: Analysis revealed that 4k tokens is the most stable input size for summarization, offering the best trade-off between context retention and output consistency.

Characterization of Parallel Compaction

Parallel compaction addresses the limitations of the sequential baseline through:

Fine-Grained Control: The operator gains direct control over summary volume via block size ( $B$ ). Smaller blocks result in more parallel workers and higher total output tokens, preserving more context.
Restored Prompt Sensitivity: Unlike the sequential baseline, parallel compaction allows prompt instructions (e.g., "very detailed") to effectively tune output volume within each block, providing a two-axis control mechanism (block size + prompt detail).
Throughput Improvements: By parallelizing the workload and leveraging prefix caching, the system achieves higher compaction throughput.

4. Results

Accuracy and Information Retention

Accuracy Gains: On both HotpotQA and LoCoMo, reducing the block size (increasing parallelism) led to a monotonic increase in downstream QA accuracy. The 16k block size (fewest workers) produced the fewest tokens and lowest accuracy, while smaller blocks (e.g., 2k) preserved more context and improved accuracy.
Context Preservation: Parallel compaction allows the system to retain a significantly higher percentage of the original context (up to ~~50% with small blocks and detailed prompts) compared to the sequential baseline (~~3%).

System Performance (Wall Time and Throughput)

End-to-End Speedup: Parallel compaction reduced E2E wall time in most configurations. For example, on the 120B model with HotpotQA, parallel compaction (2k blocks) achieved a 1.28x speedup over sequential.
Throughput: At matched compaction decode volumes, parallel compaction consistently outperformed the sequential baseline. For instance, the 70B model showed a 2.13x throughput improvement at matched token counts.
Prefill Overhead: The paper notes that very large block sizes (e.g., 16k) can suffer from prefill overhead because the unique portion of the prompt per worker is large and uncached, negating some parallelism gains. Smaller blocks (2k–4k) optimize this balance.

5. Significance and Claims

The paper claims that Parallel Compaction offers a practical, infrastructure-agnostic solution for long-horizon agent serving without requiring offline training, retrieval infrastructure, or white-box model access.

Predictability: It transforms context compaction from an unpredictable, lossy process into a tunable system where operators can precisely control the trade-off between context retention and summary volume via block count and prompt engineering.
Efficiency: It mitigates the latency bottleneck of synchronous compaction, enabling higher throughput and faster agent inference cycles.
Stability: By operating on smaller, stable 4k blocks, it reduces the run-to-run variance inherent in summarizing massive contexts, leading to more reproducible agent behavior.

The authors conclude that the optimal operating point is not necessarily the maximum context retention, but the "minimum sufficient" set of high-signal tokens required for the task, a target that parallel compaction allows operators to hit with precision.