Technical Summary: ACON: Optimizing Context Compression for Long-horizon LLM Agents

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed as agents in dynamic, real-world environments where success depends on maintaining precise records of actions and observations over extended horizons. However, these long-horizon tasks lead to unbounded context growth, creating two critical bottlenecks:

1. Prohibitive Inference Costs: The memory cost of transformers scales with context length, primarily due to massive KV cache requirements, making long-horizon reasoning computationally expensive.
2. Reasoning Degradation: Excessively long contexts dilute relevant information, distracting the model with outdated or extraneous details and degrading decision quality.

Existing compression methods fail to adequately address these agent-specific needs. Dialogue-oriented systems focus on conversational coherence rather than state tracking, while document-centric methods assume single-step reasoning. Heuristic-based approaches are often brittle and narrowly specialized, while model-optimization-based approaches (e.g., Reinforcement Learning) require access to internal model weights, making them impractical for proprietary or API-based LLMs.

2. Methodology: Agent Context Optimization (ACON)

ACON is a unified framework designed to optimally compress both environment observations and interaction histories without altering the agent model's weights. It operates through two primary stages:

A. Compression Guideline Optimization (Natural Language Space)

Unlike methods that fine-tune model parameters, ACON optimizes the prompts (compression guidelines) used by a compressor LLM. This process is model-agnostic and relies on contrastive task feedback:

1. Contrastive Analysis: The system runs the agent on a task both with and without context compression. It identifies tasks where the agent succeeds without compression but fails with compression.
2. Feedback Generation: An optimizer LLM analyzes the successful baseline trajectory versus the failed compressed trajectory to generate natural language feedback identifying what critical information (e.g., variables, states, facts) was lost or distorted.
3. Iterative Refinement: This feedback is used to update the compression guideline prompt. The process involves two specific optimization steps:
   • Utility Maximization (UT): Focuses on maximizing task reward by ensuring critical state information is preserved.
   • Compression Maximization (CO): Conditions only on successful compressed trajectories to identify and remove redundant information, minimizing context cost while maintaining sufficiency.

B. Compressor Distillation

To mitigate the computational overhead of repeatedly invoking a large LLM for compression, ACON distills the optimized compressor into a smaller model.

• A "teacher" compressor (large LLM with optimized guidelines) generates compressed outputs from input histories/observations.
• A "student" model (smaller LLM) is trained via cross-entropy on these input-output pairs.
• This allows the expensive large model to be reserved for the agent's decision-making, while the lightweight student handles the compression.

3. Key Contributions

• ACON Framework: A unified approach for optimizing the compression of both interaction histories and environment observations tailored for multi-step, long-horizon agentic tasks.
• Failure-Driven Guideline Optimization: A model-agnostic method that refines compression prompts through natural language feedback derived from agent failures, eliminating the need for weight updates or access to proprietary model internals.
• Cost-Efficient Deployment: Demonstrates that optimized compressors can be distilled into smaller models, preserving over 95% of the teacher's performance while significantly reducing overhead.
• Empirical Validation: Extensive evaluation across three benchmarks (AppWorld, OfficeBench, and Multi-objective QA) showing that ACON reduces peak token usage by 26–54% while improving task success rates compared to existing baselines.

4. Experimental Results

The authors evaluated ACON on three benchmarks requiring 15+ interaction steps:

• Token Efficiency: ACON reduced peak token usage by 26–54% across benchmarks. For instance, on AppWorld, it reduced peak tokens by over 25% while maintaining accuracy comparable to the uncompressed baseline.
• Task Success: ACON improved task success rates over existing compression baselines (FIFO, Retrieval, LLMLingua, Naive Prompting). On AppWorld, it achieved a 32.4% relative improvement for the Qwen3-14B agent (from 25.6% to 33.9%) and a 45.6% gain on 8-objective QA.
• Distillation Performance: Distilled compressors (e.g., Qwen3-14B, Phi-4) retained over 95% of the performance of the gpt-4.1 teacher compressor.
• Small Agent Enablement: By mitigating context distraction, ACON enabled smaller models (like Qwen3-14B) to function effectively as long-horizon agents, achieving performance improvements of 20–46% depending on the benchmark.
• Cost Analysis: While the guideline optimization step is lightweight (under $2 per domain), the distillation of compressors to smaller models (e.g., Qwen3-14B) reduced compression API costs by 99.1% compared to using a large teacher model.

5. Significance and Claims

The paper claims that ACON lays a foundation for more general, memory-efficient, and deployable long-horizon LLM agents. Its primary significance lies in:

• Accessibility: By being model-agnostic and not requiring gradient updates, it makes advanced agentic capabilities accessible for proprietary API-based models where internal weights are unavailable.
• Resource Efficiency: It significantly lowers the computational footprint of AI systems by reducing token costs and enabling the use of smaller, cheaper models for both compression and agent execution.
• Equalization: It acts as an "equalizer," allowing smaller language models to approach the performance of much larger models by providing them with concise yet informative contexts, thereby lowering barriers to entry for researchers with limited resources.

The authors note limitations, including the primary focus on GPT models due to resource constraints and the inherent latency introduced by the generative compression process itself. However, they posit that the trade-off between reduced token costs and added compression latency is favorable for long-horizon tasks.