🤖 AI

AGORA: Adapter-Grounded Observation-Action Retention for Inference-Free Prompt Compression in LLM Agents

The AGORA paper proposes an adapter-grounded, inference-free prompt compression method for LLM agents that overcomes the structural limitations of existing token-level extractive compressors by isolating a structural floor as the primary quality bottleneck and achieving 1.0–11.5x adaptive compression through a learned scorer while maintaining near-uncompressed performance.

Original authors: Haoran Zhang, Zhaohua Sun

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoran Zhang, Zhaohua Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine an LLM agent as a highly skilled detective trying to solve a mystery. To solve the case, the detective looks at a long notebook filled with clues, actions they've taken, and what they saw afterward. As the investigation goes on, this notebook gets huge—sometimes over 100,000 words long. If the notebook gets too big, the detective gets overwhelmed and can't think clearly.

The problem is: How do you shrink this notebook without deleting the clues that actually matter?

The Problem: The "Wrong" Kind of Shrinker

Researchers tried using standard "summarizers" (called token-level compressors) that work great for normal reading tasks, like summarizing a news article. These tools work by deleting words that seem "predictable" or "boring."

But for an agent detective, this approach is a disaster. The paper calls this "Action-Grammar Destruction."

Here's the analogy:
Imagine the detective's notebook contains a specific instruction: "Search for [red shoes size 10]."

The words "Search," "for," and the brackets [ ] are the action grammar. They are the commands that tell the computer what to do.
However, because these words are so common and predictable in English, the standard summarizer thinks, "Oh, 'for' and brackets are boring. I'll delete them to save space."
The Result: The notebook now says "red shoes size 10."
The Crash: The computer environment sees this and says, "I don't know what to do with this. It's not a command." The agent crashes or fails.

The paper tested 17 different scenarios and found that every single time they used these standard summarizers, the agent's performance collapsed to near zero, even though they successfully saved space. The tool was working perfectly, but it was deleting the only things that made the agent work.

The Solution: AGORA (The Smart Filing System)

The authors created a new system called AGORA. Instead of acting like a word-scrubber, AGORA acts like a smart filing clerk who understands the structure of the detective's notebook.

AGORA works in three simple layers:

The "Never-Touch" Floor (Structural Parser):
AGORA has a rule: "Never delete the System Instructions, the Current Clue, or the last two steps."
- Analogy: Imagine the detective's notebook has a red sticky note on the last two pages and the cover page. AGORA says, "No matter what, we keep these pages exactly as they are." This ensures the agent always knows what it's doing and what just happened.
The "Relevance Score" (The 125M Parameter Scorer):
For the older pages in the notebook, AGORA uses a small, fast AI (a 125-million-parameter model) to ask: "If we remove this old clue, will the detective make a different decision right now?"
- If the answer is "No, this old clue doesn't matter," it gets deleted.
- If the answer is "Yes, this clue is critical," it stays.
- Key Point: This happens in about 2 milliseconds. It doesn't ask a giant AI to rewrite the story; it just checks if the page is important.
The "Greedy" Filler:
Once the "Never-Touch" pages are safe, AGORA fills the remaining space with the most important old clues until the notebook is small enough.

Why It's a Big Deal

No Extra Cost: Most other methods try to summarize the notebook by asking a second giant AI to do the work. This costs extra money and time for every single step. AGORA does it without asking a second AI, saving money and time.
It Works: In 8 out of 9 different test scenarios, AGORA kept the agent performing at 75% or higher of its original speed and accuracy, even with a much smaller notebook.
Adaptive: It automatically figures out how much to shrink the notebook. Sometimes it shrinks it by 1x (a little), sometimes by 11x (a lot), depending on how much "fluff" is in the history.

The Bottom Line

The paper proves that you can't just treat an AI agent's history like a normal book. You have to respect the "grammar" of its commands. AGORA is a lightweight, fast, and free way to keep the agent's memory clean without accidentally deleting the instructions that make it work. It's the difference between a summarizer that deletes the "Stop" sign from a road map and a filing system that keeps the map readable but removes the old, irrelevant street names.

Technical Summary: AGORA – Adapter-Grounded Observation-Action Retention for Inference-Free Prompt Compression in LLM Agents

1. Problem Definition

Large Language Model (LLM) agents accumulate long trajectories of actions and observations as they interact with environments. As task horizons extend (e.g., exceeding 100k tokens by step 30 on WebShop), compressing this context becomes a practical necessity. However, the paper argues that standard prompt compression techniques designed for general single-turn LMs are structurally inappropriate for LLM agents.

The core issue identified is action-grammar destruction. In agent trajectories, tokens carrying critical action semantics (identifiers, brackets, action verbs like search[...] or click[...]) are often highly predictable from the surrounding natural language context. Consequently, token-level extractive compressors—which rank tokens by self-information or perplexity and remove the lowest-scoring ones—systematically delete these semantic tokens. The resulting "compressed" prompt contains grammatical English but ungrammatical agent commands, causing the environment to reject the action and the agent to fail.

The authors demonstrate that across 17 experimental cells spanning two independent token-level method families (Selective Context and LLMLINGUA-2), every configuration collapses to a mean reward ( $m_r$ ) of $\leq 0.05$ , despite achieving 1.3–13.3× compression.

2. Methodology: AGORA

To address this, the authors propose AGORA (Adapter-Grounded Observation-Action Retention Architecture), an inference-free, step-level compressor. Unlike token-level methods, AGORA operates on the granularity of complete (action, observation) pairs, ensuring that action syntax remains intact.

AGORA consists of three primary components:

Structural Prompt Parser:
- Segments the agent's trajectory into role-typed blocks (System, Task, Past Steps, Current Observation, Pending Response).
- Groups past interactions into discrete step pairs $(a_i, o_i)$ .
- This parsing is deterministic and task-agnostic, relying on role markers (e.g., [ASSISTANT], [USER]).
Always-Keep Floor (Deterministic Quality Floor):
- Unconditionally retains specific critical content to preserve format and recency:
  - System message ( $c_{sys}$ ) and Task instruction ( $c_{task}$ ).
  - Current observation ( $c_{now}$ ) and any pending response ( $c_{pend}$ ).
  - The most recent $K_{recent}=2$ steps.
- This layer ensures that the "invariant" required for the agent to function (e.g., knowing the current state and recent history) is never lost.
Learned Step Relevance Scorer:
- A 125M-parameter RoBERTa-base classifier trained to score the relevance of remaining past steps against the current observation.
- Training Signal: The model is trained on counterfactual next-action-change labels. For each past step $s_i$ and current observation $c_{now}$ , the agent backbone is queried $K=8$ times (half with $s_i$ , half without). The label is the fraction of times the canonicalized next action changes. This creates a soft label $y_i \in [0, 1]$ indicating the step's criticality.
- Inference: The scorer runs a single forward pass ( $\sim 2$ ms/step) with no API calls. It outputs a probability $p_i$ that removing step $s_i$ would change the next action.
Greedy Budget Fill:
- Given a target character budget (e.g., 25% of original length), the system fills the remaining space after the "Always-Keep Floor" by greedily selecting non-floor steps in descending order of their relevance scores $p_i$ .
- Dropped steps are replaced by a single elision marker (e.g., [... N step(s) elided ...]).

3. Key Contributions

The paper makes three primary contributions:

Identification of Action-Grammar Destruction: The authors formally diagnose and name the structural failure mode where token-level compressors fail on agents. They provide systematic evidence (17 cells) showing that these methods collapse agent performance because they remove low-self-information tokens that are essential for action syntax.
AGORA Architecture: A novel, inference-free step-level compressor that combines a structural parser, a deterministic "always-keep" floor, and a learned relevance scorer trained on counterfactual labels. It achieves adaptive compression (1.0–11.5×) with zero per-step LLM overhead.
Component Attribution: Through ablation studies, the paper isolates the specific value of each design choice:
- The Always-Keep Floor is the dominant lever for quality retention (removing it drops performance significantly).
- The Learned Scorer is the primary driver of adaptive compression range, allowing the system to vary compression ratios based on trajectory complexity while maintaining a fixed keep ratio.

4. Experimental Results

The authors evaluated AGORA across 9 cells (3 environments: ALFWorld, ScienceWorld, WebShop; 3 backbones: qwen3.5-flash, gpt-4o-mini, gpt-5-mini) against 10 baselines (including token-level extractive methods and LLM-based abstractive compressors).

Quality Retention: AGORA is the only inference-free method to retain $\geq 75\%$ $\geq 75%$ of the uncompressed performance in 8 out of 9 cells (the exception being ScienceWorld with gpt-5-mini at 73%).
- Average mean reward ( $m_r$ ) with AGORA: 0.387 (92% retention vs. No-Compression).
- In contrast, token-level methods (Selective Context, LLMLINGUA-2) collapsed to $m_r \leq 0.05$ .
Compression Efficiency: AGORA achieves an adaptive end-to-end compression range of 1.0–11.5× using a single fixed configuration ( $\rho=0.25$ ), adapting to the specific trajectory regime without per-cell tuning.
Cost: AGORA incurs zero additional LLM token costs per step (inference-free). While LLM-based compressors (HiAgent, AgentDiet, ACON) often achieve lower costs per task in long-trajectory scenarios due to aggressive compression, they pay 8–126k extra tokens per task for their own compressor calls. AGORA is superior in scenarios where per-step LLM overhead is prohibitive or where retention is the binding constraint.
Ablation: Removing the "Always-Keep Floor" caused the largest performance drop ( $\Delta m_r \approx -0.088$ ). Removing the scorer reduced performance only slightly ( $\Delta m_r \approx -0.031$ ) but eliminated the adaptive compression range, confirming the floor handles quality while the scorer handles compression volume.

5. Significance and Claims

The paper claims that AGORA represents a necessary shift in how we approach prompt compression for agents. The authors argue that the "rank-and-delete" paradigm of token-level compression is structurally mis-specified for agents because it optimizes for information density rather than structural integrity of action commands.

Significance:

Paradigm Shift: It moves the design space from token-level to step-level compression, preserving the syntactic integrity of agent commands.
Inference-Free Viability: It demonstrates that high-quality compression for agents does not require expensive, per-step LLM calls, making it viable for deployment with frozen, black-box backbones.
Robustness: By prioritizing structural preservation (the "floor") over semantic summarization, AGORA avoids the catastrophic failure modes seen in general-purpose compressors.

The authors remain modest, noting that AGORA is not a universal cost winner (LLM-based methods often win on pure $/task in long-trajectory settings) and that their evaluation relies on point estimates due to wide confidence intervals ( $n=30$ ). However, they assert that AGORA provides the only viable inference-free solution that maintains high performance across diverse agent settings without breaking action grammar.