💬 NLP

Self-Compacting Language Model Agents

The paper introduces SelfCompact, a training-free scaffolding framework that combines a model-invoked compaction tool with a lightweight rubric to enable adaptive, context-aware summarization of agent traces, significantly reducing token costs while improving performance on math and search benchmarks compared to fixed-interval approaches.

Original authors: Tianjian Li, Jingyu Zhang, William Jurayj, Xi Wang, Chuanyang Jin, Mehrdad Farajtabar, Eric Nalisnick, Daniel Khashabi

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianjian Li, Jingyu Zhang, William Jurayj, Xi Wang, Chuanyang Jin, Mehrdad Farajtabar, Eric Nalisnick, Daniel Khashabi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very difficult puzzle, like a complex math problem or a deep web search. You have a brilliant assistant (the AI) who is working on it. But here's the catch: the assistant has a very short-term memory. It can only hold a certain amount of information in its head at once.

As the assistant works, it writes down its thoughts, searches for clues, and makes notes. Eventually, this "scratchpad" gets so full that it spills over. When that happens, the assistant has to throw away old notes to make room for new ones.

The Problem: The "Rotting" Scratchpad
The paper calls this phenomenon "Context Rot."

Imagine your assistant is writing a story. If they keep writing without ever editing, the beginning of the story gets pushed so far back in their memory that they forget the main character's name. Or worse, they might forget a crucial clue they found five minutes ago because they were too busy writing about a dead-end idea they had an hour ago.

Current systems try to fix this by using a rigid timer. They say, "Every time you write 10,000 words, stop and summarize what you've written so far."

The Flaw: This is like stopping a chef in the middle of chopping onions just because they've been cooking for 10 minutes. They might be in the middle of a perfect recipe step, and summarizing now forces them to throw away the half-chopped onion and the specific spice they just added. They lose their place and have to start guessing.

The Solution: SELFCOMPACT (The Self-Editing Assistant)
The authors propose a new system called SELFCOMPACT. Instead of a timer, they give the assistant two new tools:

A "Summarize" Button: The assistant can choose to summarize its own notes.
A "Rubric" (A Rulebook): This is a simple checklist the assistant reads before hitting the button.

The rulebook asks the assistant three questions before it is allowed to summarize:

Did I just finish a specific step? (e.g., "I found the answer to this sub-question.")
Am I stuck in a loop? (e.g., "I'm just repeating the same search over and over.")
Is it safe to throw away the old notes? (e.g., "I have all the facts I need for this part.")

The Analogy: The Detective's Case File
Think of the AI as a detective solving a mystery.

The Old Way (Fixed Timer): Every hour, a strict supervisor walks in and says, "Stop! Summarize your case file." The detective might be in the middle of connecting two clues. The supervisor forces a summary, and the detective accidentally throws away the half-finished connection. The detective gets confused and starts guessing.
The New Way (SELFCOMPACT): The detective has a rulebook. They only summarize when they have successfully solved a specific clue (like "The butler did it") or when they realize they are stuck in a dead-end alley.
- If they just solved a clue, they summarize: "Okay, the butler did it. Now I need to find the motive." They throw away the messy notes about the butler's alibi because they are now solid facts.
- If they are still trying to figure out who did it, the rulebook says, "Don't summarize yet! You're still thinking."

What the Paper Found
The researchers tested this on seven different AI models using hard math problems and complex web searches.

Better Results: The AI using the "Rulebook" (SELFCOMPACT) got more answers right than the AI using the "Timer" or the AI that never summarized. On math problems, it improved scores by up to 18 points.
Cheaper: Because the AI only summarizes when it's truly necessary, it doesn't waste time or money summarizing when it doesn't need to. It saved 30% to 70% in costs compared to doing nothing.
No Training Needed: The best part is that they didn't have to teach the AI how to do this. They just gave it the tool and the rulebook, and the AI figured out how to use them on its own.

The Big Takeaway
The paper shows that AI models are actually quite good at knowing when they are "stuck" or when they have "finished a step," but they need a little nudge (the rulebook) to realize it. By letting the AI decide when to clean its memory, rather than forcing it on a schedule, we get smarter, cheaper, and more reliable results.

What the Paper Does NOT Claim

It does not claim this works for medical diagnosis or clinical uses.
It does not claim this will fix all AI problems forever.
It does not claim that the AI is "conscious" or "aware" in a human sense; it just follows the rules provided to it very well.

Technical Summary: Self-Compacting Language Model Agents

Problem: Context Rot and Rigid Compaction

As language model (LM) agents tackle longer-horizon tasks—ranging from competition math requiring tens of thousands of tokens to agentic search orchestrating hundreds of tool calls—their generated trajectories accumulate "stale content." This phenomenon, termed context rot, occurs when flawed early reasoning, outdated search results, or abandoned candidate programs anchor subsequent generations, degrading performance.

Existing solutions manage this via fixed-interval compaction, triggering summarization when a token threshold (e.g., 30% of the context window) is met. The paper argues this approach is fundamentally flawed because it ignores the trajectory structure. A fixed trigger cannot distinguish between a model that has just resolved a sub-problem (a safe time to summarize) and one that is mid-derivation or stuck (a dangerous time to summarize). As illustrated in the paper's Figure 1, poorly timed summarization can wipe verified facts mid-reasoning, forcing the model to re-derive information or guess, thereby wasting computational budget and reducing accuracy.

Methodology: SELFCOMPACT

The authors propose SELFCOMPACT, a training-free scaffold that allows the LM agent to decide when and how to compact its own context. The system operates without fine-tuning or external supervision, relying on two inference-time elements paired together:

The Compaction Tool: An inline summarization tool ( $S$ ) that the model can invoke to summarize the accumulated context ( $x, y_{1:t}$ ) into a condensed version ( $\tilde{y}$ ). The generation then resumes from the new context $(x, \tilde{y})$ .
The Lightweight Rubric: A set of task-specific criteria that gates the invocation of the tool. At periodic probe intervals (every $N$ $N$ tokens or tool calls), the model is asked to evaluate its current state against the rubric.
- When to Fire (COMPRESS): When a sub-task is resolved (e.g., a final answer is boxed in math, or a search unit is closed with verified facts) or the trajectory is converging.
- When to Suppress (CONTINUE): When the model is mid-derivation, mid-search, or stuck in a loop.

Implementation Details:

KV Cache Reuse: To minimize cost, the rubric probe and summarizer prompts are appended to the existing context rather than substituting it. This preserves the Key-Value (KV) cache of the trajectory ( $y_{1:t}$ ), meaning the model only pays for the prefill of the new instruction, not the entire history.
Workflow: The scaffold runs a loop where the model generates, pauses at intervals, evaluates the rubric, and conditionally triggers summarization. If the rubric returns COMPRESS, the summarizer fires; otherwise, the probe is discarded, and generation continues.

Key Contributions

Novel Scaffold: Introduction of SELFCOMPACT, which exposes summarization as an inline tool paired with a rubric that dictates invocation timing. This requires no parameter updates.
Empirical Validation: Demonstration across seven models (four Qwen variants for math; three deployed agents for search) and six benchmarks (IMO-Answerbench, HMMT, BrowseComp, etc.).
Ablation Insights: Proof that the rubric is essential. Providing the tool alone leads to uneven, often unhelpful usage (models calling it at wrong times or not at all). The rubric closes this "meta-cognitive gap," enabling reliable adaptive compaction.

Results

The paper reports that SELFCOMPACT matches or exceeds fixed-interval summarization while significantly reducing token costs.

Competition Math: On benchmarks like IMO-Answerbench and HMMT, SELFCOMPACT improved accuracy over a no-compaction baseline by up to 18.1 points. It outperformed fixed-interval summarization in 11 out of 12 benchmark/model configurations.
Agentic Search: On BrowseComp and DeepSearch QA, SELFCOMPACT added 5–9 accuracy points over the baseline.
Cost Efficiency: Despite the overhead of an extra LLM call for the rubric probe, SELFCOMPACT achieved 30–70% lower per-question token costs compared to the no-compaction baseline. This is because it avoids the massive cost of processing stale, unsummarized context in later turns.
Timing Analysis: Unlike fixed-interval methods that compress strictly at a 30% threshold, SELFCOMPACT's rubric triggers compression earlier (often when a sub-goal is resolved), preventing the accumulation of "rot" before it becomes harmful.

Significance and Claims

The paper claims to be the first to introduce and evaluate rubric-based adaptive compaction timing in a training-free setting. Its primary significance lies in reframing "when to compact" not as a capability that must be baked into model weights via fine-tuning or reinforcement learning, but as a meta-cognitive capability that can be supplied via a lightweight scaffold.

The authors highlight a "meta-cognitive gap": while unprompted models cannot reliably detect when their own context is rotting, a simple rubric bridges this gap. This suggests that complex context management strategies can be elicited from off-the-shelf models without the computational expense of training, offering a practical solution for deploying long-horizon agents on current hardware constraints.

The work remains modest in its scope, focusing on training-free interventions and acknowledging that frontier models with stronger inherent introspection might eventually require less scaffolding. However, for the current generation of open-weight and deployed agents, SELFCOMPACT offers a robust, cost-effective mechanism to sustain performance over extended reasoning horizons.