Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory

Imagine you are trying to solve a massive mystery, but you only have a tiny notebook to write down clues. This is the daily reality for "language agents"—smart computer programs powered by Large Language Models (LLMs) that chat with us, solve problems, and make decisions. These agents are like detectives who need to remember everything they've seen, heard, or read to get the answer right. But here's the catch: their "notebooks" (called context windows) have a strict page limit. If they try to stuff too many clues in, the notebook tears, or the detective gets overwhelmed and starts making mistakes. Plus, every extra page costs money and time to read.

So, the big question for these digital detectives is: How do you fit the most important clues into a tiny space without losing the key details? You have two main choices. You can either keep the raw clues exactly as they are (Retention), hoping they all fit, or you can rewrite and combine them into a shorter summary (Consolidation). The first method is safe but bulky; the second is efficient but risky because you might accidentally throw away a crucial fact while trying to save space. For a long time, scientists weren't sure which strategy was better or when to switch between them.

This paper, titled "Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory," tackles that exact dilemma. The researchers discovered that there is no single "best" way to manage memory. Instead, the right choice depends entirely on how tight the "budget" (the space limit) is. They found a fascinating "tipping point": when the budget is extremely tight, summarizing and merging clues is a lifesaver, boosting accuracy by up to 48% because it squeezes in information that would otherwise be left out. However, once the budget is loose enough to fit the raw clues comfortably, summarizing actually hurts performance because it blurs important details. The authors built a smart, lightweight tool called OAS (Offline Abstraction-Safety) that acts like a traffic cop, looking at the situation before writing anything and deciding: "Do we need to compress, or can we keep the original?" Their experiments show that this smart switching works better than just sticking to one rule, proving that knowing when to summarize is just as important as knowing how.

The Detective's Dilemma: To Keep or to Summarize?

Imagine you are a detective trying to solve a case, but your notebook is shrinking. You have a pile of evidence: witness statements, photos, and receipts. If you try to paste the whole photo into your notebook, it might not fit. If you don't fit it, you lose the evidence. If you try to describe the photo in a sentence, it fits, but you might miss a tiny detail like the color of the suspect's hat.

In the world of AI, this is the battle between Retention and Consolidation.

• Retention is like photocopying the evidence and taping it in. It's perfect because nothing is lost, but it takes up a lot of space.
• Consolidation is like writing a summary. It saves space and can combine ten pages of notes into one, but there's a risk: the summary might miss the one fact that solves the case.

For a long time, AI systems just picked one strategy and stuck with it. Some always summarized; others always kept the raw text. But the researchers in this paper asked a smarter question: Why not have a smart assistant that decides which strategy to use based on how much space is left?

The "Tipping Point" Discovery

The team ran a series of experiments using two famous puzzle sets (called benchmarks) to see what happens when you change the size of the notebook. They tested budgets ranging from very tiny (32 tokens) to quite large (256 tokens).

Here is what they found, and it's a bit like a seesaw:

1. When the budget is super tight (32 or 64 tokens): The raw evidence simply doesn't fit. If you try to keep the original text, you have to throw most of it away, and the AI gets a score of almost zero. But if you use Consolidation (specifically a method called "Abstract" or "Merge"), the AI can summarize the clues, fit them all in, and the accuracy jumps up by a massive 48%. In this zone, summarizing is a hero.
2. When the budget is loose (256 tokens): The raw evidence fits perfectly. In this case, summarizing becomes a villain. It starts to blur the details, and the AI's accuracy drops by about 8% to 11% compared to just keeping the original text. Here, "less is more" is a bad idea; you want "more is more."

The paper calls this the Budget Crossover. It proves that the best strategy isn't fixed; it flips depending on how much pressure you are under.

The Smart Traffic Cop: OAS

So, how does an AI know when to switch? The researchers built a tool called OAS (Offline Abstraction-Safety). Think of OAS as a super-smart traffic cop standing at the intersection of "Keep" and "Summarize."

Before the AI even starts writing a summary, OAS looks at a few clues:

• How much space is left?
• How many clues are there?
• How messy or repetitive are the clues?

Based on these pre-written notes, OAS predicts the score for each option. It asks: "If I summarize, will I gain more points by fitting in new info, or will I lose points by messing up the details?"

• If the predicted gain is high, OAS says, "Go! Summarize!"
• If the predicted gain is low or negative, OAS says, "Stop! Keep the original!"

The researchers tested this "traffic cop" on the puzzle sets. They found that OAS successfully learned to switch strategies at the right moment. When the budget was tight, it chose summarization and boosted the score. When the budget was loose, it chose to keep the raw text and avoided the drop in accuracy.

Which Summarizer is Best?

The paper also tested three different ways to summarize, like different styles of writing a summary:

1. Merge: Combining several notes into one big block.
2. Abstract: Writing a high-level summary that captures the main idea.
3. Rewrite: Rephrasing each note individually to make it shorter.

They found that when space is tight, Merge and Abstract are usually the winners. They are better at combining information from different notes to save space. Rewrite is often less effective because it treats each note separately and doesn't save as much space. However, the paper notes that there isn't one "perfect" summarizer for every single situation; it depends on the specific clues.

What the Paper Does NOT Say

It is important to know what this paper doesn't claim. The authors are very careful not to say that summarizing is always better, or that it's always worse. They explicitly rule out the idea that there is a single "magic number" of words that works for everyone. Instead, they show that the "magic number" changes depending on the situation.

They also don't claim that their tool (OAS) is perfect or that it works for every single AI in the world. They tested it on specific datasets and found it works well there, but they admit that in the real world, things might be messier. They also didn't find a way to make the summarization process itself perfect; they just found the best time to use it.

The Takeaway

The main lesson from this paper is that flexibility is key. Just like a human detective wouldn't use the same strategy for a small neighborhood theft as they would for a massive international conspiracy, an AI shouldn't use the same memory strategy for every question.

The researchers showed that by watching the "budget pressure," an AI can decide whether to keep its raw notes or compress them. When the space is tight, compressing is a lifesaver. When the space is open, keeping the raw details is the safest bet. Their tool, OAS, acts as the brain that makes this switch automatically, ensuring the AI gets the best possible score without wasting space or losing critical facts. It's a small but powerful step toward making AI detectives smarter, more efficient, and less likely to forget the clues that matter most.