When Does Context Compaction Pay in Long-Horizon Agents?

Imagine a digital assistant that can work for hours, solving complex problems by reading files, running code, and checking its own progress. To do this, the assistant must remember everything it has done so far, keeping a running log of its actions and the results it sees. This log is called its "context." As the work continues, this log grows longer and longer, eventually becoming so large that it slows the assistant down and costs a significant amount of money to process. To manage this, software engineers have developed a trick called "context compaction." Instead of carrying the entire history forward, the system asks the assistant to write a short summary of what happened, discards the old details, and continues the work using only that summary. It is a bit like a traveler who, after walking a hundred miles, writes a postcard to their future self describing the journey, then burns the detailed diary to save space in their backpack. The hope is that this summary is enough to keep the traveler on track while saving time and resources.

A new study by researcher Yuyao Wang investigates whether this strategy actually works for long-term tasks. The researcher tested this idea on nine difficult, multi-step problems using two different advanced language models. They set up a controlled experiment where they saved the state of the agents at specific moments and then let them continue to the finish line using two different methods. In one method, the agents kept their full history, acting as a baseline. In the other, the agents switched to using summaries whenever their history grew too long. The researcher measured not just the final cost, but also how the change in strategy affected the agent's behavior, the number of steps it took, and the time it spent waiting for summaries to be written.

The results revealed a surprising twist. While the math suggested that switching to a summary should save money, the reality was often the opposite. For one of the models tested, the agents that switched to summaries ended up costing more money than those that kept their full history. The reason was not that the summaries were bad, but that the act of summarizing changed the agent's behavior in a way that created extra work. When the agent received a summary instead of the full history, it often felt the need to re-read files or check information it had already seen, leading to a cascade of extra requests. These extra steps, which the researcher called "read-only" actions, added up quickly. In many cases, the money saved by having a shorter history was completely wiped out by the cost of these extra, unnecessary checks. In fact, for the primary model tested, the strategy increased costs in 20 out of 21 different states when continued to the end of the task.

The study also looked at how long these agents had to work after the switch to determine if the strategy could ever pay off. The researcher found that context compaction only becomes a financial win if there is a very large amount of work left to do after the summary is written. If the task is nearly finished, the cost of writing the summary and the extra steps the agent takes to adjust to it are not worth the savings. However, for tasks with a long road ahead, the strategy can work, but only under specific conditions. The price of the technology matters greatly; for one model, the strategy only saved money if the cost of reading from the computer's memory was extremely high, far higher than what is currently charged by most providers. For the other model tested, the strategy was much more effective, saving money even with current prices, but this was largely because that model's behavior changed less drastically when it received a summary.

Beyond the cost, the researcher also examined how fast the agents worked. They simulated the process on a single computer chip to see if shorter histories would make the system faster. They found that while shorter histories did speed up the processing of individual steps, the time spent writing the summaries and the extra steps the agents took often made the entire process slower. In some scenarios, the agents using summaries took nearly twice as long to finish the same task as those with the full history, even though they were never waiting in a line. This happened because the time saved on processing the long history was lost to the time spent generating the summary and the extra work the agent performed.

The study concludes that context compaction is not a simple fix for long-running tasks. It is a trade-off that depends heavily on the specific model being used, the cost of the technology, and how much work remains. The researcher found that simply cutting the history short often confuses the agent, causing it to repeat work it has already done. To make this strategy work, systems need to be designed carefully to preserve the most recent steps and ensure the agent does not feel the need to double-check its own history. Until these conditions are met, keeping the full history, despite its cost, often remains the more reliable and sometimes even the cheaper path for agents tackling long, complex problems.