On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification
This paper exposes the fragility of memory-based self-improving agents by demonstrating that their performance is highly sensitive to evaluation variance and task ordering due to underspecification, thereby advocating for more rigorous evaluation protocols and enhanced human oversight.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a digital assistant that doesn't just follow a script but learns from its own mistakes, keeping a written journal of what worked and what didn't so it can get better at future jobs. This is the promise of self-improving agents, a type of artificial intelligence designed to navigate complex digital worlds, like web browsers, and refine its skills over time. Instead of being reprogrammed by a human every time a new challenge arises, these systems maintain a textual memory bank, storing lessons from past tasks to guide their next actions. The hope is that they could eventually automate routine work, adapt to new situations, and discover solutions humans might miss. But for these systems to be trusted in the real world, they must be reliable. If an agent makes a mistake, it shouldn't just be a one-time glitch; the error shouldn't snowball, causing the system to learn the wrong lessons and fail repeatedly in ways that are hard to predict.
A team of researchers at Salesforce AI Research decided to put this promise to a rigorous test. They took two popular methods for building these self-improving agents and subjected them to a stress test that went far beyond how they had been evaluated before. Rather than running a single experiment and reporting the result, they ran the same set of tasks multiple times to see how much the results varied. They also shuffled the order in which the tasks appeared, removing the neat, easy-to-hard progression that previous studies had used. What they found was a picture of fragility. The agents were not the steady, reliable learners they were hoped to be. In fact, the very process of trying to make them self-improve often made them less stable, amplifying small errors into large swings in performance.
The researchers discovered that the performance of these agents was surprisingly sensitive to chance. When they ran the same experiment three times, the results often differed significantly. In some cases, the gap between the best run and the worst run was as large as ten percentage points. This means that an agent might succeed brilliantly in one attempt and fail miserably in the next, even though the underlying technology and the tasks were identical. When the self-improving mechanism was turned on, this instability often got worse. The system would learn from its first few tasks, but because the initial steps were influenced by random chance, the "lessons" it stored could be slightly off. These small, random deviations would compound over time, leading the agent down a path that was very different from its peers. The researchers noted that in nearly three-quarters of their test cases, adding the self-improving loop increased the variance, making the agent's behavior less predictable rather than more consistent.
Another major finding was that these agents relied heavily on the order in which they encountered tasks. Previous studies had mostly tested them on tasks arranged from simple to difficult, a setup that acted as a hidden training guide. The agents seemed to learn well in this environment because they were gently eased into harder problems. However, when the researchers scrambled the order, presenting the tasks in a random sequence, the agents' performance dropped. Instead of getting better, they actually performed worse than they did without any memory at all. This suggests that the agents were not truly learning general skills; they were simply memorizing a specific sequence of events. When that sequence was broken, the agents struggled to adapt, revealing that their "intelligence" was fragile and tied to a specific, artificial setup.
To understand why this was happening, the researchers looked closely at the notes the agents were writing in their memory banks. They found that the agents were often learning the wrong things because the instructions they received about the world were incomplete. For example, the agents were sometimes told to use computer code or application programming interfaces to solve problems, even though the environment they were working in was a standard web browser that could not run such code. The agents would write these impossible strategies into their memory, and then try to use them again and again, getting stuck in loops of failure. Similarly, when a task was vague, the agents would overthink it, creating elaborate but incorrect explanations for why they failed, which they would then try to apply to future, unrelated problems. One agent even started using a complex mathematical formula to estimate travel times because the map website it was using failed to load, and it decided to "learn" this workaround as a permanent strategy, even though it was a fluke.
The team tried to fix these issues by giving the agents more detailed information about their environment and the rules of the game. They provided specific feedback on why a task failed and clarified what actions were actually possible. This helped. When they added these details, the agents performed better, and the drop in performance caused by random task ordering was reduced by about a third. However, a significant gap remained. Even with better instructions, the agents still struggled in the most challenging, random conditions. This suggests that while providing clearer rules helps, there are deeper, uncharacterized reasons why these systems are so fragile. The agents are still prone to learning "wrong lessons" that can cascade into future failures.
The work concludes with a call for a more cautious approach to developing and testing these systems. The researchers argue that we cannot rely on single-run tests or neatly ordered task lists to judge if an agent is ready for the real world. Instead, we need to stress-test them under messy, unpredictable conditions and report how they perform across many different runs. They also emphasize the need for human oversight. Because these agents can learn incorrect strategies from incomplete information, there must be a way for humans to intervene, spot the bad lessons, and correct them before the agent causes irreversible damage. The path forward is not just about building smarter agents, but about building systems that are robust enough to handle the uncertainty of the real world without falling apart.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.