AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
AutoSaddler is an automatic framework that optimizes LLM agent harnesses by iteratively diagnosing failure traces from execution logs and generating structured code patches, significantly improving agent robustness and performance on long-horizon tasks across multiple benchmarks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, researchers are increasingly building systems that do more than just answer questions; they act. These "agents" are software programs powered by large language models that can plan a sequence of steps, use digital tools, and interact with a computer environment to solve complex problems. Imagine asking a computer to organize a week's worth of travel, book flights, and reserve hotels. The agent doesn't just give you a list; it logs in, searches, compares prices, and makes the bookings. However, these systems often struggle when tasks become long or complicated. A small mistake early in the process, like misinterpreting a single instruction, can cascade into a total failure hours later. To prevent this, engineers build a protective layer of code around the agent, known as a "harness." This harness acts as a supervisor, checking the agent's work, managing its tools, and ensuring it stays on track. For years, designing this harness has been a slow, manual job, requiring human experts to tweak instructions and settings by hand, a process that is difficult to scale and often leaves the system fragile.
A team of researchers has introduced a new method called AutoSaddler that automates this entire design process. Instead of relying on human intuition to fix the supervisor code, the system treats the harness itself as a piece of software that can be improved through a rigorous cycle of testing and learning. The researchers formulated the problem as an offline learning task, meaning the system learns from a collection of past failures rather than trying to figure things out in real-time. They fed the system batches of tasks that the agent had previously failed to complete. The system then analyzed the detailed records of these failures, known as execution traces, to understand exactly where and why the agent went wrong. It did not just guess; it dug into the code and the logs to find the root cause, much like a mechanic diagnosing a car engine by listening to the noise and checking the parts rather than just guessing which bolt to tighten.
Once the system identified the problem, it generated a specific, structured fix, or "patch," to the harness code. These patches were not random edits; they were carefully categorized into three types: changes to the agent's instructions, changes to the tools it could use, and changes to the logic that controlled its behavior. The system then tested these patches immediately on the same batch of tasks to see if they actually worked. If a patch fixed the immediate problem without breaking anything else, the system moved to a second, crucial step: checking if the fix would work on new, unseen tasks. This step was vital to ensure the system wasn't just memorizing the answers to the specific problems it had seen but was actually becoming smarter in a general way. The researchers built a memory system, a directed graph that tracked the history of every change, every success, and every failure, allowing the system to learn from its entire history rather than just the most recent attempt.
The results of this approach were significant. When tested on three different benchmarks involving complex tasks, the system consistently outperformed the original, manually designed harnesses. On a set of general assistant tasks, the automated system improved the success rate by nine percentage points. On a benchmark for software engineering tasks, it improved by nearly ten points, and on a terminal-based problem-solving test, it also gained ten points. These gains were substantial enough to surpass not only the original manual designs but also other automated methods that attempted to optimize the system. The researchers found that the key to this success was not just trying many random changes, but rather focusing on deep diagnosis, making targeted structural changes, and rigorously selecting only those changes that proved to be broadly useful.
The study explicitly ruled out several common approaches that might seem intuitive but proved ineffective. The researchers showed that simply reflecting on a failure without deep investigation into the code and logs led to shallow fixes that did not hold up. They also demonstrated that allowing the system to make unconstrained, random edits to the code resulted in a chaotic search that rarely found the right solution, often getting stuck on minor text tweaks while ignoring major structural improvements. Furthermore, they found that optimizing only for the specific tasks seen during training caused the system to overfit, meaning it became excellent at those specific tasks but failed when faced with new variations. The system only succeeded when it was forced to validate its changes against a separate set of unseen tasks, ensuring the improvements were durable and general.
Through extensive testing, the researchers confirmed that their method was robust and efficient. The system reached high levels of performance using far fewer computer resources than competing methods, requiring roughly ten times fewer attempts to learn the same lessons. This efficiency came from the system's ability to learn deeply from each failure, rather than just trying to brute-force a solution through millions of random guesses. The study suggests that the future of reliable AI agents lies in these automated, self-improving frameworks that can systematically debug and refine their own operating instructions. By treating the harness as code that can be evolved through evidence-based diagnosis and structured repair, the researchers have opened a path toward more capable and trustworthy artificial intelligence systems that can handle the long, complex tasks of the real world without constant human intervention.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.