HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
HarnessCompass is a novel automatic harness evolution framework that enhances agent performance and generalization by enforcing task-agnostic constraints, integrating proactive agent feedback, and decoupling component optimization, thereby achieving superior results on SWE-bench Verified compared to existing methods.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you've just built a brilliant, super-smart robot brain. It can read books, write code, and solve puzzles better than almost anyone else. But there's a catch: this brain is like a genius who has never left their bedroom. It doesn't know how to talk to the outside world, how to use a screwdriver, or even how to ask for help when it gets stuck. To make this brain actually do things in the real world, you need to build it a "body" and a "control panel." In the world of artificial intelligence, this control panel is called a harness. It's the software layer that tells the AI how to see the computer screen, what tools it can use, and how to react when things go wrong. Think of it as the translator between the AI's wild thoughts and the rigid rules of a computer.
For a long time, engineers had to build these control panels by hand, tweaking them like a mechanic tuning a race car. But as AI brains get smarter, the old control panels become the bottleneck. They can't keep up. So, scientists started trying to let the AI fix its own control panel automatically. They created "meta-agents" (little supervisors) that watch the AI work, see where it fails, and try to rewrite the rules to make it better. It sounds like a dream, but there's a problem: these supervisors often get too clever for their own good. They start memorizing the specific test questions instead of learning general skills, kind of like a student who memorizes the answers to a practice quiz but fails the real exam because the questions are slightly different. They also tend to blame the wrong things for mistakes and try to fix everything all at once, which often makes the system more confused.
This is where a new framework called HarnessCompass comes in. The researchers behind it realized that to build a truly smart, adaptable control panel, you can't just let the supervisor run wild. You need to give it a compass. Instead of letting the AI make any change it wants, HarnessCompass forces it to follow three strict rules. First, it bans "overfitting." The AI isn't allowed to hard-code answers for specific problems; it has to learn general principles that work for any problem, not just the ones it's practicing on. Second, it asks the AI to speak up. Instead of just looking at the final score, the system asks the AI, "Hey, where did you feel stuck? What tool did you wish you had?" This gives the supervisor a first-person perspective on the struggle, rather than just seeing the crash from the outside. Third, it stops the AI from trying to fix the engine, the tires, and the steering wheel all at the same time. Instead, it optimizes them one by one, making sure a fix for the engine doesn't accidentally break the brakes.
The results of this disciplined approach are impressive. When tested on a massive set of real-world coding challenges called SWE-bench Verified, HarnessCompass took an AI agent and boosted its success rate from 54% to 66% in just 5 rounds of evolution. That's a huge jump, and it happened much faster than previous methods, which needed 20 rounds to reach a lower score of 63%. But the real magic isn't just the speed; it's the generalization. When the researchers took the control panel evolved for one specific AI model and tested it on a completely different AI model (one it had never seen before), it still worked better than the original setup. This suggests that HarnessCompass didn't just teach the AI to memorize the test; it actually taught it how to be a better engineer in general. The paper suggests that by adding these layers of discipline—constraining the changes, listening to the AI's own complaints, and fixing parts separately—we can build AI systems that are not only powerful but also reliable enough to handle the messy, unpredictable real world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.