AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering
The paper introduces AgentForge, an open-source multi-agent framework that enforces execution-grounded verification through a mandatory sandboxed environment, achieving a 40.0% resolution rate on SWE-Bench Lite and significantly outperforming single-agent baselines by leveraging execution feedback as a primary supervision signal.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to fix a broken machine in a giant, messy workshop. You have a very smart, but slightly overconfident, apprentice named LLM (Large Language Model).
The apprentice is great at guessing what the machine should look like. If you ask, "How do I fix this gear?" the apprentice will write down a perfect-looking plan and a new gear design. But here's the problem: The apprentice has never actually turned the wrench. They are guessing based on what they've read in books, not on what happens when the machine actually runs. Sometimes, their "perfect" gear fits on paper but jams the whole engine when you try to use it.
Most current AI systems are like this apprentice working alone. They guess, write code, and hope for the best.
AgentForge is a new way of working. Instead of one overconfident apprentice, AgentForge sets up a specialized construction crew that works together in a safe, isolated testing room (a "sandbox").
Here is how the crew works, using a simple analogy:
1. The Five Specialized Workers
Instead of one person doing everything, AgentForge splits the job into five distinct roles, like a movie production crew:
- The Planner (The Director): Before anyone touches a tool, this agent looks at the broken machine and the blueprints. They don't write code; they write a step-by-step script: "First, we need to check the engine block. Then, we swap the spark plug. Finally, we test the brakes."
- The Coder (The Mechanic): This agent follows the Director's script. They don't rewrite the whole car; they just swap out the specific broken part (a "patch"). They are careful to only change what's necessary.
- The Tester (The Quality Inspector): Once the Mechanic swaps the part, the Inspector doesn't just look at it. They run the engine. They try to drive the car to see if it actually works. If the car stalls, they scream, "It's broken!" and give the Mechanic the exact error log.
- The Debugger (The Fixer): If the Inspector says it's broken, the Fixer steps in. They look at the error log, figure out what went wrong, and try a different repair. They keep trying until the car runs smoothly.
- The Critic (The Final Judge): Once the car runs, the Judge looks at the whole process. Did we fix the problem? Did we break anything else? If yes, they sign off on the work.
2. The "Safe Room" (The Sandbox)
The most important part of AgentForge is the Sandbox.
Imagine a glass-walled garage where the crew works.
- Safety: If the Mechanic accidentally puts a bomb in the engine, it only explodes inside the glass garage. It doesn't blow up the real workshop or the computer running the AI.
- Truth: The crew can't guess if the car works. They must start the engine and listen to the sound. If the engine makes a weird noise, the system knows immediately. This is called "Execution-Grounded Verification." It means the AI learns from real results, not just its own imagination.
3. The Memory Bank
The crew also has a shared notebook (Memory).
- Episodic Memory: If they fixed a similar broken engine last week, they look it up to see what worked.
- Repository Index: They have a live map of the entire workshop, so they know exactly where every tool and part is located, preventing them from trying to use a wrench that doesn't exist.
Why is this better?
In the old way (Single-Agent), the apprentice guesses, writes a whole new engine, and hopes it works. They might get it right 14% of the time.
With AgentForge, the crew works in a loop:
- Plan.
- Build a tiny fix.
- Test it in the safe room.
- If it fails, fix it based on the real error.
- Repeat until it works.
Because they actually run the code to check if it works, they get it right 40% of the time—a huge jump from the old method.
The Bottom Line
AgentForge teaches AI that you can't just talk about fixing things; you have to actually do it and see what happens. By breaking the job into a team of specialists and forcing them to test their work in a safe, real environment, we get software that is much more reliable, much less likely to crash, and actually solves real-world problems.
It's the difference between a student who memorizes the answer key and a student who actually takes the test, gets the questions wrong, learns from the feedback, and tries again until they get an A.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.