EvoClaw: Evaluating AI Agents on Continuous Software Evolution
The paper introduces EvoClaw, a novel benchmark utilizing the DeepCommit pipeline to evaluate AI agents on continuous software evolution, revealing a critical performance gap where agents' success rates drop significantly from isolated tasks to long-term maintenance scenarios due to struggles with error propagation and technical debt.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Idea: From "One-Off" Tasks to "Long-Haul" Driving
Imagine you are hiring a robot to fix your house.
- Old Benchmarks (The "One-Off" Test): You ask the robot to "fix the leaky faucet." It does it perfectly. You ask it to "paint the door." It does that too. You give it a score of 10/10.
- The Reality (Continuous Evolution): In the real world, you don't just fix one thing and leave. You fix the faucet, then you realize the pipes are old and need replacing, which messes up the wall, which requires repainting, which reveals a crack in the foundation. The robot has to keep working on the same house for months, dealing with the mess it made yesterday while trying to fix today's problem.
EvoClaw is a new test designed to see if AI agents can handle this "long-haul" reality, rather than just being good at isolated tasks.
The Problem: The "Snowball Effect"
The paper found a shocking truth: AI agents are great at starting, but terrible at finishing.
When tested on single, isolated tasks (like fixing just the faucet), top AI models score over 80%. But when asked to work on a continuous project where they have to build on their own previous work, their scores crash to under 40%.
The Analogy:
Imagine a student taking a math test.
- Isolated Test: They solve 10 separate, easy problems. They get an A.
- Continuous Test: They have to solve Problem 1, then use the answer from Problem 1 to solve Problem 2, and so on.
- The Result: The AI makes a tiny mistake on Problem 1. Because it didn't catch it, the answer for Problem 2 is wrong. By Problem 5, the answer is so garbled the whole thing collapses. The AI keeps making new features, but it breaks the old ones. This is called Error Accumulation or the Snowball Effect.
The Solution: DeepCommit (The "Smart Librarian")
To test this properly, the researchers needed a way to turn messy, real-world code history into a clear set of instructions. Real code history is like a chaotic diary: "Fixed a typo," "Added a feature," "Deleted a file," "Oops, that broke it, let's undo it."
They built a tool called DeepCommit.
The Analogy:
Think of a chaotic construction site with hundreds of workers shouting over each other.
- Raw Commits: The workers' individual shouts ("Move that brick," "Paint that wall," "Oops, drop that brick").
- DeepCommit: A smart foreman who listens to all the shouting, ignores the noise (like fixing a typo), and groups the work into logical Milestones.
- Milestone 1: "Build the Foundation."
- Milestone 2: "Frame the Walls."
- Milestone 3: "Install the Roof."
DeepCommit organizes these milestones into a map (a DAG) showing what must happen before what. It turns a messy history into a clear, step-by-step recipe.
The Test: EvoClaw (The "Endless Construction Project")
Using DeepCommit, they created EvoClaw, a benchmark where AI agents must act as the "foreman" for a software project.
How it works:
- The AI starts with a base code (the empty lot).
- It is given a stream of tasks (Milestones) one by one.
- It must fix the current task without breaking the previous ones.
- If it breaks something from Milestone 1 while working on Milestone 5, the whole project fails.
The Results:
- The "New Feature" Trap: The AI is very good at adding new things (Recall). It can build a new room.
- The "Regression" Failure: The AI is terrible at keeping the old things working (Precision). While building the new room, it accidentally knocks down the kitchen.
- The Verdict: Even the smartest AI models (like Claude Opus or GPT-5) struggle to maintain a codebase over time. They get stuck in a loop of fixing their own mistakes, eventually running out of steam.
Key Takeaways (The "Moral of the Story")
- We need to stop testing AI in a vacuum. Just because an AI can write a function doesn't mean it can maintain a software system for a year.
- The "Snowball" is real. Small errors early on destroy the ability to do complex work later.
- Exploration is key. The AI agents that did the best didn't just blindly type code. They spent time "reading the manual" and "checking the blueprints" (exploring the codebase) before making changes.
- Verification is the hero. The agents that frequently ran tests to check their work (even if it slowed them down) were much more successful than those who just guessed and hoped.
In a Nutshell
EvoClaw is a reality check for AI. It shows that while our AI agents are brilliant "one-hit wonders," they are currently terrible "long-term employees" who can't handle the messiness of real-world software evolution. We need to teach them how to manage their own mistakes before we can trust them with our digital infrastructure.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.