Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios
This paper presents a process-oriented empirical analysis of 8 top-ranked software development agents on the SWE-Bench benchmark, revealing how Python execution errors correlate with resolution success, identifying prevalent and challenging error types, and uncovering three previously unknown bugs in the benchmark platform itself.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you hire a team of incredibly smart, AI-powered apprentices to fix broken machines in a giant, complex factory (which represents real-world software projects like those on GitHub).
Most people judge these apprentices by looking only at the final result: "Did they fix the machine? Yes or No?"
This paper says, "Wait a minute! That's not the whole story." The authors decided to put a camera on the apprentices' heads and watch everything they did while trying to fix the machine. They wanted to see the mistakes, the confusion, the dead ends, and the "aha!" moments that happen before the final fix is submitted.
Here is the breakdown of their findings using simple analogies:
1. The Setup: Watching the Process, Not Just the Product
The researchers looked at 500 real-world repair jobs assigned to 8 top AI agents. Instead of just checking if the job was done, they analyzed 3,977 "diary entries" (trajectories) of the agents thinking, typing, and running code, plus 3,931 test logs (the final report cards).
The Big Discovery:
- One mistake isn't the end of the world: If an agent makes a mistake early on but keeps going, it can often recover and still fix the problem. It's like a chef burning a toast but saving the meal by making a new one.
- Too many mistakes are a disaster: However, if an agent keeps making mistakes (more than 10 or 15), it gets overwhelmed. It starts taking too many "thinking steps," gets confused, and the final fix usually fails. It's like a driver who keeps hitting potholes; eventually, they run out of gas or get lost.
2. The "Most Common" Mistakes (The Daily Grind)
The authors found that these AI agents trip over the same few things repeatedly. Think of these as the "flat tires" of the software world:
- Missing Ingredients (ModuleNotFoundError): The agent tries to use a tool or library that isn't installed. It's like trying to bake a cake without flour.
- Wrong Shape (TypeError): The agent tries to put a square peg in a round hole (e.g., trying to add a number to a word).
- Database Drama: The agents struggle a lot with databases (where data is stored). They often break the rules of how data should be organized, causing "Integrity Errors." This is like trying to file a tax return with numbers that don't add up.
3. The "Nightmare" Mistakes (The Hard Ones)
Some errors are so tricky that even the smartest agents get stuck in a loop, making the same mistake over and over again.
- The "OSError" (System Errors): These are errors related to the computer's operating system, like trying to open a file that doesn't exist or permissions being wrong. The agents get stuck here more often than any other error. It's like a robot trying to open a locked door and just banging on it repeatedly.
- Database Integrity: When the agent messes up the database rules, it's very hard for them to fix it on their own. They often don't understand the "rules of the game" for that specific database.
4. The "Ghost" Errors (Cross-Phase Failures)
Sometimes, an agent makes a mistake while working on the code (the "Solving Phase"), thinks it fixed it, and submits the work. But the mistake was actually a "ghost"—it wasn't really gone. When the final tests run (the "Testing Phase"), the ghost comes back, and the whole project fails.
- The Lesson: The agents are bad at knowing when they have truly fixed a problem. They often submit work that still has hidden cracks in it.
5. The Plot Twist: The Test Itself Was Broken!
Here is the most surprising part. While analyzing why some tasks failed, the researchers found 3 bugs in the testing platform itself (SWE-Bench).
- Imagine a teacher grading a student's test, but the teacher's answer key is wrong.
- In one case, the AI agents actually fixed the problem perfectly, but the test platform said, "No, you failed."
- The authors reported these bugs to the platform creators, and the platform owners confirmed them. This proves that sometimes, the AI isn't the problem; the way we are testing them is flawed.
Why Does This Matter?
This paper is like a mechanic saying, "We've been judging cars only by whether they reach the destination. But if we look at the engine logs, we see they are wasting fuel, overheating, and hitting potholes."
The Takeaways for the Future:
- Teach them to handle mistakes better: We need to train these AIs to recover from errors faster so they don't get stuck in loops.
- Fix the testing ground: We need to make sure the "tests" we use to grade these AIs are actually fair and bug-free.
- Save energy: Every time an AI makes a mistake and has to try again, it uses electricity and computing power. By fixing these errors earlier, we can make AI software development "greener" and cheaper.
In short: Don't just look at the final code. Watch the journey. That's where the real problems (and solutions) are hiding.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.