Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
This paper presents a large-scale empirical study of 33,000 AI-generated pull requests on GitHub, combining quantitative analysis and qualitative taxonomy to reveal that while documentation and CI tasks succeed most often, agentic failures are driven by factors such as large code changes, CI failures, and misalignment with human reviewer expectations.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a bustling construction site where human architects and engineers are building software. Recently, a new crew of AI robots (called "coding agents") has arrived. These robots don't just whisper suggestions to the humans; they are now building entire rooms, painting walls, and even submitting their own blueprints (called "Pull Requests" or PRs) to be added to the building.
The paper you provided is like a forensic investigation into why some of these robot-built rooms get approved and added to the building, while others get rejected and torn down. The researchers looked at over 33,000 blueprints submitted by five different robot crews on GitHub to figure out where the robots are going wrong.
Here is the breakdown of their findings, using simple analogies:
1. The "Easy Wins" vs. The "Hard Sells"
The researchers found that the robots are great at doing simple, routine chores but struggle with complex, high-stakes tasks.
- The Robots' Superpower: They are like expert gardeners when it comes to documentation, fixing the building's schedule (CI/Build), and updating the rules. These tasks get approved about 80-90% of the time. It's like the robots are perfectly good at sweeping the floor or repainting the fence.
- The Robots' Weakness: They are terrible at fixing broken engines (bug fixes) or making the building run faster (performance). These tasks get rejected the most often. It's as if the robots try to fix a leaky pipe but end up flooding the basement.
2. The "Too Big to Handle" Problem
When a robot submits a blueprint that gets rejected, it's often because it tried to do too much at once.
- The Analogy: Imagine you ask a robot to "fix the kitchen." If it comes back with a plan to rebuild the entire house, move the foundation, and redesign the roof, the human architect will say, "No way, that's too much."
- The Finding: Rejected blueprints involved more lines of code and touched more files than the accepted ones. The robots tend to be "over-enthusiastic" and make massive changes that overwhelm the human reviewers.
3. The "Broken Test" Alarm
Before a human architect looks at a blueprint, the building's automated security system runs a check.
- The Finding: Rejected blueprints frequently failed these automated security checks (CI builds). It's like the robot submitted a plan that didn't pass the fire safety inspection. If the robot's code breaks the tests, the humans usually don't even bother looking at it closely.
4. Why Do Humans Say "No"? (The 4 Reasons)
The researchers dug deeper into 600 rejected blueprints to find out why humans said no. They found four main reasons, ranked from most common to least common:
Reason #1: The "Ghost Town" (38%)
- What happened: The robot submitted a blueprint, and no human ever looked at it. The humans were too busy, or the robot got ignored, and the request just sat there until it was automatically closed.
- Analogy: The robot left a note on the architect's desk, but the architect was on vacation, and the note eventually got thrown in the trash.
Reason #2: The "Double Booking" (31%)
- What happened: The robot tried to fix a problem that someone else had already fixed in a different blueprint.
- Analogy: The robot submitted a plan to build a new bridge, not realizing another team had already started building that exact bridge yesterday.
Reason #3: The "Broken Blueprint" (22%)
- What happened: The code itself was broken, failed the tests, or didn't actually solve the problem.
- Analogy: The robot built a door, but the door doesn't open, or it's made of glass that shatters when you touch it.
Reason #4: The "Wrong Instructions" (2%)
- What happened: The robot ignored what the human asked for or violated legal rules (like copyright).
- Analogy: The human asked the robot to "paint the wall blue," but the robot painted it red and used a paint that the building's insurance doesn't allow.
The Big Takeaway
The paper concludes that while these AI robots are getting better at writing code, they are still bad at reading the room.
They don't know when to stop (they make changes too big), they don't know what's already been done (they duplicate work), and they struggle to follow complex human instructions. For these robots to succeed in the future, they need to learn to be smaller, more focused, and better at checking if their work is actually needed before they ask humans for permission to build.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.