On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
This empirical study of 567 GitHub pull requests generated by the Claude Code agent across 157 open-source projects reveals that while 83.8% of agent-assisted contributions are accepted—often without modification—they still frequently require human oversight for bug fixes, documentation, and adherence to project standards.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine software development as a massive, bustling construction site where thousands of teams are building digital skyscrapers (open-source projects). For years, the workers (developers) have been doing all the heavy lifting, drafting blueprints, and laying bricks themselves.
Recently, a new type of super-intelligent robot assistant called "Agentic Coding" (specifically a tool named Claude Code) has arrived on the site. Unlike a simple calculator that just answers questions, this robot can grab a wrench, look at the blueprints, fix a leaky pipe, write a new instruction manual, and even file the paperwork to get the work approved—all by itself.
This paper is a report card on how well these robots are actually doing their jobs in the real world. The researchers looked at 567 construction projects (Pull Requests) where this robot did the work, compared them to projects done by humans, and asked four big questions.
Here is what they found, explained with simple analogies:
1. What kind of work do the robots do? (RQ1)
The Finding: Humans and robots both fix broken things (bugs) and build new rooms (features). However, the robots have a special talent for renovations and cleaning.
- The Analogy: If a human worker is asked to "fix the house," they might fix the roof or build a new garage. If you ask the robot to "fix the house," it loves to reorganize the furniture (refactoring), write better labels on the cabinets (documentation), and add more security cameras (testing).
- The Result: Robots are surprisingly good at the tedious, rule-based cleanup work that humans often find boring. They also tend to write much longer, more detailed "work orders" (PR descriptions) explaining exactly what they did.
2. Do the bosses (maintainers) accept the robots' work? (RQ2)
The Finding: Yes, mostly! But not quite as often as they accept human work.
- The Stats: About 84% of the robot's work gets approved and merged. For comparison, about 91% of human work gets approved.
- Why do they get rejected? It's rarely because the robot built a "bad wall." Usually, the work gets rejected because:
- Timing: Another team member already fixed the problem in a different way.
- Size: The robot tried to fix the whole house in one go, making the blueprint too huge to review.
- Obsolescence: The project changed direction while the robot was working.
- The Takeaway: The robots are generally trustworthy, but they sometimes miss the "big picture" context of what the team actually wants right now.
3. Do the robots need help finishing the job? (RQ3)
The Finding: Surprisingly, robots and humans need about the same amount of "polishing" before their work is perfect.
- The Analogy: Imagine you order a custom cake. Sometimes the baker (robot) gets it 100% right the first time. Sometimes they get it 90% right and you just need to tweak the frosting.
- The Stats: About 55% of robot cakes are perfect as-is. This is almost the same as human bakers (58.5%).
- The Effort: When the cake does need tweaking, the amount of work required to fix a robot's cake is statistically the same as fixing a human's cake. The robots aren't making more mistakes that require more work; they just make different kinds of small errors.
4. What exactly do humans have to fix? (RQ4)
The Finding: When humans do have to step in and fix the robot's work, they are mostly fixing safety checks, instructions, and style.
- Bug Fixes (48%): Sometimes the robot tries to be too optimistic. It assumes a door will never break, so it doesn't build a backup plan. Humans have to add that backup plan.
- Documentation (29%): The robot might build a great new room but forget to update the map of the house. Humans have to update the "README" (the instruction manual).
- Refactoring (27%): The robot might build a wall that works, but it's in a weird spot that makes the rest of the house hard to navigate. Humans have to move it to fit the house's style.
- Style (23%): The robot might use the wrong color paint or the wrong font. Humans have to make it look like the rest of the building.
The Big Picture
Think of Agentic Coding as a very talented, fast, and eager apprentice.
- Pros: It can do a huge amount of the heavy lifting, especially the boring cleanup and testing tasks. It produces work that is often good enough to use right away.
- Cons: It sometimes misses the "vibe" of the project, writes instructions that are too long or too short, and occasionally needs a human supervisor to double-check the safety and style.
The Conclusion: The robot is not replacing the human foreman yet. Instead, it's a powerful tool that does the grunt work, allowing the human to step in, review the plans, and make the final call to ensure everything is safe, stylish, and fits the project perfectly. The paper suggests that while the robot is a great starting point, human oversight is still essential to turn a "good draft" into a "finished masterpiece."
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.