Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study
This empirical study of over 11,000 agentic pull requests reveals that relying solely on merge or rejection outcomes to evaluate AI coding agents is misleading, as a significant portion of rejections stem from workflow constraints rather than agent errors and many merges require substantial human intervention, thereby necessitating interaction-aware evaluation metrics grounded in review behavior.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a bustling, high-tech kitchen where robots (AI coding agents) are constantly trying to cook up new dishes (code changes) and serve them to the head chef (the human reviewer) for approval. The goal is to get these dishes added to the restaurant's menu (merged into the software).
For a long time, people have judged how good these robot chefs are simply by looking at the final result: Did the dish get served (Merged) or sent back to the trash (Rejected)?
This new study argues that looking at the final result is like judging a chef only by whether the plate left the kitchen, without asking why. The researchers found that the "Yes/No" label tells a very incomplete story.
Here is what they discovered, broken down into simple analogies:
1. The "Trash" Isn't Always the Chef's Fault
When a robot's dish is rejected, we often assume the robot cooked something inedible (a coding error). However, the study found that only about 36% of rejections were actually because the robot messed up the recipe.
The other two-thirds were rejected for reasons that had nothing to do with the robot's cooking skills:
- The "Wrong Night" Problem (31%): The dish was perfect, but the kitchen was already closed, or the chef had already ordered a different dish for that night. In tech terms, the code was fine, but the project workflow didn't need it, or it was a duplicate.
- The "Silent Treatment" (33%): The dish was sent back, but the chef didn't say a word. There was no note on the plate explaining why. It's impossible to tell if the robot failed or if the chef just ignored it.
The Takeaway: If you only count rejections as "failures," you are unfairly blaming the robot for things that were actually just bad timing or silence.
2. The "Served" Dish Isn't Always the Robot's Alone
When a dish gets served (merged), we assume the robot cooked it perfectly from start to finish. But the study found that about 15% of these "successes" actually required the human chef to step in and fix things before the dish could go out.
- The "Taste-Test" (Feedback Loops): The robot cooked the main course, but the chef tasted it, said, "Needs more salt," and the robot fixed it.
- The "Plating" (Human Intervention): The robot cooked the food, but the chef had to rearrange the garnish or fix the presentation before serving it.
In fact, for some robots (like Copilot and Devin), the human chef was involved in the process much more often than for others (like Codex and Cursor), who seemed to get their dishes served with almost no interaction.
The Takeaway: A "Merged" label doesn't always mean the robot did the whole job alone. Sometimes, the human did the heavy lifting at the very end to make it work.
3. Different Robots, Different Kitchens
The study noticed that different robots behaved differently depending on which "kitchen" (software project) they were in.
- Some robots were sent to kitchens with strict rules and busy chefs who constantly gave feedback. These robots had more rejections and more human help.
- Other robots were sent to kitchens where the chefs were more hands-off, leading to more "silent" approvals or rejections.
The Big Conclusion
The paper concludes that we cannot judge a robot chef's skill just by looking at the "Served" or "Rejected" stamp on the ticket.
- Rejections are often just workflow issues or silence, not bad cooking.
- Merges often involve human chefs fixing the plate before serving.
To truly understand how good these AI agents are, we need to look at the conversation between the robot and the human—the comments, the fixes, and the reasons behind the decisions—rather than just the final outcome. If we don't, we are judging the chef based on a menu that doesn't tell the whole story.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.