Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agents
This paper analyzes 654 rejected pull requests across five coding agents and a human baseline to identify unique and agent-specific rejection patterns, while proposing heuristics to address the challenge of missing reviewer feedback in 67.9% of cases.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine software development as a massive, bustling construction site. Usually, when a worker (a human developer) finishes a new room or fixes a leak, they hand in a "Change Request" (a Pull Request) to the site manager. The manager looks it over and says, "Great, let's build this," or "No, this won't work."
Recently, a new type of worker has arrived: AI Agents. These are robots that can plan, build, and hand in their own Change Requests without a human holding their hand. This paper investigates what happens when these AI workers submit their work compared to human workers.
Here is the breakdown of the study, using simple analogies:
1. The Big Problem: The "Silent Rejection"
The researchers looked at 654 rejected Change Requests from five different AI robots (like Devin, Claude Code, and GitHub Copilot) and compared them to requests made by humans.
The most shocking finding? Two out of every three rejections were silent.
- The Analogy: Imagine you submit a drawing to a contest, and it gets thrown in the trash. You ask, "Why?" and the judge just shrugs and walks away.
- The Reality: 67.9% of the rejected AI requests had no explanation from the reviewer. The request was just closed. This makes it very hard for the researchers to know why the AI failed, because the "judge" didn't write down the reason.
2. The "AI-Only" Mistakes
When the researchers did find reasons for rejection, they discovered that AI robots make mistakes that humans almost never make. They found seven specific rejection reasons that only happened with AI:
- The "Too Big" Box: Humans sometimes submit big projects, but AI robots tend to submit massive, overwhelming changes all at once. It's like an AI trying to build an entire skyscraper in a single day. The managers rejected these because they were too huge to review properly.
- The "Distrust" Badge: Some managers simply didn't trust the code because it was made by a robot. They rejected it just because it was AI-generated, fearing it might be unreliable or "hallucinated" (made up).
- The "Slop" Label: In one case, a project owner rejected an AI request and labeled it "SLOP" (a slang term for low-quality, mass-produced content), essentially saying, "This is junk generated by a machine."
- The "Experiment" Fail: Some AI requests were submitted just to see if the robot could do it, not because anyone actually wanted the code. These were rejected because they weren't meant for real use.
3. The "Robot-Specific" Quirks
Different AI robots have different personalities and settings, leading to unique rejection patterns:
- The "Devin" Effect: One specific robot, named Devin, has a setting that automatically closes its own requests if they sit idle for too long. The study found that many of Devin's rejections weren't because the code was bad, but because the robot itself said, "I'm bored, I'm closing this," after 7 days of silence.
- The "Context" Trap: Some AI robots got rejected because they couldn't see "private" files or data they needed to finish the job. It's like a robot trying to fix a lock but being locked out of the room where the key is kept.
4. The Solution: A "Trash Filter"
Since so many rejections had no notes (the "Silent Rejection" problem), the researchers realized they needed a way to clean up the data before studying it. They created a simple filter (a set of rules) to guess which rejections were truly "silent" and which ones might have hidden clues.
- The Filter Rules: They looked for requests that were:
- Closed by the person who wrote them (Self-closed).
- Closed very quickly (e.g., within 7 days).
- Closed without anyone leaving a comment.
- The Result: By using these rules, they could filter out the "noise" (the silent, unhelpful rejections) and focus on the ones where the managers actually explained why they said "No." This helps future researchers get a clearer picture of what's going wrong.
Summary
The paper concludes that while AI agents are building code faster than ever, they are getting rejected more often than humans. This isn't just because their code is buggy; it's because they:
- Submit things that are too big or complex.
- Trigger a lack of trust from human managers.
- Have unique "robot behaviors" (like auto-closing) that humans don't have.
- Often get rejected without anyone explaining why, making it hard to learn from mistakes.
The study provides a "filter" to help clean up this data so we can better understand how to make these AI workers more reliable in the future.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.