🤖 AI

Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset

This paper analyzes the AIDev dataset to identify 14 specific reasons why nearly half of AI-generated pull requests are rejected, categorizing these failure modes to provide actionable guidance for improving agent performance, task prioritization, and integration into software development workflows.

Original authors: Mahmoud Abujadallah, Ali Arabat, Mohammed Sayagh

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mahmoud Abujadallah, Ali Arabat, Mohammed Sayagh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a team of super-fast, hyper-enthusiastic robot interns to fix bugs in your software. You give them a problem, and they immediately start typing code, building a "Pull Request" (which is like a proposal to change the code).

This paper is basically a report card on what happens when these robot interns (specifically Copilot, Devin, Cursor, and Claude) try to fix things. The researchers found a startling statistic: nearly half of the time (46.41%), the human boss throws the robot's work in the trash.

Here is a breakdown of why this happens, using simple analogies:

The Big Problem: The "Wasted Effort" Bill

Every time a robot generates a fix that gets rejected, it's a double loss. First, the robot wastes its own "brain power" (computing resources and tokens). Second, and more importantly, a human has to stop what they are doing, read the robot's messy work, realize it's wrong, and write a comment explaining why. That is wasted human time.

The researchers looked at 306 of these rejected proposals to figure out exactly why the humans said "No." They found four main reasons for the rejection:

1. The "Wrong Tool for the Job" (Implementation Issues)

Sometimes the robot tries to fix the problem, but it uses the wrong approach.

The Analogy: Imagine your car won't start. You ask a mechanic to fix it. They show up and try to fix the radio instead of the engine, or they try to fix the engine with a hammer instead of a wrench.
What happened: The robots often misunderstood the instructions, fixed the wrong thing, or proposed a solution that was technically impossible or incomplete.

2. The "Broken Test" (Technical Issues)

In software, before a fix is accepted, it has to pass a series of automated tests (like a safety inspection).

The Analogy: The robot builds a new bridge, but when the inspector drives a truck over it, the bridge collapses. The robot didn't check if its own bridge could hold weight.
What happened: The code the robots wrote often failed the automated "safety tests" (CI pipelines) or broke other parts of the software that were already working.

3. The "Ghost Intern" (Provider Issues)

Sometimes the robot just stops working or gets cut off.

The Analogy: You ask the intern to write a report, but halfway through, the intern walks out of the building, or the internet connection cuts out, leaving you with a blank page.
What happened: The AI service itself crashed, the robot got "rate-limited" (ran out of allowed requests), or the session simply died before it could finish the job.

4. The "Useless Fix" (Relevance Issues)

Sometimes the robot is working on a problem that doesn't matter anymore.

The Analogy: You ask the intern to fix a leak in the kitchen. By the time the intern finishes the repair, the kitchen has been remodeled, and the leak is gone, or someone else already fixed it with a better method.
What happened: The issue the robot was fixing was low priority, the problem was already solved by someone else, or the robot sat idle for so long that the project moved on without it.

The "Cost" of the Mistake

The paper also measured how much "mess" the robots made before being rejected.

Code Churn: This is a fancy way of saying "how much code was written and then deleted." The researchers found that rejected fixes involved a median of 81 to 293 lines of code. That's a lot of typing for a mistake!
Comments: Humans had to write an average of 1 to 4.5 comments to explain why the fix was bad. Since almost half of all robot fixes are rejected, that means half of all the human comments written to these robots are just saying, "No, this doesn't work."

The Takeaway: How to Train the Interns Better

The authors suggest that to stop wasting time, humans need to give the robots better instructions before they start working. Specifically:

Give a Map: Tell the robot exactly how to fix the problem and what not to do.
Set Up the Test: Tell the robot how to check its own work to make sure it passes the safety tests before showing it to the boss.
Pick the Right Jobs: Don't ask the robot to fix tiny, unimportant bugs that aren't worth the human review time.

In short: AI agents are powerful, but right now, they are like enthusiastic interns who need very clear, specific instructions to avoid wasting everyone's time and energy.

Technical Summary: Understanding the Rejection of Fixes Generated by Agentic Pull Requests

Problem Statement

AI coding agents (specifically Copilot, Devin, Cursor, and Claude) are increasingly integrated into software development teams to autonomously generate pull requests (PRs) for bug fixes. However, initial exploration of the AIDev dataset reveals a significant inefficiency: 46.41% of the fixes proposed by these agents are rejected by human developers. This high rejection rate represents a substantial waste of resources, including:

Human Effort: Time spent reviewing, verifying, and running tests on code that is ultimately discarded.
Agent Resources: Consumption of tokens, compute power, and limited premium queries for failed attempts.
Trust: High rejection rates may undermine developer trust in AI agents, hindering their adoption.

While prior studies have analyzed rejection reasons, they often aggregate different task types (e.g., refactoring, new features) or focus on a single agent. This paper addresses the gap by exclusively focusing on bug-fixing PRs across four different agents to provide a granular understanding of why AI-generated fixes fail.

Methodology

The study employs a mixed-methods approach using the AIDev dataset, which contains over 33,000 curated PRs from 2,807 open-source projects.

Data Selection:
- The authors filtered for PRs that were (i) closed and not merged, (ii) created or co-authored by Copilot, Devin, Cursor, or Claude, and (iii) explicitly proposed code fixes.
- This resulted in a population of 1,497 rejected AI-agentic fixes.
- A representative random sample of 306 PRs was selected for detailed analysis (95% confidence level, 5% margin of error).
Qualitative Analysis (RQ1):
- Two authors independently labeled the 306 PRs by analyzing discussion threads, comments, CI results, and external artifacts to identify rejection reasons.
- Labels were iteratively refined through discussion, achieving a Cohen's $\kappa$ of 0.605 (substantial agreement).
- A third author resolved 93 conflicting cases to reach a consensus.
- Final labels were grouped into high-level themes.
Quantitative Analysis (RQ2):
- The study measured "wasted effort" using code churn (lines of code changed) and the number of comments exchanged per PR.
- These metrics were compared across the identified rejection categories to quantify the cost of each failure mode.

Key Contributions and Results

1. Taxonomy of Rejection Reasons (RQ1)

The qualitative analysis identified 14 specific reasons for rejection, categorized into four high-level themes:

Relevance of the Fix (Most Common):
- Inactivity (17.3%): PRs closed due to lack of interaction for extended periods.
- Superseded (5.9%): Replaced by other PRs or better alternatives.
- Low Priority (1.0%): Issues deemed non-critical.
- Other: Architectural changes and test-only PRs.
Implementation Issues:
- Incorrect Fix (5.6%): Functionally wrong or incomplete changes.
- Wrong Approach (2.6%): The agent solved the problem via an alternative, undesired implementation.
- Ambiguity/Insufficient (1.4%): Misunderstanding requirements or leaving key functionality incomplete.
- Wrong Repo (0.7%): Submitted to the incorrect repository.
Technical Issues:
- CI Failure (6.9%): The fix failed continuous integration pipelines or automated tests.
- Breaking Change (0.3%): Introduced unintended side effects.
Provider-Related Issues:
- Agent Failure (7.5%): The agent became unreachable, produced erroneous output, or sessions died (e.g., Devin).
- Rate Limits (1.0%): API or resource restrictions prevented task completion.

Note: 49.3% of the sample (151 PRs) had no clear reason for rejection and were labeled "Others."

2. Quantification of Wasted Effort (RQ2)

The study quantified the cost of these rejections:

Code Churn: Rejected PRs exhibit significant code churn, with medians ranging from 81 to 293 lines depending on the category.
- Technical Issues (e.g., CI failures) had the highest median churn (293 lines), suggesting larger changes are more prone to build failures.
- Implementation Issues had a median of 103 lines.
- Provider Issues had a median of 0 lines (as the agent often failed to generate code).
Comments: The median number of comments on rejected PRs ranges from 1 to 4.5. This is comparable to merged PRs (median of 2), indicating that nearly half of all discussion effort in the dataset is spent on code that is eventually discarded.

Significance and Implications

The paper argues that understanding these failure modes is critical for integrating AI agents as efficient teammates. The authors propose four specific implications for practitioners and researchers:

Guidance on Approach: Developers should provide explicit instructions on preferred solutions and constraints on unacceptable approaches (e.g., via .github/copilot-instructions.md) to reduce incorrect implementations and wrong approaches.
Validation Instructions: Agents require clearer guidance on how to validate fixes, including running specific tests and ensuring CI pipelines pass before submission to reduce technical rejections.
Task Prioritization: To minimize waste, AI agents should not be assigned low-priority or irrelevant tasks that result in high code churn and review effort without commensurate value.
Prompt Engineering Research: Future work should focus on developing templates and metrics to assess prompt quality, ensuring agents are guided to produce successful, testable fixes.

The study concludes that while AI agents are powerful, their current failure modes—particularly regarding implementation correctness, CI validation, and task prioritization—result in significant resource waste. Better integration requires human guidance on how to fix, how to validate, and what to prioritize.