Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study

Technical Summary: Why Are AI Agent–Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study

Problem Statement
While autonomous coding agents (e.g., OpenAI Codex, Devin, GitHub Copilot) are increasingly integrated into real-world software development workflows to generate fix-related pull requests (PRs), their practical effectiveness is contingent upon acceptance by project maintainers. Existing research often focuses on benchmark-based task success or developer productivity, abstracting away the downstream PR review and integration process. Consequently, there is limited empirical understanding of why AI-generated fix-related PRs are closed without being merged in real-world settings. These unmerged PRs represent wasted computational effort and increased review overhead, exposing potential limitations in agent reasoning, validation strategies, and human-AI collaboration.

Methodology
The authors conducted a mixed-methods empirical study using the AIDEV-POP dataset, a large-scale collection of AI-authored PRs mined from GitHub repositories released in November 2025. The study focused on 8,106 fix-related PRs authored by five widely used AI coding agents: OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code.

The methodology involved two primary phases:

1. Quantitative Analysis (RQ1): The authors classified the integration outcomes of all 8,106 PRs into three states: merged, closed without merging, or open. For merged PRs, they calculated the "time-to-merge" (the duration between PR creation and closing) to characterize integration latency across different agents.
2. Qualitative Analysis (RQ2): To understand the reasons for non-integration, the authors performed a manual analysis of a statistically significant sample of 326 closed-but-unmerged PRs. Two authors independently labeled these PRs using an open coding approach after a calibration phase, spending approximately 100 person-hours. They achieved a Cohen's kappa of 0.82, indicating substantial inter-rater agreement. This process resulted in a structured catalog of failure reasons derived inductively from the data.

Key Contributions

• Large-Scale Empirical Data: The study provides the first large-scale analysis of integration outcomes for fix-related PRs across five major AI agents, moving beyond synthetic benchmarks to real-world repository data.
• Structured Failure Catalog: The authors constructed a catalog of 12 distinct failure reasons explaining why AI-authored fix PRs are closed without merging. These range from technical issues (e.g., test failures) to process-related dynamics (e.g., inactivity, redundancy).
• Agent-Specific Performance Profiles: The study reveals significant disparities in merge rates and failure modes across different agents, challenging the notion of a uniform "AI agent" performance.

Results

• Integration Outcomes: Overall, 65.0% of fix-related PRs were successfully merged, while 26.1% were closed without merging, and 8.9% remained open. However, success rates varied drastically by agent:

  • OpenAI Codex: Highest merge rate at 81.6%.
  • Devin: Lowest merge rate at 42.9%, with 54.0% of its PRs closed without merging.
  • GitHub Copilot: Merge rate of 42.4%.

• Latency: Merged PRs showed substantial variability in submission-to-merge time. OpenAI Codex exhibited consistently shorter and more concentrated merge times, whereas Copilot and Devin showed broader distributions with heavier upper tails, indicating higher integration latency and coordination overhead.

• Failure Causes: The manual analysis of 326 unmerged PRs identified the following primary reasons for closure:

  1. Resolved by Another PR (RAPR): 22.1% (The issue was fixed by a different PR).
  2. Test Case Failures (TCF): 18.1% (The fix failed existing tests or introduced new ones).
  3. Incorrect or Incomplete Fixes (IIF): 15.3% (The fix did not resolve the root cause or introduced errors).
  4. Closed Due to Inactivity (CDI): 9.2%.
  5. Low Priority/Obsolete (LPO): 8.0%.
  6. Incomplete Review Process (IRP): 5.5%.
  7. Fix Rejected by Maintainers (FRM): 4.9%.
  8. No Review Conducted (NRC): 4.6%.
  9. Closed Without Explicit Rationale (CWR): 4.0%.
  10. Deployment Failures (DF): 3.1%.
  11. Other Reasons (OTH): 3.1%.
  12. Build Failures (BF): 2.1%.

• Agent-Specific Failure Patterns:

  • OpenAI Codex failures were dominated by Test Case Failures (54.9%), suggesting issues with validation readiness rather than review interaction.
  • Devin failures were primarily driven by Inactivity (54.0%), indicating challenges in sustaining engagement during review cycles.
  • GitHub Copilot failures were largely due to issues being Resolved by Another PR (31.8%), pointing to redundancy and timing mismatches.

Significance and Claims
The paper claims that while AI agents can frequently produce viable fixes (65% merge rate), a substantial fraction of their contributions fail to integrate due to a combination of technical shortcomings and workflow dynamics. The authors emphasize that syntactically plausible patches are often insufficient for successful integration.

Key takeaways highlighted by the authors include:

• Validation vs. Compilation: Failures are more frequently associated with test case failures and incorrect logic than with build or deployment failures, indicating that current agents struggle with validation readiness rather than basic code generation.
• Workflow Dynamics: Process-related factors, such as issues being resolved by other PRs or inactivity, play a non-trivial role in integration outcomes, suggesting that agent success depends not just on code quality but on alignment with project timing and review workflows.
• Future Directions: The findings expose key limitations in current agents and highlight the need for test- and context-aware agent designs that better support automated validation, review interactions, and alignment with project-specific expectations. The study serves as a foundation for improving human-AI collaboration in software maintenance.