💻 computer science

Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance

This empirical study analyzes 7,156 pull requests to reveal that while Devin shows a unique positive temporal trend in acceptance, task type is the dominant factor influencing success rates, with OpenAI Codex demonstrating the most consistent high performance across diverse categories despite no single agent excelling in all task types.

Original authors: Giovanni Pinna, Jingzhi Gong, David Williams, Federica Sarro

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Giovanni Pinna, Jingzhi Gong, David Williams, Federica Sarro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a busy construction site. Instead of human workers, you have five different AI robots (OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code) that can write code, fix bugs, and write documentation. Your goal is to see which robot is the best at getting its work approved by the site inspectors (the "Pull Request Acceptance Rate").

This paper is like a detailed report card comparing these five robots over several months. Here is what they found, explained simply:

1. The "Job Type" Matters More Than the Robot

The biggest surprise wasn't which robot was the fastest, but what kind of job they were doing.

The Analogy: Imagine asking a robot to "write a poem" versus "build a bridge." Writing a poem is usually easier to get approved than building a bridge.
The Finding: The paper found that the type of task is the most important factor.
- When the robots were asked to do documentation (like writing manuals or comments), they got approved 82% of the time.
- When they were asked to build new features (like adding a new room to a house), they only got approved 66% of the time.
The Lesson: If you judge a robot just by its overall score, you might be fooled. A robot that mostly does easy "poem-writing" tasks will look like a superstar, even if it's terrible at "bridge-building."

2. No Single Robot Wins Every Category

There is no "best robot" for everything. Each one has a specific superpower.

OpenAI Codex: This is the reliable all-rounder. It performed consistently well across almost every type of job, never dropping below a decent score.
Claude Code: This is the Documentation Specialist. It was the best at writing manuals and creating new features, but we have to be careful because it didn't do many other tasks in this study.
Cursor: This is the Bug Fixer. When it came to fixing broken things, it was the top performer.
Devin: This robot started slow but got better over time. It was the only one that showed a clear trend of improvement week after week, climbing from about 60% approval to 80%.

3. Time Changes Things (But Not for Everyone)

The researchers watched these robots for months to see if they learned or improved.

The Analogy: Think of it like a student taking a test every week.
- Devin is like the student who studies hard and keeps getting higher scores every week.
- The others are like students who are already smart; they start high and stay high, but they don't show much improvement over time. They just stay consistent.

4. Why the "Global Score" is Misleading

The paper warns against looking at a single "average score" for a robot.

The Analogy: Imagine two chefs. Chef A only cooks simple salads (which everyone loves). Chef B only cooks complex, spicy stews (which are harder to get right). If you just look at the "average customer rating," Chef A might look better. But that doesn't mean Chef A is a better chef; it just means they were given easier ingredients.
The Finding: The researchers had to separate the robots by the type of job they did to get a fair comparison. Once they did that, they found that the "best" robot changed depending on whether the job was fixing a bug, writing a test, or updating a document.

Summary

If you want to hire an AI coding assistant, don't just ask, "Who is the best?"

If you need bug fixes, look at Cursor or OpenAI Codex.
If you need new features or documentation, Claude Code or OpenAI Codex might be your best bet.
If you want a robot that seems to be learning and improving over time, Devin is the one to watch.

The main takeaway is that context is king. You can't judge a tool without knowing exactly what job it's being asked to do.

Technical Summary: Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance

Problem Statement

The rapid integration of AI-powered coding assistants into software engineering workflows has shifted the paradigm from simple code completion to autonomous agents capable of generating functions, fixing bugs, and creating pull requests (PRs) independently. However, systematic comparisons of these agents' effectiveness remain limited. Existing evaluations often rely on global performance metrics that fail to distinguish between an agent's inherent capability and the specific distribution of tasks it is assigned. For instance, an agent handling primarily documentation tasks (which typically have higher acceptance rates) may appear superior to one handling complex feature implementations, even if the latter possesses stronger underlying capabilities. This confounding effect, combined with a lack of longitudinal data on how agent performance evolves over time, creates a gap in understanding the true efficacy of different AI coding agents in real-world software development.

Methodology

This study presents an empirical analysis of 7,156 closed pull requests from the AIDev dataset, specifically the AIDev-POP subset (repositories with 100+ stars). The dataset covers five popular AI coding agents: OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code.

Data Filtering and Preprocessing:
The authors filtered the raw dataset of 33,596 PRs to retain only:

Closed PRs.
PRs from repositories with permissive licenses (MIT or Apache-2.0).
PRs that received at least one review or comment from a human other than the creator, ensuring meaningful evaluation.

Metrics and Statistical Approach:

Success Metric: Defined as the acceptance rate (proportion of merged PRs).
Temporal Analysis (RQ1): Linear regression models ( $y = \beta_0 + \beta_1 \cdot t + \varepsilon$ ) and LOESS smoothing were applied to acceptance rates over time to identify evolution patterns.
Factor Analysis (RQ2): The study examined correlations between acceptance rates and task types, as well as review frequency.
Comparative Analysis (RQ3): To address workload heterogeneity, the authors performed task-stratified pairwise comparisons. Using Pearson's Chi-square tests of independence (and Fisher's exact test for sparse data), they compared agents within specific task categories (e.g., features, fixes, documentation).
- Significance Control: A Bonferroni correction was applied across 64 tests ( $\alpha \approx 0.00078$ ) to control for multiple comparisons.
- Effect Size: The phi coefficient ( $\phi$ ) was reported to measure effect magnitude.

The study explicitly acknowledges its observational nature, noting that while it documents patterns, it does not claim causality due to potential confounders like repository concentration, user behavior changes, and model updates.

Key Contributions

Temporal Evolution Analysis: The study reveals heterogeneous evolution patterns across agents. Devin is the only agent demonstrating a consistent positive trend in acceptance rate over a 32-week period, whereas others remain largely stable.
Task-Stratified Methodology: The paper introduces a methodology that controls for workload heterogeneity by evaluating agents within specific task categories, preventing misleading conclusions drawn from global averages.
Dominance of Task Type: Empirical evidence demonstrates that task type is a dominant factor influencing acceptance rates, creating a 29 percentage point gap between the highest (chore: 84.0%) and lowest (performance: 55.4%) performing categories. This gap often exceeds the variance observed between different agents within the same category.
Agent-Specific Strengths: The analysis identifies that no single agent outperforms all others across all task types. Performance is highly context-dependent.

Key Results

Temporal Trends (RQ1):

Devin showed a consistent improvement of +0.77% per week ( $R^2 = 0.34$ ), rising from ~60% to ~80% acceptance.
OpenAI Codex and GitHub Copilot exhibited plateau behavior, maintaining consistently high acceptance rates from their first observed week.

Performance Factors (RQ2):

Task Type: Documentation tasks achieved an 82.1% acceptance rate, compared to 66.1% for new features—a 16 percentage point gap.
Review Frequency: GitHub Copilot PRs received significantly more reviews per PR (4.94) compared to OpenAI Codex (1.39), correlating with lower acceptance rates, though the authors note this may reflect task complexity or repository policies rather than a direct causal link.

Agent Comparisons (RQ3):

OpenAI Codex demonstrated consistently high acceptance rates across all nine analyzed task categories (ranging from 59.6% to 88.6%), leading in fix (83.0%) and refactor (74.3%) tasks.
Claude Code led in documentation (92.3%) and features (72.6%), though with a smaller sample size.
Cursor excelled in test tasks (77.8%) and fix tasks (80.4%).
Statistical Significance: Of 64 stratified comparisons, only 6 were statistically significant after Bonferroni correction. All significant differences involved fix or feat tasks. Notably, Devin underperformed relative to Codex, Copilot, and Cursor specifically in fix tasks.

Significance and Implications

The paper argues that the "best" AI coding agent is not a universal concept but is contingent upon the specific task at hand. The significant variance in acceptance rates based on task type suggests that global performance metrics can be misleading.

For Practitioners: The choice of agent should be driven by the specific workflow. For bug fixes, Cursor and OpenAI Codex show strong performance, whereas Devin shows relative weakness in this domain. Documentation tasks show minimal differentiation between agents. The authors suggest practitioners may benefit from adopting a combination of agents tailored to different workflows.
For Researchers: The study advocates for task stratification as a standard practice in future evaluations. Global metrics should be accompanied by task distributions, and comparisons should be stratified to control for confounding variables. The authors also highlight the need for complementary metrics (e.g., static analysis, technical debt) beyond simple acceptance rates to capture long-term code quality.

The study concludes that while AI agents are transforming software development, their evaluation requires nuanced, context-aware methodologies that account for both task heterogeneity and temporal dynamics.