🤖 AI

When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests

This empirical study reveals that the successful integration of agent-authored pull requests depends less on iteration intensity and more on aligning with human review workflows, particularly through strong reviewer engagement and adherence to coordination norms like avoiding force pushes.

Original authors: Costain Nachuma, Minhaz Zibran

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Costain Nachuma, Minhaz Zibran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling construction site where, for the first time, a team of highly skilled robotic architects has been hired to help build a skyscraper. These robots don't just lay bricks; they design entire new wings, fix broken elevators, and even paint the walls. They submit their work to the human foreman (the code reviewer) for approval before it gets added to the building.

This paper is essentially a report card on how well these robotic architects are working with their human foremen. The researchers wanted to know: Do the humans accept the robots' work? How long does it take? And what makes a robot's proposal get approved or rejected?

Here is the breakdown of their findings, using simple analogies:

1. The Big Picture: Robots Are Getting the Job Done (Mostly)

The researchers looked at nearly 34,000 proposals (called "Pull Requests") submitted by AI agents.

The Result: About 71% of the time, the human foremen said, "Great job, let's build this!" (Merged).
The Catch: About 22% of the time, the foremen said, "No, don't build this," and closed the door (Closed without merging).
The Wait: Some proposals were left sitting on the desk, undecided (Open).

The Analogy: Imagine a restaurant kitchen. The robots are the new line cooks. Most of the time, the Head Chef tastes the dish and says, "Perfect, serve it!" But sometimes, the dish is too salty, or the presentation is wrong, and the Chef sends it back to the trash. Interestingly, some robot cooks are much better at this than others. One robot (OpenAI_Codex) was a star chef with an 82% approval rate, while another (Copilot) struggled, getting approved less than half the time.

2. What Makes a Robot's Proposal Get Approved? (The "Secret Sauce")

The researchers dug deep to find out why some proposals succeeded and others failed. They found that it wasn't just about how good the code was. It was about how the robot behaved during the review process.

Here are the main factors, translated into everyday terms:

A. The "Human Touch" is Everything (Reviewer Engagement)

The Finding: The single biggest predictor of success was whether a human actually looked at the proposal and gave feedback.
The Analogy: Think of a robot submitting a design for a new park. If the robot just drops the blueprint on the desk and walks away, the project stalls. But if the robot waits, listens to the human's comments ("Maybe move the fountain here?"), and then immediately moves the fountain, the project gets approved.
Key Takeaway: Robots that engage in a back-and-forth conversation with humans are much more likely to succeed.

B. Don't Rewind the Tape (Coordination Stability)

The Finding: If a robot changes its proposal after a human has already started reviewing it (a "force push"), it lowers the chances of approval.
The Analogy: Imagine you are reading a book someone wrote. Suddenly, they rip out the first three chapters and replace them with new ones without telling you. You are now confused, have to re-read everything, and you get annoyed.
Key Takeaway: Robots need to be stable. If they keep rewriting their work while humans are trying to review it, it disrupts the flow and makes humans less likely to say "yes."

C. Keep It Simple (Change Size)

The Finding: The bigger the change the robot tries to make, the harder it is to get approved.
The Analogy: If a robot says, "I'm going to rebuild the entire foundation of the house," the human foreman gets nervous and says, "Whoa, that's too much risk. Let's start with a new door." Small, manageable changes are easier to trust and approve.

D. More Work ≠ Better Work (Iteration Intensity)

The Finding: Just because a robot tries many times to fix a problem (lots of revisions) doesn't mean it will get approved.
The Analogy: Imagine a student who keeps rewriting their essay 10 times. If they just keep changing random words without listening to the teacher's specific advice, the teacher won't give them an A. The quality of the conversation matters more than the quantity of attempts.

3. The "Why" Behind the Rejections

The researchers also looked at the rejected proposals to see what went wrong. They found three main reasons robots failed:

The "Design Clash": The robot wanted to build a modern glass tower, but the human wanted a brick Victorian house. They fundamentally disagreed on the vision.
The "Broken Promise": The robot promised to fix a leak, but the fix was incomplete or broke something else.
The "Lost Connection": The robot and human stopped talking. The robot didn't respond to the human's feedback, or the human stopped looking at the proposal.

The Bottom Line

This study teaches us that AI isn't just a code generator; it's a teammate.

For AI agents to be successful in the real world, they can't just spit out perfect code and walk away. They need to learn social skills:

Listen to feedback.
Don't change the plan while the boss is looking at it.
Keep changes small and manageable.
Align with the team's existing rules and style.

If AI agents learn to play nice with human review processes, they will become the ultimate teammates. If they don't, they will just be annoying interns who keep rewriting the blueprint while the foreman tries to read it.

1. Problem Statement

Autonomous coding agents (e.g., Copilot, Devin, Cursor) are increasingly submitting pull requests (PRs) to GitHub to fix bugs, add features, and extend tests. While prior studies have documented the volume and aggregate merge rates of these agent-authored PRs, there is a critical gap in understanding how these contributions integrate into human-driven review workflows.

The core problem is that adoption does not equate to effective collaboration. It remains unclear whether autonomous agents can function as effective socio-technical collaborators or if they merely produce "correct" patches that fail due to misalignment with human review norms, coordination costs, and iteration expectations. The authors seek to determine which specific collaboration signals during the review process predict successful integration.

2. Methodology

The study employs a mixed-methods approach combining large-scale quantitative analysis with qualitative case studies.

Dataset

Source: The AIDev dataset (Zenodo version 3), accessed in November 2025.
Scope: A curated subset of popular repositories (≥100 GitHub stars).
Sample Size: 33,596 agent-authored PRs from 1,797 distinct developers across 2,807 repositories, involving five major agents (OpenAI_Codex, Copilot, Devin, Cursor, Claude_Code).
Definition: A PR is "agent-authored" if it has a non-null agent label in the pr_task_type table.

Quantitative Analysis (RQ1 & RQ2)

RQ1 (Integration & Resolution): Analyzed outcome distributions (merged, closed-unmerged, open) and resolution latency (time-to-decision) using Wilson score confidence intervals.
RQ2 (Collaboration Signals): Modeled PR integration as a binary outcome (merged vs. closed-unmerged) using logistic regression with repository-clustered standard errors.
- Predictors:
  - Iteration Intensity: Commit count (bucketed or log-transformed).
  - Change Magnitude: Log-transformed lines of code changed ( $\Delta$ LOC) and number of files modified.
  - Coordination Stability: Binary indicator for force pushes.
  - Testing Behavior: Binary indicator for added tests.
  - Reviewer Engagement: Binary indicator for receiving a review and log-transformed time-to-first-review.
- Controls: Agent identity (using Claude Code as reference) and repository effects.

Qualitative Analysis

Sample: A purposive and random sample of 60 agent-authored PRs (30 merged, 30 not merged).
Method: Manual coding using a predefined codebook derived from prior PR literature to identify the "primary driver" of the outcome (e.g., actionable loops, design disagreements, policy issues).

3. Key Contributions

Empirical Baseline: Establishes the first detailed baseline for agent-authored PR integration rates and resolution speeds across different AI agents.
Signal Identification: Identifies specific review-time collaboration signals (beyond code quality) that correlate with successful integration, shifting the focus from "can the agent write code?" to "can the agent collaborate?"
Mechanism Explanation: Provides a qualitative explanation for why certain signals matter, distinguishing between "actionable convergence" and "coordination disruption."
Replication Package: The authors released a complete replication package (scripts and configuration) to ensure reproducibility.

4. Key Results

RQ1: Integration Outcomes and Latency

Overall Success: 71.5% of agent-authored PRs were merged, 21.6% closed without merging, and 6.9% remained open.
Agent Heterogeneity: Success rates vary significantly by agent.
- OpenAI_Codex: Highest merge rate (82.6%) and fastest decision time (median <1 hour).
- Copilot: Lowest merge rate (43.0%) and slower resolution (median 13 hours).
Implication: Integration success is not uniform; it depends heavily on the specific agent's alignment with repository norms.

RQ2: Collaboration Signals

Reviewer Engagement is Dominant: Receiving at least one review is the strongest predictor of merging. Agents that attract reviewer attention are significantly more likely to be integrated.
Coordination Disruption is Penalized: Force pushes during review are consistently associated with a lower likelihood of merging. They disrupt shared understanding and increase coordination costs.
Change Size Matters: Larger changes (more $\Delta$ LOC and files) are less likely to be merged, indicating reviewers apply similar risk heuristics to agents as they do to humans.
Iteration Intensity is Secondary: Simply increasing the number of commits or adding tests does not independently increase merge likelihood once reviewer engagement and coordination stability are accounted for.
Qualitative Findings:
- Success Pattern: "Actionable review loops" where agents respond to specific feedback with targeted revisions.
- Failure Patterns: Design disagreements, incomplete solutions, and coordination breaks (e.g., force pushes).
- Key Insight: Iteration without convergence (addressing specific reviewer concerns) is insufficient.

5. Significance and Implications

Shift in Perspective: The study argues that for AI agents to be effective teammates, they must align with socio-technical norms (review workflows, coordination stability) rather than just generating technically correct code.
Design Guidelines for Agents: Future autonomous agents should be designed to:
- Prioritize small, incremental changes.
- Avoid force pushes during active review.
- Explicitly respond to reviewer feedback to close the "actionable loop."
Human-in-the-Loop: Human reviewers play a critical gatekeeping role; their engagement is the primary catalyst for integration. Agents that fail to engage reviewers or disrupt the workflow are likely to be rejected regardless of code quality.
Future Research: Highlights the need to study how agents can better interpret social cues and project-specific policies to reduce coordination costs.

In conclusion, the paper demonstrates that the integration of AI-authored code is a collaborative process mediated by human review. Success depends less on the volume of iterations and more on the agent's ability to navigate review dynamics, maintain coordination stability, and converge on reviewer expectations.