💻 computer science

Predicting Acceptance and Review Effort in Human and Agent Pull Requests

This paper demonstrates that machine learning models using only submission-time features can accurately predict the acceptance of both human and agent pull requests but struggle to forecast review effort, suggesting their best use is as advisory tools for triage rather than automated decision-makers.

Original authors: Kartik Ghanshyambhai Pansuriya, Ehsan Ghorbani, Deepak Singh, Eman Abdullah AlOmar

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Kartik Ghanshyambhai Pansuriya, Ehsan Ghorbani, Deepak Singh, Eman Abdullah AlOmar

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling software workshop where code changes are like packages waiting to be shipped. In the old days, only human workers sent these packages. But now, a new crew of AI robots has joined the team, dropping off their own code packages alongside the humans. The managers (called "maintainers") are suddenly overwhelmed. They have a mountain of packages to check, and they need to know two things immediately: Which packages will get approved and shipped? and Which ones will require a huge, exhausting inspection?

This paper is like a detective trying to solve that mystery using only the clues visible on the outside of the package the moment it arrives, before anyone has even opened the box or started talking about it.

The Big Discovery: The "Package Label" Tells a Lot

The researchers found that you can actually make a pretty good guess about whether a package will be accepted just by looking at its label and the box it's in. They built a smart computer system that looked at things like:

How long the description on the label was.
How many files were inside the box.
What time of day it was sent.
The reputation of the workshop it came from.

When they tested this system, the computer got really good at spotting the "approved" packages. In fact, the best models (using a method called Random Forest) got a score of 0.958 out of 1.0 for correctly identifying which packages would be accepted. That's like a detective solving 96 out of 100 cases just by glancing at the address!

However, there's a catch. The paper explicitly warns that this system isn't a magic "yes/no" button. The computer is good at sorting the "likely yes" from the "likely no," but it's not perfect enough to make the final decision on its own. The researchers suggest using it as a helpful assistant to tell managers, "Hey, look at this one first!" rather than an automated robot that says, "Ship it!" without a human ever looking.

The Harder Mystery: How Much Work Will It Be?

The second question was trickier: "How much effort will it take to review this?" The researchers tried to predict two things: how many comments the package would get (the "chatter") and how many hours it would sit in the queue before being shipped (the "wait time").

Here, the results were much more modest. The computer could guess the amount of chatter with an error margin of about 1.00 comment on average, but it only explained 20% of the reasons why some packages got more comments than others. Predicting the wait time was even harder; the computer's guesses were off by an average of 24.00 hours, and it only explained 12% of the variation.

Why was this so hard? The paper suggests that the wait time isn't just about the package itself. It's like waiting for a bus: even if your ticket is perfect, you might still wait a long time if the bus driver is on a break, the traffic is bad, or the bus is full. Similarly, a code package might wait a long time because the reviewers are busy, the automated testing tools are slow, or the team just has a slow workflow. The paper argues that you can't predict these "traffic jams" just by looking at the package label.

Humans vs. Robots: Do They Play by Different Rules?

The team also wondered if the AI robots' packages were harder to predict than the humans'. They found that the AI packages were actually a bit easier to sort because they followed very neat, regular patterns. The human packages were messier and more varied.

Interestingly, the AI packages tended to get a bit more "chatter" (comments) and took a little longer to ship. The paper suggests this might be because the human managers are being extra careful, double-checking the robot's work to make sure it doesn't have any hidden tricks. But despite this extra scrutiny, the computer was still able to predict the outcome for the robots just as well as for the humans.

What the Paper Says "No" To

It's important to know what this study didn't do. The researchers explicitly ruled out using any information that appears after the package is opened. They didn't look at the comments people wrote, the results of the automated tests, or the final decision to merge the code. If they had used that info, it would be like trying to predict the weather by looking at the ground after it rained. The whole point was to see if you could guess the outcome before the rain starts.

They also ruled out the idea that the computer could replace human reviewers. The results show that while the computer is a great "triage" tool (like a traffic cop directing cars), it cannot fully explain the complex social and workflow reasons why a review takes so long.

The Bottom Line

In short, the paper suggests that we can build a smart system to help managers prioritize their work. If a package has a clear label, a reasonable size, and comes from a good source, the system can say, "This one looks safe, let's check it first." But when it comes to guessing how long a review will take or how much arguing will happen, the system hits a wall. The paper concludes that these tools are best used as a helpful sidekick to human judgment, not as a replacement for it. The future of this tech might get better if we add more clues, like what the reviewers are currently busy with or how the code actually works, but for now, the "outside of the box" clues only tell us part of the story.

Technical Summary: Predicting Acceptance and Review Effort in Human and Agent Pull Requests

Problem Statement
As AI coding agents increasingly submit code changes alongside human developers, software maintainers face a new challenge in managing pull request (PR) queues. While prior research has established that PR outcomes can be predicted using submission-time signals, these studies largely assumed human-authored contributions. The emergence of agent-authored PRs introduces uncertainty regarding whether established predictive signals (e.g., patch size, description quality) remain valid when the review queue contains a mix of human and AI contributions. Furthermore, while acceptance prediction is a binary outcome, estimating the review effort (time and interaction required) is more complex, as it depends on socio-technical factors like reviewer availability and workflow timing that are not visible at submission time.

Methodology
The study utilizes the AIDev dataset, which contains PRs from both human developers and AI agents. The authors constructed a leakage-aware prediction pipeline designed to simulate a real-world early-triage scenario.

Data Constraints: The feature set is strictly limited to information observable at the moment a PR is opened. This explicitly excludes post-submission artifacts such as reviewer comments, CI feedback, merge decisions, and timestamps.
Feature Engineering: Features were categorized into four groups:
1. PR Metadata and Text: Title/body length, presence of URLs, code blocks, and lightweight task-intent tags (e.g., "fix," "feature") derived via keyword rules.
2. Repository Context: Stars, forks, primary language, and log-transformed popularity.
3. Temporal Signals: Hour of day and day of week of submission.
4. Diff Statistics: Lines added/deleted, file count, total churn, and file-type composition.
Experimental Design: The study evaluated four distinct views to ensure robustness: Pooled (all PRs), Human-only, Agent-only, and Balanced (down-sampled to equal human/agent ratios).
Models:
- RQ1 (Acceptance Prediction): A binary classification task evaluated using Logistic Regression, Random Forests (RF), Gradient Boosting (GBC), Extra Trees (ET), and Multi-Layer Perceptrons (MLP).
- RQ2 (Review Effort Estimation): A regression task predicting review-comment count and time-to-merge using Random Forest Regressors and Gradient Boosting Regressors.
Validation: Experiments used 5-fold stratified cross-validation. Hyperparameter tuning and preprocessing were performed strictly within the training fold to prevent data leakage.

Key Contributions

Mixed-Contributor Analysis: The paper is among the first to systematically evaluate PR prediction models in a setting containing both human and agent-authored contributions, testing the stability of predictive patterns across contributor types.
Leakage-Aware Pipeline: The study rigorously enforces a "submission-time only" constraint, ensuring that models do not rely on information (like CI results or comments) that would not be available for real-time triage.
Dual-Task Evaluation: By addressing both acceptance (RQ1) and effort estimation (RQ2), the paper distinguishes between the feasibility of predicting whether a PR will be merged versus how much work it will require.

Results

Acceptance Prediction (RQ1):
- Tree-based models (RF, GBC, ET) achieved high performance with F1 scores above 0.95 across all views.
- The most influential predictors were textual clarity (body/title length) and metadata, rather than raw diff size.
- While the majority baseline achieved high F1 due to class imbalance (most PRs are merged), its AUC was 0.50, indicating no ranking ability. The learned models significantly outperformed the baseline in ranking (AUC ~0.67).
- Agent-authored PRs showed slightly higher separability due to more regular formatting, but human PRs remained predictable.
Review Effort Prediction (RQ2):
- Prediction was significantly more difficult than acceptance.
- Comment Count: Achieved an $R^2$ of 0.20 and MAE of 1.00.
- Time-to-Merge: Achieved an $R^2$ of 0.12 and MAE of 24.00 hours.
- For effort estimation, diff-related variables (lines added/deleted, churn) became more prominent predictors than text features, suggesting that the scope of change drives workload more than the presentation of the PR.
- Agent-authored PRs tended to receive more discussion and longer review cycles, but the prediction error rates were comparable to human PRs.

Significance and Claims
The paper concludes that early PR signals are sufficient to support acceptance triage but insufficient to fully explain review effort.

Advisory Role: The authors explicitly state that these models should serve as advisory tools to help maintainers prioritize unclear or potentially costly PRs, rather than as automated decision-makers. The moderate AUC values indicate that models are not perfect ranking systems and should not replace human judgment.
Human-Agent Dynamics: The presence of agent-authored PRs does not render early prediction infeasible. However, agent PRs often undergo more scrutiny, suggesting that authorship type influences review dynamics even if it does not drastically alter the predictability of the outcome from submission-time features.
Limitations: The study acknowledges that review effort is a "noisier socio-technical construct" heavily influenced by factors outside the PR itself (e.g., reviewer availability, CI delays). Therefore, the RQ2 results are presented as early estimates of workload rather than definitive explanations.

The paper positions its findings as a practical step toward managing mixed human-AI development environments, emphasizing that while acceptance can be estimated early, accurate effort prediction requires richer context (semantic analysis, CI state, reviewer load) not available at submission time.