💻 computer science

Novice Developers Produce Larger Review Overhead for Project Maintainers while Vibe Coding

This study reveals that novice developers using AI coding agents ("vibe coders") generate significantly larger volumes of code that require more extensive review, lower acceptance rates, and longer resolution times compared to experienced developers, indicating that low-experience contributors cannot simply replace experts without increasing the review burden on maintainers.

Original authors: Syed Ammar Asdaque, Imran Haider, Muhammad Umar Malik, Maryam Abdul Ghafoor, Abdul Ali Bangash

Published 2026-03-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Syed Ammar Asdaque, Imran Haider, Muhammad Umar Malik, Maryam Abdul Ghafoor, Abdul Ali Bangash

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling construction site where everyone is trying to build houses. In the past, you needed a master architect to draw the blueprints and a skilled crew to lay the bricks. If a novice tried to build a house, they might make mistakes, take a long time, and need a lot of help from the foreman to fix things.

Now, imagine a new technology arrives: Magic Bricks. These are AI robots that can lay bricks instantly just because you tell them what to do. This is what the paper calls "Vibe Coding." You don't need to know how to mix cement; you just "vibe" with the AI, give it a prompt like "Build me a two-story house with a red door," and it spits out a finished wall.

The big question the researchers asked was: "If a novice uses these Magic Bricks, can they build just as well as an expert who also uses them? Or does the novice still cause more trouble for the foreman?"

Here is what they found, translated into everyday terms:

1. The Novice is a "Speed Demon" (But a Messy One)

The researchers looked at thousands of projects on GitHub (a giant online library of code). They split the coders into two groups:

The Veterans (ExpHigh): People who have been coding for years.
The Newbies (ExpLow): People who are just starting out.

The Surprise: When the Newbies used the AI, they didn't just build a little house; they tried to build a skyscraper.

The Newbies submitted code changes that were 2.15 times bigger (more "commits" or steps) and touched 1.47 times more files than the Veterans.
The Analogy: Imagine the Veteran asks the AI, "Fix the leaky faucet." The AI fixes it perfectly. The Newbie asks, "Fix the faucet," but the AI, sensing the Newbie's lack of direction, decides to rebuild the entire kitchen, the plumbing, and the roof. The Newbie is generating massive amounts of work, thinking they are being productive.

2. The "Foreman" (Project Maintainer) is Overwhelmed

In software projects, there are "Maintainers" (the foremen) who have to check every piece of work before it gets accepted.

More Criticism: The Newbies' giant projects got 4.5 times more comments from the foreman. The foreman had to say, "Hey, this doesn't fit," "That's the wrong color," and "You broke the plumbing again."
More Rejections: The Newbies' projects were 31% less likely to be accepted. They were rejected more often because they were too messy or didn't fit the building's rules.
Longer Wait Times: It took the Newbies 5 times longer to get their projects approved. The foreman had to spend hours untangling the Newbie's giant, messy construction site.

3. Why is this happening? (The Two Big Problems)

The researchers dug into why the Newbies were causing so much trouble, even with the AI helping them. They found two main issues:

Problem A: The "Wrong Tool" Syndrome (Infrastructure Mismatch)
The AI is great at writing code, but it doesn't know the specific rules of your construction site.
- Analogy: The AI builds a beautiful door, but it's the wrong size for your specific doorframe. The Newbie doesn't know how to measure the frame, so they just keep asking the AI to "make it fit" by changing the door size, rather than fixing the frame. They end up debugging the environment (the doorframe) instead of the code, wasting everyone's time.
Problem B: The "Missing Context" Problem (Integration Friction)
The AI writes code that works in a vacuum, but real software is like a giant puzzle where every piece must fit perfectly with the others.
- Analogy: The AI builds a perfect window, but it doesn't know that the window needs to match the specific security locks the building already has. The Newbie doesn't realize this mismatch. The foreman has to stop, look at the window, and say, "This doesn't lock with our system. Go fix it." The Newbie has to go back and manually tweak the AI's work to make it fit the existing puzzle.

The Bottom Line

The paper concludes that AI is a powerful tool, but it doesn't replace the need for experience.

For the Bosses (Project Managers): You cannot just swap out your expert builders for newbies with AI and expect the same results. If you do, your "foremen" (maintainers) will be buried under a mountain of paperwork and reviews. You need to hire more reviewers or train the newbies on how to check the AI's work.
For the Newbies: Just because the AI can write code fast doesn't mean you understand the building. You still need to learn how to verify the work, understand the rules of the project, and know when to stop the AI from building a skyscraper when you only needed a shed.

In short: AI helps you write code faster, but if you don't know what you're doing, you're just writing more bad code faster, and someone else has to pay the price to fix it.

1. Problem Statement

The rise of "Software 3.0," characterized by AI agents collaborating with or autonomously performing programming tasks, has led to the emergence of "Vibe Coding." This workflow involves human developers directing AI agents via natural language prompts and validating the generated code.

While industry reports suggest AI tools enhance productivity, preliminary studies indicate that AI-assisted development may introduce new bottlenecks, such as increased task completion times due to verification challenges. A critical, unanswered question remains: Can novice developers using AI (vibe coders) effectively substitute for experienced developers? Specifically, does developer experience still matter when AI handles the code generation, or does the burden of verification shift disproportionately to project maintainers when novices are involved?

2. Methodology

The authors conducted a large-scale empirical study using the AIDev dataset, a corpus of AI-assisted Pull Requests (PRs) from GitHub open-source projects.

Data Source: 22,953 PRs generated by 1,719 distinct "vibe coders" (humans supervising AI agents) across repositories with >100 stars.
Definition of Vibe Coder: Humans who direct AI agents via prompts and validate code, excluding fully autonomous bot accounts.
Experience Segmentation:
- Developers were categorized based on an Experience Score: $\frac{\text{Total Commits}}{\text{Account Age}}$ .
- $Exp_{High}$ : Top two quartiles (Q3 & Q4) of experience scores (859 developers).
- $Exp_{Low}$ : Bottom two quartiles (Q1 & Q2) of experience scores (860 developers).
Metrics Analyzed:
- Contribution Magnitude: Number of commits and files changed per PR.
- Merge Effort: PR acceptance rate, resolution time (creation to merge), and review volume (total comments).
Statistical Analysis: The study utilized Mann-Whitney U tests for continuous variables (commits, files, time, comments) and Chi-Square tests for categorical variables (acceptance rates). Multiple comparison corrections (Benjamini-Hochberg) were applied to mitigate Type I errors.

3. Key Contributions

First Comparative Study: This is the first study to systematically compare AI-assisted contributions (vibe coding) across different levels of human developer experience.
Empirical Evidence of Overhead: It provides quantitative evidence that while AI enables novices to generate code rapidly, it does not eliminate the need for expertise; instead, it shifts the verification burden to maintainers.
Identification of Friction Points: Through manual inspection of high-volume review PRs, the authors identified specific technical failure modes (Infrastructure Mismatch and Integration Friction) unique to novice vibe coders.

4. Key Results

The study reveals a counterintuitive finding: Novice vibe coders ( $Exp_{Low}$ ) produce larger, more complex, and lower-quality contributions than experienced vibe coders ( $Exp_{High}$ ), resulting in significantly higher maintenance overhead.

A. Contribution Magnitude (RQ1)

Commits: $Exp_{Low}$ submitted 2.15× more commits per PR than $Exp_{High}$ .
Files Changed: $Exp_{Low}$ modified 1.47× more files per PR than $Exp_{High}$ .
Significance: These differences were statistically significant ( $p < 0.05$ ) across most PR categories, including features and style changes.

B. Merge Effort (RQ2)

Acceptance Rate: $Exp_{Low}$ $E x p_{L o w}$ PRs had a 31% lower acceptance rate compared to $Exp_{High}$ $E x p_{H i g h}$ .
- Example: In documentation PRs, $Exp_{High}$ had a 93.06% acceptance rate vs. 75.39% for $Exp_{Low}$ .
Resolution Time: $Exp_{Low}$ $E x p_{L o w}$ PRs took 5.16× longer to resolve (merge or close) than $Exp_{High}$ $E x p_{H i g h}$ PRs.
- Example: In "chore" PRs, resolution time was 0.61 days for $Exp_{High}$ vs. 2.83 days for $Exp_{Low}$ .
Review Volume: $Exp_{Low}$ PRs received 4.52× more review comments, indicating a massive increase in the effort required by maintainers to guide and correct novice contributions.

C. Root Causes of Friction

Manual analysis of high-review-volume PRs identified two primary failure modes for novices:

Infrastructure Mismatch: Novices struggle to align AI-generated code with specific build environment constraints (e.g., CI timeouts, runtime limitations), often debugging via CI because they cannot replicate the environment locally.
Integration Friction: AI-generated code often lacks necessary system context. Novices fail to align generated features with existing project architectures, privacy schemas, or integration standards, requiring extensive manual verification.

5. Significance and Implications

For Project Managers: Replacing experienced developers with novice vibe coders is not a safe strategy without increasing review capacity. The "productivity gain" of AI for novices is offset by a 5× increase in review time and a 31% drop in acceptance rates.
For Teams: Development teams must adapt by:
- Implementing adaptive PR review cycles (e.g., assigning extra reviewers to novice PRs).
- Focusing training on verification skills (testing, debugging, and architectural alignment) rather than just code generation.
- Using automated checks to catch infrastructure mismatches early.
For Research: The study establishes a framework ( $Exp_{High}$ vs. $Exp_{Low}$ ) for analyzing human-AI collaboration dynamics, suggesting that "experience" remains a critical variable even in the era of Software 3.0.

Conclusion: While AI allows novices to generate code at a scale previously impossible, it fundamentally shifts the workload from writing to verifying. Without sufficient expertise to validate and tune AI outputs, novice vibe coders create a disproportionate burden on project maintainers, leading to slower integration and higher rejection rates.