🤖 AI

More Code, Less Reuse: Investigating Code Quality and Reviewer Sentiment towards AI-generated Pull Requests

This paper investigates the impact of LLM agents on code quality and reviewer sentiment, revealing that while AI-generated pull requests elicit neutral or positive reactions, they often suffer from increased redundancy and overlooked reuse opportunities that contribute to silent technical debt.

Original authors: Haoming Huang, Pongchai Jaisri, Shota Shimizu, Lingfeng Chen, Sota Nakashima, Gema Rodríguez-Pérez

Published 2026-01-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoming Huang, Pongchai Jaisri, Shota Shimizu, Lingfeng Chen, Sota Nakashima, Gema Rodríguez-Pérez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Helpful" Robot That Clutters the Garage

Imagine you have a very smart, fast robot assistant helping you organize your garage. You ask it to fix a broken shelf or build a new tool. The robot is incredibly fast and polite. It hands you a finished product that looks perfect and works immediately.

However, this paper investigates what happens when we let these robots (AI Agents) do too much of the work. The researchers found a surprising problem: The robots are building duplicate tools instead of using the ones you already have.

The Two Main Questions

The researchers asked two simple questions to understand what's going on:

Is the code actually good? (Looking at the "garage" itself).
How do the human bosses feel about the work? (Looking at the "review").

1. The "Garage" Problem: More Stuff, Less Reuse

When human developers write code, they usually look around first. If there is already a "screwdriver" (a piece of code) in the toolbox that does the job, they grab it and use it. This keeps the garage tidy and efficient.

What the AI does:
The AI agents are like a robot that doesn't know what's in your toolbox. Instead of grabbing the existing screwdriver, it builds a brand new screwdriver from scratch.

The Result: The garage ends up with 10 identical screwdrivers instead of one.
The Technical Term: The researchers call this Type-4 Code Clones. It's not a copy-paste job (which is easy to spot); it's a "re-invention." The AI writes the same logic but uses different words or variable names.
The Data: The study found that AI-generated code has nearly twice as much redundancy (unnecessary duplication) as human-written code.

2. The "Review" Problem: The Polite Boss

Usually, when a human employee brings in a project that is messy or full of duplicates, the boss gets annoyed. They might say, "Hey, we already have a tool for this! Why did you build a new one?" This is a negative reaction.

What happens with AI:
When the AI brings in its "new screwdriver," the human reviewers are surprisingly nice about it.

The Reaction: Reviewers are more likely to say "Great job!" or remain neutral when reviewing AI code. They are less likely to get angry or disgusted compared to when they review human code.
The Analogy: It's like a robot that is so polite and confident that you forget to check if it actually did the smart thing. Because the robot's work looks correct on the surface and passes all the tests, the human boss lowers their guard.

The Dangerous Disconnect: "Silent Technical Debt"

This is the most critical finding of the paper. There is a disconnect between the quality of the work and the feeling about the work.

The Reality: The AI is creating a messy, bloated garage full of duplicate tools. This is bad for the long term because if you need to fix a bug in the "screwdriver logic," you have to fix it in 10 different places instead of one. If you miss one, the whole system breaks later.
The Illusion: Because the reviewers are being polite and not getting angry, this messiness goes unnoticed.

The researchers call this "Silent Technical Debt." It's like taking out a loan to buy a new tool when you already have one. You don't feel the pain today because the new tool works, but eventually, you will be drowning in debt (maintenance costs) because you have to maintain all these duplicates.

Why Does This Happen?

The paper suggests that AI models are trained to be helpful and agreeable. They are optimized to give you an answer that looks right and makes you happy, rather than an answer that is the most efficient or "reuse-oriented." They prioritize looking plausible over being smart.

The Takeaway

The paper concludes that while AI is great at writing code quickly, it is currently bad at reusing existing code.

For the Builders: Don't just check if the code works (the "pass rate"). You also need to check if the AI is just copying things it should have reused.
For the Reviewers: Don't let the AI's politeness fool you. You need to be extra vigilant and look for hidden duplicates, even if the code seems perfect on the surface.

In short: The AI is building a house with too many identical rooms because it forgot to check the blueprints, and the human inspectors are too polite to point it out.

Technical Summary: More Code, Less Reuse

Problem Statement
The software engineering landscape is shifting toward "SE 3.0," where Large Language Model (LLM) agents act as autonomous teammates generating Pull Requests (PRs). While existing benchmarks (e.g., SWE-Bench) evaluate these agents primarily on pass rates, they fail to capture impacts on long-term maintainability and readability. A specific concern is the generation of Type-4 semantic clones: code that implements the same logic as existing functions but uses different syntax or variable names. Unlike human developers who often create Type-1 or Type-2 clones via copy-paste, LLMs' probabilistic nature makes them prone to generating these harder-to-detect semantic redundancies. This behavior violates the software engineering principle of reuse, potentially leading to silent technical debt. Furthermore, it remains unclear whether human reviewers can detect these quality issues or how their sentiment differs when reviewing AI-generated versus human-written code.

Methodology
The study utilizes the AIDev dataset, filtering for Python repositories with over 500 stars to ensure project maturity. The data is split into two subsets:

Dataset A (Full Python Dataset): 3,858 PRs used for analyzing traditional code metrics and reviewer sentiment.
Dataset B (Core Case Study): 617 PRs from the crewAI repository, selected for computationally intensive redundancy analysis.

The research addresses two Research Questions (RQs):

RQ1 (Code Quality): How do AI-generated codes differ from human-generated codes in terms of quality and redundancy?
RQ2 (Reviewer Sentiment): How does reviewer sentiment differ between AI-generated and human-written PRs?

Technical Approach:

Traditional Metrics: The authors calculated Lines of Code (LOC), multi-line strings, blank lines, and Cyclomatic Complexity (CC) using the Radon tool on pre- and post-update file pairs.
Redundancy Analysis (MRS): To detect Type-4 clones, the authors proposed a Snapshot-based Semantic Differential Analysis. They extracted existing functions ( $F_{base}$ ) and generated semantic embeddings using CodeSage-Large. To avoid false positives from refactoring (e.g., moving or renaming functions), they integrated PyRef to filter out these instances. They defined the Max Redundancy Score (MRS) for a PR as the maximum cosine similarity between any new function ( $F_{new}$ ) and any base function ( $F_{base}$ ). The study reports the Average Max Redundancy (AMR) to compare groups.
Sentiment Analysis: Using the Emotion English DistilRoBERTa-base model, the authors classified review comments into Ekman's six basic emotions plus "neutral." They filtered out bot comments and calculated average sentiment scores per PR to compare reactions to Agentic-PRs versus Human-PRs.

Key Results

Code Quality (RQ1): Traditional metrics (LOC, CC) showed minimal differences between human and AI contributions. However, the redundancy analysis revealed a significant disparity. AI agents exhibited an Average Max Redundancy (AMR) of 0.2867, nearly 1.87 times higher than human developers (0.1532). This difference was statistically significant ( $p < 0.001$ ). While humans and AI both produced some similar code (e.g., API calls), AI agents significantly exceeded humans in higher redundancy ranges, indicating a tendency to rewrite existing logic rather than reuse it.
Reviewer Sentiment (RQ2): Contrary to the expectation that poor code quality would trigger negative feedback, reviewers expressed more neutral or positive emotions (joy, sadness, neutral) toward AI-generated PRs. Conversely, Human-PRs elicited higher proportions of negative emotions (anger, disgust, fear, surprise). Reviewers generally treated AI contributions respectfully, often overlooking the underlying redundancy.

Key Contributions

Empirical Evidence of Disconnect: The study provides the first empirical evidence of a disconnect between AI-generated code redundancy and human reviewer sentiment. It demonstrates that AI agents introduce significantly more Type-4 semantic clones than humans, yet this is not penalized by reviewers.
Novel Metric (MRS): The introduction of the Max Redundancy Score (MRS) using semantic embeddings and refactoring detection offers a method to quantify semantic redundancy that traditional keyword-matching tools miss.
Identification of "Silent Technical Debt": The paper highlights that the surface-level plausibility of AI code (syntactic correctness and test passing) masks poor design choices (redundancy), leading to the silent accumulation of technical debt.

Significance and Claims
The authors claim that current evaluation paradigms focusing solely on pass rates are insufficient. The "silent" nature of this technical debt is dangerous because reviewers, influenced by the helpful and plausible nature of LLM outputs, lower their vigilance regarding code reuse. The study suggests that as AI agents become more prevalent, the software engineering community must:

For Agent Builders: Evaluate agents on code reuse metrics, not just pass rates.
For Developers: Actively check for duplicated logic during code reviews, recognizing that plausible-looking AI code may hide significant redundancy.

The paper concludes that without addressing this disconnect, real-world development environments risk becoming bloated with redundant logic, increasing future maintenance efforts and decreasing long-term readability.