AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality
This paper presents an empirical study using the AIDev dataset to reveal that while AI coding agents frequently introduce maintainability and security issues in build code, they also effectively remove existing code smells and achieve a high acceptance rate of over 61% by developers, highlighting the need for AI-aware quality assessment frameworks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine software development as building a massive, complex house. Usually, you have two types of workers: the architects who design the rooms (the source code) and the construction managers who handle the logistics, like ordering materials, scheduling the crew, and ensuring the foundation is solid (the build code).
For a long time, we've been testing new AI robots to see if they can act as these architects. But this paper asks a different question: Can these AI robots also be good construction managers?
The authors, Anwar Ghammam and Mohamed Almukhtar, decided to put the AI's "construction management" skills to the test. They looked at a massive pile of real-world construction logs (GitHub pull requests) where AI agents tried to fix or update the "construction manuals" (build files like Maven, Gradle, and Makefiles).
Here is what they found, broken down simply:
1. The "Smell" Test: Did the AI mess up?
In the world of coding, a "code smell" isn't a literal odor. Think of it like a bad habit in a recipe.
- Good habit: "Add 2 cups of flour."
- Bad habit (Smell): "Add flour from the bag in the garage" (Hardcoded path) or "Use whatever flour is left over" (Wildcard usage).
The researchers found that when AI agents tried to write these construction manuals, they sometimes introduced bad habits.
- The Problem: In about 66 cases, the AI created new "smells." The most common bad habits were lack of error handling (not having a plan B if something breaks) and hardcoded paths (writing specific addresses into the code that won't work if you move the house).
- The Culprits: Not all AI robots were equal. One robot, "Copilot," introduced the most bad habits. Another, "Claude," didn't introduce any, but it only tried on three files, so it's hard to judge.
2. The "Fix-It" Crew: Did the AI clean up?
The researchers also wondered if the AI could act like a professional home inspector and remove bad habits that were already there.
- The Result: Yes! In 31 cases, the AI successfully cleaned up the construction manuals.
- How? The AI acted like a smart renovator. It took messy, scattered instructions and organized them. For example, instead of writing a secret password directly in the manual (a security risk), the AI moved it to a secure "vault" (external properties). It also removed unused tools and updated old, broken instructions.
- The Takeaway: The AI isn't just a chaotic builder; sometimes it's a very effective refactorer (someone who reorganizes things to make them better).
3. The Human Bosses: Did the humans say "Yes"?
Finally, the team asked: Do the human construction managers trust the AI?
- The Verdict: Surprisingly, yes. More than 61% of the AI's suggestions were accepted and merged immediately.
- The Reaction: Human reviewers often just said "Looks Good To Me" (LGTM) or even joked, "You can take my job now!" They rarely had to make changes before approving the AI's work. This suggests that, right now, humans trust the AI to handle these construction logistics with very little supervision.
The Big Picture
Think of this study as a report card for AI agents trying to manage the "plumbing and wiring" of software projects.
- The Good News: The AI is surprisingly good at cleaning up messy instructions and organizing the build process. Humans trust it enough to let it do the work most of the time.
- The Caution: The AI isn't perfect. It sometimes leaves behind "bad habits" like missing safety checks or using outdated tools.
The Conclusion: We shouldn't just let the AI run the construction site blindly. We need to build better "safety inspectors" (tools) to catch the AI's bad habits before they become part of the final house. The goal is to keep the AI's ability to clean up while stopping it from accidentally creating new messes.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.