How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
This large-scale study of over 24,000 merged GitHub pull requests reveals that AI coding agents differ significantly from human contributors in commit frequency and file modification patterns, while demonstrating slightly higher consistency between their pull request descriptions and actual code changes.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine software development as a massive, bustling construction site where thousands of people (human developers) and a new wave of robot foremen (AI coding agents) are building and fixing buildings (software code) every day.
This paper is like a forensic investigation into the work logs of these two groups. The researchers wanted to know: When a robot foreman submits a repair plan (a "Pull Request") to the construction site, how is it different from a human foreman's plan?
Here is the breakdown of their findings, translated into everyday language:
1. The Setup: The Great Data Dump
The researchers grabbed a massive dataset from GitHub (a giant library of code) containing over 24,000 repair plans submitted by AI bots and 5,000 submitted by humans. They looked at two main things:
- The Work: How much did they actually change? (Did they fix a single loose screw or rebuild a whole wall?)
- The Note: Did the written explanation match the work done? (Did the note say "fixed the leak" when they actually replaced the roof?)
2. Finding #1: The "Swiss Army Knife" vs. The "Sledgehammer"
The Question: How do the robots and humans differ in how they tackle problems?
The Analogy:
- Humans are like Sledgehammers. When a human decides to fix something, they often take a big swing. They might touch many different rooms in a house, write a long list of changes, and break the job down into many small steps (commits). Their work is broad and variable.
- AI Agents are like Precision Swiss Army Knives. They tend to be very focused. They usually make small, localized fixes. Instead of rebuilding the whole kitchen, they just tighten the faucet.
The Surprising Detail:
Not all robots are the same!
- Some robots (like Devin or Copilot) are extremely precise, making tiny, surgical changes.
- Others (like Claude Code or OpenAI Codex) are a bit more like humans; they sometimes take bigger swings and touch more files.
The Big Takeaway:
The biggest difference isn't how much code they change, but how they organize it. Humans tend to spread their changes across many files and many steps. AI tends to keep things compact and focused. If you see a repair plan that touches 50 different files, it's likely human. If it's a tiny, focused fix, it's likely a robot.
3. Finding #2: The "Post-It Note" Test
The Question: Do the robots write good notes to explain their work?
The Analogy:
Imagine you hand a mechanic a car with a broken light.
- Bad Note: "Fixed stuff." (Vague)
- Good Note: "Replaced the headlight bulb on the driver's side." (Clear)
The researchers used two types of "tests" to check the notes:
- The Word Match Test (Lexical): Do the words in the note appear in the code? (e.g., Does the note say "bulb" and the code mention "bulb"?)
- Result: Neither humans nor robots were great at this. The words rarely matched perfectly because code is technical and notes are natural language.
- The "Vibe Check" Test (Semantic): Does the meaning of the note match the meaning of the code, even if the words are different?
- Result: Both groups were excellent here. Whether it was a human or a robot, the notes accurately captured the intent of the fix.
The Twist:
The robots actually did slightly better than humans at matching their notes to their code. Their "Post-It notes" were consistently accurate. They didn't get confused or write vague descriptions as often as humans did.
4. Why Does This Matter?
This study helps us understand how to work with these new robot coworkers:
- For Code Reviewers: If you see a repair plan that is tiny and touches only one file, don't panic thinking it's a mistake. It's likely a robot doing a precise job. However, if a robot tries to change 50 files at once, you should double-check it, because that's not their usual style.
- For Trust: We can trust that when a robot says, "I fixed the login bug," it actually fixed the login bug. Their descriptions are reliable.
- For the Future: As we move into "Software Engineering 3.0," we need to change how we manage projects. We can't treat robots exactly like humans. We need to expect them to be more modular, focused, and consistent in their communication.
Summary
In short: AI coding agents are not just "faster humans." They are a different species of contributor. They work in smaller, more focused bursts and are surprisingly good at explaining what they did. They aren't trying to rebuild the whole house; they are just tightening the screws, and they're doing a very consistent job at it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.