Who Writes the Docs in SE 3.0? Agent vs. Human Documentation Pull Requests
This study analyzes nearly 2,000 documentation pull requests in the emerging SE 3.0 era to reveal that AI agents are submitting significantly more documentation changes than humans, yet these contributions are often integrated with minimal human review, raising critical concerns about documentation quality assurance and the reliability of human-AI collaboration.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a bustling construction site where a new kind of worker has arrived: the AI Agent. In the past, these robots were like silent assistants who only answered when you asked a specific question. But in this new era, called "SE 3.0," they are more like autonomous teammates. They can walk onto the site, decide what needs fixing, and even submit their own blueprints for changes without being told to do so.
This paper is like a site inspection report that asks a simple but crucial question: "Who is actually writing the instruction manuals (documentation) for these projects, and are the human foremen checking their work?"
Here is what the researchers found, broken down into everyday terms:
1. The Robots Are Writing the Manuals (More Than You Think)
The researchers looked at nearly 2,000 "Pull Requests" (which are like formal requests to update a project's files). They wanted to see who was doing the writing.
- The Result: The AI agents were the heavy lifters. They submitted 1,478 documentation updates, while human developers only submitted 519.
- The Analogy: Imagine a library where the robots are writing 75% of the new book summaries and shelf labels, while the human librarians are only writing the remaining 25%.
- The Surprise: The robots didn't just work together with humans. In fact, 96% of the files were edited either by a robot or by a human, but rarely by both working on the same file at the same time. It's as if the robots and humans are working in separate shifts, rarely crossing paths to collaborate on the same page.
2. The "Mixed Bag" Problem
The researchers also noticed something odd about what the robots were changing.
- The Result: About 29% of the requests labeled as "documentation updates" by the robots didn't actually touch any documentation files at all. They were changing code or other non-document files.
- The Analogy: It's like a robot raising its hand and saying, "I'm here to fix the library's sign!" but instead, it goes and rearranges the furniture in the back room. The label said "Sign Fix," but the work was something else entirely. This can confuse the human reviewers who are expecting to see a sign being fixed.
3. The "Trust Fall" (Are Humans Checking the Work?)
This is the most critical part of the study. When a robot submits a change, a human usually has to review and approve it. The researchers wanted to know: Do humans actually read and edit the robot's work, or do they just hit "Approve"?
- The Result: The humans are mostly accepting the robots' work as-is.
- In 85% of cases, the humans kept almost everything the robot wrote.
- In 34% of cases, the humans didn't delete a single line of the robot's additions.
- On average, 86.8% of the lines the robot added stayed in the final version.
- The Analogy: Imagine a student handing in an essay to a teacher. The teacher glances at it, says "Looks good," and files it away without crossing out a single word or adding a single correction.
- The Concern: While this shows the robots are doing a decent job, it raises a red flag. If the robots are writing the manuals and the humans aren't really editing or checking them, there's a risk that the manuals could be slightly wrong, confusing, or outdated, and no one would catch it.
The Bottom Line
The paper concludes that AI agents have become the primary writers of software documentation in these projects. However, the human "reviewers" are often acting more like stampers than editors. They are letting the robots' work pass through with very little follow-up.
The authors warn that while this is efficient, it creates a new risk: We might be trusting the robots to write the instruction manuals without enough human eyes to double-check that the instructions are actually correct. They suggest we need better ways to ensure these robot-written manuals are reliable before we hand them over to the rest of the world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.