Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories
This paper presents a validated multi-method census of 180 million repositories revealing that single-signal detection methods severely underestimate AI coding agent prevalence—missing up to 97% of activity—and that different detection channels capture distinct, non-overlapping populations of agents and types of work, necessitating a holistic approach to accurately measure their impact on the open-source supply chain.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the world of open-source software as a massive, bustling city with over 180 million buildings (repositories). For a long time, we thought we knew who was building these structures: mostly human architects. But recently, a new kind of invisible construction crew has arrived: AI Coding Agents.
The problem? These AI crews are masters of disguise. Some wear bright uniforms, some leave a signature on the blueprints, and some work so quietly they look exactly like human workers.
This paper is like a massive, high-tech census that tried to count these invisible workers. The researchers realized that if you only look for one specific sign (like a uniform), you will miss almost everyone. Here is what they found, explained simply:
1. The "One-Signal" Trap
Imagine trying to count all the birds in a forest.
- Method A: You only count birds that wear red hats.
- Method B: You only count birds that sing a specific song.
- Method C: You only count birds that leave a specific type of feather.
If you only use Method A, you might think there are only 10 birds. But if you combine all three methods, you might find there are actually 300.
The Paper's Finding: The researchers found that relying on just one way to spot AI (like looking for a specific bot account name) misses 97% of the activity. For one popular AI tool called "Claude Code," looking for its official bot account only found 28,000 commits. But when they looked for all the different ways it leaves a trace (like messages in the code or configuration files), they found 850,000. That is a 30-fold difference.
2. The Four Types of "Disguises"
The researchers created a "Wanted Poster" with four categories to catch these agents:
- Type A: The Uniformed Bot. These are easy to spot. They log in with a specific robot email address (like
bot@anthropic.com). It's like a construction worker wearing a bright orange vest. - Type B: The Signature. These agents don't use a robot account, but they leave a note in the message saying, "Generated by Replit." It's like a human worker signing their name on the blueprint with a special stamp.
- Type C: The Hidden Suffix. These are humans using AI tools who add a tiny tag to their name, like
John (aider). It's like a human worker wearing a small pin that says "AI Assistant." - Type D: The Silent Ghost. These are the sneakiest. They don't change the code messages or the author names at all. They only leave a "setup file" (like a
.cursorrulesfile) in the project folder. It's like a ghost that only leaves a toolbox behind but never touches the bricks. This group is huge. For example, GitHub Copilot is mostly found this way; it's everywhere in the setup files but almost invisible in the actual code history.
3. Two Different Cities (The "Channel Disconnect")
The researchers compared their "Commit City" (looking at code changes) with another study's "Pull Request City" (looking at project proposals).
They found these two cities are almost entirely empty of each other's residents!
- Codex (an AI agent) is the king of the "Pull Request City," creating thousands of proposals, but it is a ghost in the "Commit City."
- Claude Code is the king of the "Commit City," making thousands of direct changes, but it rarely shows up in the "Pull Request City."
The Lesson: If you only look at one type of city, you get a completely wrong picture of what the AI is actually doing. One group seems to be building new features, while the other seems to be fixing broken pipes. The paper concludes that how the AI is deployed (as a bot vs. a direct tool) changes what kind of work it appears to do, not necessarily the tool itself.
4. When Did They Arrive?
The census looked at data from late 2024 to early 2026.
- The "Born with AI" Group: Some projects were built from day one with these AI agents. They are like houses built with a robotic arm from the foundation up.
- The "Legacy" Group: Many old, established projects (some decades old) suddenly started using these agents years after they were built. It's like an old family home suddenly getting a smart-home renovation.
5. Are They Doing Good Work?
The researchers checked if the AI was making mistakes.
- Bug Fixes: The AI agents seemed to be used mostly for fixing bugs and maintenance, rather than building brand-new features (in the commit channel).
- Quality: Interestingly, the code written by these agents was less likely to be reverted (undone) than human code in some cases, suggesting it was stable. However, when they did make changes, the changes were often larger than a human would make.
The Bottom Line
We cannot just look for a "bot account" to know how much AI is helping us build software. The AI workforce is diverse, invisible, and uses many different "uniforms." If we only look for the obvious ones, we are blind to the vast majority of the work being done. To truly understand the software supply chain, we need to look for the uniforms, the signatures, the name tags, and the toolboxes.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.