Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
This study of Microsoft's early 2026 rollout of command-line AI coding agents reveals that adoption is driven by social networks, retention correlates with coding activity rather than demographics, and users achieve a 24% increase in merged pull requests, demonstrating that these tools offer sustained productivity gains beyond novelty effects.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a massive software company (Microsoft) deciding to give its thousands of engineers a new, super-smart assistant. This isn't just a spell-checker; it's a "command-line agent" that can type code, fix bugs, and build features on its own if you tell it what to do. Think of it like handing every engineer a personal robot butler that lives inside their computer terminal.
In early 2026, Microsoft rolled out two versions of these robots: Copilot CLI (made by GitHub, which Microsoft owns) and Claude Code (made by Anthropic).
The company had three big worries:
- Who will actually try the robots?
- Who will keep using them, or will they just play with them for a day and quit?
- Is the robot worth the money? (These robots are expensive to run, costing millions of dollars in "tokens" per year).
Here is what the researchers found, explained simply:
1. The "Water Cooler" Effect (Adoption)
The biggest factor in whether an engineer tried the robot wasn't their job title, how long they'd been at the company, or their age. It was who they knew.
- The Analogy: Imagine a new dance craze starts at a party. You are most likely to try it if you see your friends doing it, especially if your boss or the people you work with most closely are dancing.
- The Finding: If an engineer saw their peers (people they reviewed code with) or their skip-level managers (their boss's boss) using the tool, they were much more likely to try it. In fact, if more than 25% of a specific peer group was using it, the odds of an engineer trying it jumped by over 200%.
- The Surprising Twist: Engineers who were already heavy users of the old AI tools (inside their code editor) were actually less likely to stick with the new command-line robot. It's like someone who loves their current car might try a new model, but if they like their old car too much, they won't switch over permanently.
2. The "Busy Bee" Effect (Retention)
Once people tried the tool, who kept using it?
- The Analogy: Think of the tool as a new, high-tech power drill. A professional contractor who is already drilling holes every day is the most likely to keep using the new drill. A hobbyist who rarely drills might try it once and put it back in the box.
- The Finding: The engineers who were already very busy (merging lots of code) were the ones who stuck with the tool. The more code they shipped, the more they kept using the robot. Junior engineers and managers didn't show a strong pattern of sticking with it one way or the other.
3. The "Super-Productivity" Boost (Impact)
The most important question: Did the robots actually help?
- The Analogy: Imagine giving a group of writers a magical typewriter that finishes their sentences. Do they write more books?
- The Finding: Yes. Engineers who used these tools merged 24% more Pull Requests (code changes) than they would have without them.
- The "No Fade" Surprise: Usually, when people get a new toy, they get excited for a month and then get bored. This study looked at four months and found the boost never went away. The engineers didn't just get faster; they started tackling bigger, more complex projects they usually avoided. One engineer said, "I no longer think about narrow solutions; I can use the agent to think broadly."
4. Tool Showdown: Copilot vs. Claude
Microsoft offered two different robots. Which one worked better?
- The Finding: Copilot CLI (the one Microsoft owns) resulted in about twice as much extra work as Claude Code.
- Why? The paper suggests two reasons:
- Microsoft engineers might just be better at using the tool their own company built because it fits their specific workflow.
- The two tools might be used for different types of tasks, but the one Microsoft built seemed to unlock more "merged code" for their specific environment.
5. The Bottom Line
The study concludes that these AI agents aren't just a "novelty" (a fun toy that gets old quickly). They are a real productivity booster that lasts.
However, the authors add a crucial warning: More code doesn't automatically mean better code.
- The Analogy: If you give a chef a machine that chops vegetables 24% faster, they can make more soup. But that doesn't mean the soup tastes better. The study measured how much soup was made, not how good it tasted.
In short: If you want your team to use these AI tools, don't just send an email. Make sure the team sees their peers and leaders using them. And if they do use them, expect them to get significantly more work done, provided they are already busy workers. But remember, you still need to check if that extra work is actually high quality.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.