Are Coding Agents Generating Over-Mocked Tests? An Empirical Study
This empirical study of over 1.2 million commits reveals that coding agents are significantly more likely than non-agents to generate tests and, notably, to over-mock them, raising concerns about test maintainability and effectiveness while highlighting the need for better agent configuration guidance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have hired a team of super-fast, super-smart robot assistants (called "Coding Agents") to help you build a house. These robots don't just lay bricks; they can also write the instruction manuals, check the plumbing, and even write the safety inspection reports (which in software are called "tests").
This paper is like a detective report that went into the construction sites of 2,168 different digital houses to see how these robots are actually doing their job. Specifically, the researchers wanted to know: Are these robots writing too many "fake" safety checks?
Here is the breakdown of what they found, using simple analogies:
1. The "Over-Protective" Robot
In software testing, a "mock" is like a dummy actor in a movie. If you are filming a scene where a character talks to a bank manager, but you don't have a real bank manager available, you might use a friend in a suit who just says pre-written lines. This is a "mock." It makes the scene easier to film (the test is faster and easier to write), but it doesn't prove the real bank manager would actually behave that way.
The study found that the robot assistants are obsessed with using these dummy actors.
- The Stat: When human developers write a test, they use a "dummy actor" about 26% of the time. When the robots write a test, they use a dummy actor 36% of the time.
- The Metaphor: It's as if the robots are so afraid of the real world (real databases, real internet connections) that they prefer to rehearse the entire play with cardboard cutouts instead of real people. They do this more often than humans do.
2. The Robots Love Writing Safety Checks
The researchers also checked how often the robots actually wrote the safety inspection reports (tests) in the first place.
- The Stat: About 23% of the work the robots did involved writing or changing tests. Humans only did this about 13% of the time.
- The Metaphor: The robots aren't just laying bricks; they are actively writing the rulebooks. In fact, in brand-new construction sites (repositories created in 2025), the robots are responsible for writing 17% of all the safety checks, a number that is growing fast.
3. One-Size-Fits-All Tools
The paper looked at what kind of dummy actors the robots were using. There are different types of "test doubles" (fake objects): some just sit there (dummies), some pretend to be real (stubs), and some watch what happens (spies).
- The Finding: Humans use a variety of tools. They might use a "spy" here and a "fake" there.
- The Robot Habit: The robots are lazy with their variety. They use the "Mock" tool 95% of the time.
- The Metaphor: Imagine a toolbox. Humans have a hammer, a screwdriver, a wrench, and a saw. The robots show up with a giant hammer and try to use it for everything, even when they need a screwdriver. They rely on one specific type of fake object almost exclusively.
4. The Danger of "Too Fake"
Why does this matter? The paper warns that while using too many "dummy actors" makes it easier for the robots to write the tests quickly, it might make the tests less useful.
- The Risk: If you only test your house with cardboard cutouts, you might think the plumbing is perfect. But when you turn on the real water, the pipes might burst because the cardboard didn't react like real metal.
- The Conclusion: The robots are generating tests that are easy to create but might fail to catch real problems because they are too isolated from reality.
What Should We Do?
The authors suggest that since these robots are so eager to use "fake" actors, we need to give them better instructions.
- The Fix: Just like you would tell a human apprentice, "Don't use a fake pipe for the main water line," we need to write specific rules in the robots' instruction manuals (configuration files). We need to tell them: "Use real objects whenever possible, and only use fakes when absolutely necessary."
In short: The robots are working hard and writing lots of safety tests, but they are a bit too eager to use "fake" versions of reality, which might make their safety checks less reliable. We need to teach them to mix in more real-world scenarios.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.