Naive Defect-Recidivism Mining Is Inflated by Agent Workflow Artefacts: A Construct-Validity Study at Corpus Scale
This study demonstrates that mining version-control history to compare AI-agent and human fix durability is severely inflated by workflow artifacts, revealing through manual validation and programmatic correction that the initially observed higher recidivism rate for AI agents is an illusion that disappears when these artifacts are accounted for.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of software development, code is never truly finished; it is a living thing that requires constant repair. When a programmer fixes a bug, the goal is to make the error disappear forever. However, sometimes a fix fails, and the same problem returns, forcing the team to start over. In the software industry, this return of a problem is known as "recidivism." For years, researchers have studied how often human developers cause these returns, using automated tools to scan millions of lines of code and count how many times a fix is undone or a problem is reopened. Now, artificial intelligence has entered the workshop. AI coding agents, powered by large language models, are writing their own fixes and submitting them for approval. A natural question arose: are these AI fixes more durable than those made by humans, or do they fail more often? To answer this, scientists began mining vast archives of code history, looking for the same signals of failure they had used for humans. They expected to find a clear answer, but the data they found was hiding a trick.
A team of researchers set out to test a specific method for measuring these failures. They defined a metric called "Defect Resolution Debt," which simply counts the number of times a fix attempt is followed by another attempt on the same problem. If a fix is perfect, the count is zero. If the problem comes back, or if the fix is reverted, the count goes up. They applied this method to a massive collection of over three thousand software defects, comparing fixes made by a popular AI agent, GitHub Copilot, against those made by human developers. At first glance, the results were startling and seemed to confirm a worst-case scenario. The automated system flagged the AI fixes as failing much more often than the human ones. Specifically, the AI was flagged for recidivism in 13.5 percent of cases, compared to only 8.2 percent for humans. The statistical tools suggested this difference was real and significant, implying that the AI was less reliable at solving problems permanently.
However, the researchers suspected that the automated signal was being misled by the way the AI works, rather than by the quality of the code itself. They decided to look closer, moving beyond the automated counts to manually inspect the actual history of two hundred of these flagged problems. They hired two independent reviewers to examine the digital paper trail for each case, asking a simple question: did the problem genuinely return, or was the signal just a misunderstanding of the workflow? The human reviewers found that the automated system was being fooled by a specific pattern unique to how the AI operates. When the AI attempts to fix a bug, it often opens a "draft" version of a fix first, then closes it and opens a "final" version to replace it. To the automated counter, this looks like two separate attempts, suggesting the first one failed. In reality, it was just one continuous effort. This "draft-to-final" pattern was happening far more often with the AI than with humans, creating a false impression of failure.
Once the researchers corrected for this workflow artifact, the story changed completely. When they removed the cases where the AI was simply refining its own draft, the difference in failure rates vanished. The corrected data showed that the AI was not failing more often than humans; in fact, the gap between the two groups disappeared, with the corrected rates becoming statistically indistinguishable. The initial alarm had been a mirage created by the way the AI wrote its code, not by the durability of the code itself. The study also uncovered a second issue: many of the "fixes" being counted were not actually bug fixes at all. They were tasks like updating documentation or adding new features, which were mistakenly labeled as bug repairs in the data collection process. This inflated the error rate for both groups, but it did not explain why the AI looked worse than humans in the first place.
The researchers went a step further to ensure their findings were solid. They created a new, automated method to detect and remove these draft-to-final pairs across the entire dataset of thousands of defects, without needing human eyes to read every single one. This programmatic correction confirmed their manual findings: the AI's apparent excess of failures dropped from 13.5 percent down to 6.5 percent, while the human rate dropped slightly to 7.4 percent. The statistical significance of the difference disappeared entirely. In a final, more rigorous check, they matched individual AI fixes with human fixes from the same project and time period to ensure they were comparing apples to apples. In this tightly controlled comparison, the data suggested a reversal in the direction of the initial finding, but the researchers explicitly noted this specific result is "hypothesis-generating" rather than "confirmatory." They emphasized that this finding is not yet established and requires further validation before it can be treated as a definitive verdict on durability.
The ultimate lesson from this work is not about whether AI is better or worse than humans at coding, but about how we measure it. The study demonstrates that a headline number derived from an unvalidated automated signal can be completely wrong, capable of reversing the truth entirely. The initial finding that AI fixes were 1.64 times more likely to fail was not just an exaggeration; it was an inversion of reality caused by a blind spot in the measurement tool. The researchers concluded that the AI's fixes are not less durable than human fixes, and they may even be more so, but proving this requires a measurement protocol that accounts for the unique ways AI agents operate. The study provides a corrected method for future research, ensuring that when we compare the durability of AI and human code, we are counting real failures, not just the artifacts of a new kind of workflow.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.