Beyond Resolved Rate: A Non-Functional Quality Study
This study reveals that while newer AI models resolve more repository-level coding tasks than earlier versions, they do not demonstrate consistent improvements in non-functional quality metrics such as static analysis, code complexity, or resource usage on the tasks both generations successfully solve.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where computers have learned to write code, acting like tireless junior programmers who can fix bugs, build features, and even refactor entire software projects. This is the realm of Large Language Models (LLMs) in software engineering. For a long time, the only way we judged these digital coders was by asking a simple question: "Did they fix the bug?" If the code passed the tests, it got a gold star. This is called "functional correctness." But just because a car engine starts doesn't mean the brakes work, the paint is durable, or the fuel efficiency is good. In the real world, software needs to be secure, easy to update later, and fast enough not to crash your computer. These are "non-functional qualities." The big question researchers are asking now is: As these AI models get smarter and newer, are they just getting better at fixing the immediate problem, or are they also writing cleaner, safer, and more efficient code?
This paper, titled "Beyond Resolved Rate," dives into that exact mystery. The authors, a team from Linköping University in Sweden, decided to stop just counting how many bugs the AI fixed and start inspecting how it fixed them. They treated the AI models like contestants in a cooking competition. The judges (the researchers) gave them a specific task: fix a broken recipe (a bug in a software project). The older models were the veterans, and the newer models were the up-and-coming stars. The goal wasn't just to see who could serve a dish that tasted right (passed the tests), but to see if the newer chefs were using better ingredients, creating less waste, and making the kitchen safer for the next cook.
The researchers set up a rigorous experiment using a popular benchmark called SWE-bench Lite, which contains real-world software repair tasks. They pitted two generations of models against each other from two different families: the commercial "Claude" family and the open-source "DeepSeek" family. They took the patches (the code fixes) generated by the older and newer models and ran them through a battery of high-tech inspections. They used tools like CodeQL and CodeScene to scan for security risks, messy code structures, and maintainability issues. They also timed how long the code took to run and measured how much memory it gobbled up, treating the computer like a hungry beast that needs to be fed efficiently.
The results were a bit of a plot twist. The newer, "smarter" models definitely fixed more bugs. They solved more instances than their older siblings, earning them a higher "resolved rate." However, when the researchers looked at the quality of the code for the tasks both models managed to solve, the story changed. The newer models didn't show any consistent improvement in the non-functional qualities. In fact, the data suggested that the newer models were just as likely to introduce code smells, security risks, or performance hiccups as the older ones.
Specifically, the study found that for the tasks both models solved, the newer models didn't produce significantly cleaner code. The static analysis tools showed that the number of new problems introduced was roughly the same for both generations. In terms of performance, the newer models were slightly more "greedy" with resources. On the common tasks, the newer Claude model used about 0.048 seconds more CPU time and roughly 4.5 MiB more peak memory than the older one. The newer DeepSeek model used about 0.5 MiB more memory. While these numbers are small, they indicate that getting better at fixing the bug didn't automatically mean getting better at writing efficient code.
The authors also looked at the "flavor" of the code. They checked if the newer models were avoiding specific bad habits, like confusing variable names or messy imports. The results were mixed and inconsistent; sometimes the newer model was better at a specific rule, sometimes the older one was, but there was no clear trend that the newer generation was universally superior in code quality.
Ultimately, the paper suggests that while AI models are getting better at the "what" (fixing the bug), they aren't necessarily getting better at the "how" (writing high-quality, maintainable code) just by virtue of being newer. The authors caution that a higher success rate in fixing bugs doesn't guarantee better software engineering overall. They argue that we need to look beyond the simple pass/fail score and start measuring the hidden costs of AI-generated code, such as security risks and maintenance headaches, to truly understand how these digital assistants are performing in the real world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.