SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
This paper introduces SWE-NFI, a benchmark comprising 188 real-world tasks and 92 executable rules designed to evaluate coding agents on non-functional improvements, revealing that while agents achieve high functional correctness, they significantly lag behind human developers in executing behavior-preserving code enhancements.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a robot to write code. For a long time, the only way to test if the robot was doing a good job was to ask, "Does the program work?" If the robot built a calculator that added numbers correctly, it got a gold star. But in the real world, writing software is like building a house. Just because the roof doesn't leak and the doors open doesn't mean the house is a joy to live in. Maybe the wiring is a tangled mess behind the walls, the paint is peeling, or the instructions for how to use the light switch are written in invisible ink. These are the "non-functional" parts of a house: how easy it is to fix, how safe it is, and how nice it looks.
Recently, smart computer programs called "coding agents" have gotten very good at building the basic structure of software. They can fix bugs and add new features. But nobody really knew if these agents could also do the messy, boring, but super important work of cleaning up the code, organizing the wiring, and writing clear instructions without breaking anything that was already working. This is the question a team of researchers from universities in Canada, China, and Singapore decided to answer. They wanted to see if these digital helpers could be more than just functional builders and become true software caretakers.
To find out, the researchers created a new test called SWE-NFI. Think of this test as a giant, automated inspector's checklist. Instead of just checking if a program runs, the checklist has 92 specific rules about how "good" the code looks and feels. These rules check things like: "Did you write a clear explanation for what this function does?" (Documentation), "Did you handle errors safely so the program doesn't crash?" (Error Handling), and "Did you use modern tools instead of old, rusty ones?" (Library Constraints).
The researchers built this test using 188 real-life examples taken from actual software projects that humans had already fixed. They took the "before" version of the code, gave it to various coding agents, and asked them to make it better without changing what it actually did. Then, they used their 92-rule checklist to grade the agents. They also had a "human reference" score, which is how well a real human developer improved that same piece of code in the real world.
The results were a bit of a reality check. The best coding agents were actually quite good at making sure the code still worked (getting a 70.0% pass rate on functional correctness). However, when it came to making the code better in the ways humans care about, the agents struggled. They fell short of human developers in almost every category.
The biggest gap was in "Logic Patterns," which is like the structural integrity of the code. Humans improved this area by an average score of 1.5, but the best coding agents only managed between 0.0 and 1.3. It's as if the robots could build a sturdy wall, but they couldn't figure out how to make the bricks line up perfectly or how to add a nice archway.
The study also found some interesting quirks:
- One file vs. Many: The agents were okay at fixing a single file, but when they had to coordinate changes across multiple files at once (like fixing the wiring in the kitchen and the living room at the same time), they got much worse.
- Cost vs. Quality: The researchers checked how much "fuel" (computer time and data tokens) the agents used. They found that spending more money or time didn't always mean better results. One agent spent 100 times more resources than another but didn't produce significantly better code improvements.
- Consistency: Once an agent managed to write code that actually worked, it was pretty consistent at making those small improvements. The real problem was just getting the code to work in the first place.
In short, the paper suggests that while our coding robots are getting very good at the "what" (making things work), they still have a long way to go on the "how" (making things maintainable, safe, and clean). They are great apprentices who can follow instructions, but they haven't quite learned the art of being a master craftsman who knows how to tidy up the workshop. The researchers hope that by having this new, strict checklist, we can teach the next generation of coding agents to not just build software, but to care for it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.