From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs
This paper introduces the RegressionBug4APR benchmark to empirically evaluate automated program repair on regression bugs, revealing that while traditional tools fail, LLM-based approaches significantly improve repair success rates when enhanced with context-aware bug-inducing change information.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Software is never truly finished. It is a living thing that grows and shifts to meet new needs, with developers constantly adding features or fixing old problems. But in the rush to improve, a common and frustrating mistake happens: a change that is meant to help accidentally breaks something that was working perfectly before. This is called a regression. Imagine you are fixing a leak in a roof, but in the process, you accidentally knock a hole in the wall. The leak is gone, but now you have a new, bigger problem. In the world of software, these regressions are notoriously difficult to find and fix because they are hidden in the history of code changes, often lurking for years before anyone notices. For decades, researchers have tried to build computer programs that can automatically fix these bugs, hoping to save human developers from endless hours of debugging. However, the tools they built were mostly designed for general errors and struggled when faced with the specific, historical nature of a regression.
A team of researchers from the University of Melbourne and the Singapore University of Technology and Design decided to investigate whether modern artificial intelligence, specifically large language models, could do a better job at this difficult task. These models are advanced computer systems trained on vast amounts of text and code, capable of understanding instructions and generating new content. The researchers wanted to know if these smart systems could not only find the broken code but also understand why it broke by looking at the specific change that caused the problem. To test this, they first had to build a new, high-quality collection of real-world software errors, because the old collections they used were outdated or did not contain the right kind of mistakes. They created a benchmark called RegressionBug4APR, which includes 200 confirmed regression bugs from popular software projects written in Java and Python. They carefully verified each bug to ensure it was a true regression, meaning a feature that worked in an earlier version of the software stopped working after a specific update.
With this new collection in hand, the researchers put various repair tools to the test. First, they tried the traditional automated tools that have been used for years. These tools work by guessing small changes to the code, like swapping a word or removing a line, to see if the error goes away. The results were stark: these traditional tools failed to fix a single one of the 200 bugs. They simply could not handle the complexity of these specific errors. The researchers then turned to the newer, more powerful artificial intelligence models. These models performed much better, with the most advanced ones successfully fixing a significant number of bugs. However, the researchers noticed that the models were still guessing in the dark. They were given the broken code and the error message, but they were not told which specific change in the software's history had caused the break.
To see if giving the models more context would help, the researchers tried a new approach. They fed the artificial intelligence the exact code changes that had introduced the bug, along with the notes the original developer wrote when making that change. This is similar to giving a mechanic not just the broken car, but also the specific wrench they used to tighten the bolt that caused the problem. When the models were given this extra information about the "bug-inducing change," their performance jumped dramatically. The best setup, which used a conversational style where the model could ask for feedback and try again, managed to fix 39 out of the 200 bugs. This was a 1.6 times improvement over the same model without the extra history. The researchers found that this context helped the models understand the root cause of the error, allowing them to decide whether to simply undo the bad change or to make a more subtle correction that kept the good parts of the update while fixing the bad parts.
The study also revealed that not all bugs are created equal. Some errors happen in the exact spot where the code was changed, while others happen far away in parts of the program that were not touched at all. The models found the distant errors much harder to fix, even with the extra history. Furthermore, the researchers analyzed the mistakes the models did make. Sometimes the models guessed the wrong cause of the error, or they created a fix that passed the tests but was logically wrong, essentially tricking the test system rather than solving the real problem. Despite these limitations, the findings are clear: traditional automated repair tools are ineffective for regression bugs, but modern artificial intelligence shows great promise, especially when it is allowed to look at the history of the code to understand how the mistake happened. This suggests that the future of fixing software lies not just in smarter algorithms, but in giving those algorithms the full story of how the software evolved.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.