Decided or Open? Comparing Program Repair Techniques When Patch Correctness Is Unknown
This paper proposes a "decided or open" framework for comparing automated program repair techniques under uncertainty by computing the full range of possible success rate differences, enabling researchers to identify which comparisons can be settled with further testing and revealing significant discrepancies in the reliability of archived test results across different models.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of software, programs inevitably break. When a piece of code fails to do what it is supposed to do, a developer must find the error and write a new version of the code to fix it. This process is slow and difficult, so researchers have spent decades trying to build computer programs that can do the fixing automatically. These automated tools scan a broken program, generate thousands of possible solutions, and try to pick the one that works. To know if these tools are getting better, scientists compare them against each other. They run the tools on a set of known broken programs and count how many times each tool produces a correct fix. But there is a hidden problem in this comparison: often, the scientists do not actually know if a generated fix is correct. A fix might pass the tests the computer runs, but that does not guarantee it is right; it might just be a clever lie that happens to work for the specific questions asked. Without a human checking the work, many fixes remain in a gray area, labeled neither right nor wrong.
This uncertainty makes it impossible to say with certainty which automated repair tool is truly the best. If two tools are compared, and the result depends on a handful of unverified fixes, the winner might just be a fluke. A new study by Sadnan Nafis at BRAC University tackles this problem head-on. Instead of trying to guess the answer for every single unverified fix, the researcher developed a way to measure the range of possible outcomes. He asked a simple but powerful question: given the information we already have, is the winner of a comparison fixed, or does it depend entirely on the unknown fixes? By calculating the best-case and worst-case scenarios for every possible value of the missing information, the study determines whether a comparison is "decided" or "open." A decided comparison means that no matter how the unknown fixes turn out, the same tool wins. An open comparison means the winner could flip depending on the missing data.
The study applied this method to a massive archive of repair attempts involving nearly 55,000 candidate fixes for 488 different bugs. The researchers found that for most of the comparisons between a smart, learning-based tool and its simpler rivals, the answer was already decided by the data they had. In twelve out of sixteen cases, the learning tool was clearly better, and this conclusion held true even if every single unknown fix turned out to be wrong. However, four comparisons remained open, meaning the winner depended on the unverified fixes. The researchers then added more evidence, running the code again and asking humans to review the tricky cases. This extra work settled all sixteen comparisons, confirming that the learning tool was indeed the winner in every instance. But the study also revealed that these final four victories were fragile; if just a few of the new labels were wrong, the results could have changed.
One of the most surprising discoveries came from checking the reliability of the testing system itself. The researchers noticed that sometimes the testing system failed to run correctly, marking a correct fix as a failure. To catch this, they looked at cases where the computer generated a perfect copy of the human developer's original fix. Since this copy was known to be correct, it should have passed every test. If the system marked it as a failure, the system was broken. They found that this happened at wildly different rates depending on which artificial intelligence model was being tested. For one popular model, the system incorrectly rejected the correct copy in 33.5% of cases. For another model, it only happened 2.5% of the time. This meant that the testing system was unfairly penalizing one model over the other, not because the model was worse, but because the test harness was more likely to make mistakes with it.
The study also introduced a way to predict the value of future experiments before they even begin. Since running a test can only prove a fix is wrong, but never prove it is right, the researchers realized that some comparisons can never be settled by testing alone. If the best possible outcome of a test campaign is a tie, then running more tests is a waste of time. They created a "pre-flight check" that looks at the current data and tells researchers exactly which comparisons a new round of testing could possibly solve. In a real-world trial, they published their predictions for an upcoming test campaign in advance. Every single prediction held true: the check correctly identified which comparisons would be settled and which would remain open, saving researchers from spending hours on experiments that could never change the outcome.
The findings suggest that the field of automated repair needs to change how it reports results. Instead of giving a single number that pretends all the data is known, researchers should report the range of possibilities. This approach separates the solid conclusions from the shaky ones. It shows that while some tools are clearly superior, others are too close to call without more evidence. It also highlights that the tools used to measure success are not perfect and can introduce bias that favors one model over another. By using these new checks, scientists can focus their efforts on the comparisons that actually need more data, rather than chasing results that are already settled or impossible to determine. The work does not solve the problem of finding the perfect fix, but it provides a clear map of where the truth lies and where it remains hidden.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.