Investigating the application of differential fuzzing to support the identification and suppression of equivalent mutants: An experimental study
This experimental study demonstrates that differential fuzzing is a practical, language-agnostic approach for efficiently identifying and suppressing equivalent mutants across diverse real-world software projects, achieving near-perfect mutation scores while generating actionable inputs to strengthen traditional test suites.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Software testing is the process of checking if a computer program works correctly, but finding every possible mistake is nearly impossible because programs are complex and inputs are endless. To tackle this, researchers use a technique called mutation testing, which works by intentionally introducing tiny, realistic errors into the code to see if the existing tests can catch them. If a test fails to spot the error, it means the test suite is not strong enough. However, this method faces a stubborn obstacle: some of these fake errors are so subtle that they do not change the program's behavior at all, making them impossible to detect. These are known as equivalent mutants, and identifying them usually requires a human expert to spend a significant amount of time reading the code line by line, a slow and expensive process that has long held back the widespread use of this testing method.
At the same time, another testing method called fuzzing has become a standard tool for finding security flaws. Fuzzing works by feeding a program massive amounts of random or malformed data to see if it crashes. While effective at finding bugs that cause a program to stop working, traditional fuzzing often misses the kind of subtle errors that change how a program calculates or behaves without causing a crash. A new study by researchers at the University of São Paulo explores a way to combine these two worlds. They investigated a technique called differential fuzzing, which runs the original program and a version with a tiny error side by side, feeding them the exact same data and comparing the results. If the two versions produce different outputs, the error is caught. The researchers wanted to see if this approach could automatically identify those elusive equivalent mutants and do so much faster than a human could.
To test this idea, the team selected six specific functions from four different, well-known open-source software projects written in four different programming languages: C++, C, Go, and Python. These projects included Bitcoin Core, a cryptocurrency protocol; OpenSSL, a cryptography library; LND, a payment channel network; and Arrow, a date and time library. Using a tool that can generate errors for any programming language, they created 1,090 valid variations of these functions. Before the automated testing began, the researchers manually inspected the errors to remove the ones that were obviously equivalent, leaving a set of challenging cases to see if the automated system could distinguish the rest. They then ran five different testing scenarios on these errors, ranging from standard unit tests to time-limited fuzzing sessions that ran for five minutes per error.
The results showed that traditional fuzzing, which only looks for crashes, was the least effective method, catching very few of the errors. In contrast, the differential fuzzing approach proved to be remarkably powerful. When given a five-minute time limit to test each error, this method successfully identified and confirmed the behavior of the errors in five out of the six functions, achieving a success rate between 98 and 100 percent. For the one function where it initially struggled, the researchers found that adding a simple list of expected keywords helped the system understand the input better, eventually allowing it to reach a perfect score as well. The study also revealed that the time required to find these errors was surprisingly short; on average, the system found the differences in about 30 seconds, far faster than the 15 minutes it typically takes a human to analyze a single case manually.
Beyond just finding the errors, the researchers discovered that the data used during the fuzzing process held hidden value. The collection of inputs that the fuzzing tool generated, known as a seed corpus, contained specific test cases that the standard unit tests had missed. These inputs were able to kill mutants that the traditional test suites could not detect. This suggests that the data already being generated by security teams can be reused to strengthen regular test suites, turning a byproduct of security testing into a resource for general software quality. The study also analyzed how long it took to find the hardest-to-detect errors, finding that while most were found quickly, a few required significantly more time and showed a pattern where the difficulty varied greatly, much like how some tasks take a little longer while others take a very long time.
The researchers concluded that differential fuzzing offers a practical, language-agnostic way to support the classification of these difficult errors. It does not require complex new tools or language-specific setups, as it simply compares the original code against the modified version. By treating the surviving errors as candidates for manual review rather than trying to classify every single one, the method drastically reduces the amount of human effort needed. The study suggests that this approach can be integrated into existing workflows to efficiently filter out the errors that are truly equivalent, leaving human experts to focus only on the small number of cases that remain uncertain. This finding indicates that a simple, automated comparison of program behaviors can solve a problem that has long been considered too costly and time-consuming for widespread industrial adoption.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.