A Methodological Analysis of Empirical Studies in Quantum Software Testing
This paper presents a methodological analysis of 59 empirical studies in quantum software testing to characterize current practices, identify inconsistencies and limitations, and provide recommendations for improving the design and reporting of future research in the field.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the emerging field of quantum computing, scientists are building machines that process information in ways fundamentally different from the laptops and servers we use every day. Instead of bits that are strictly zero or one, these machines use quantum bits, or qubits, which can exist in a delicate blend of states simultaneously. This unique behavior allows them to solve certain complex problems with incredible speed, but it also makes the software they run notoriously difficult to verify. Unlike classical software, where you can simply run a program and check if the answer is right, quantum programs produce results that are probabilistic. When you measure a quantum system, the act of observation itself changes the state, collapsing a range of possibilities into a single outcome. This means that to know if a quantum program is working correctly, researchers must run it many times and analyze the statistical patterns of the results, rather than looking for a single definitive pass or fail. As these quantum systems grow larger and more complex, ensuring they function as intended has become a critical challenge for the entire field.
A team of researchers from Beihang University and Kyushu University recently set out to understand how the scientific community is currently tackling this challenge. They conducted a comprehensive review of 59 empirical studies published between 2018 and 2025, examining how scientists design and report their experiments to test quantum software. Their goal was not to judge which specific testing method was the best, but rather to map out the landscape of how these studies are constructed. They found that while the field is growing rapidly, there is a lack of shared standards. Researchers are using vastly different methods, tools, and criteria, making it difficult to compare results or build upon previous work. The team organized their analysis around ten key questions, covering everything from the types of programs being tested to the computer hardware used to run the experiments.
One of the most striking findings was the variety of quantum programs used as test subjects. The researchers discovered that studies employed 92 different types of quantum algorithms and subroutines. Among these, the Quantum Fourier Transform was the most frequently used, appearing in 29 of the studies, followed by well-known algorithms like Grover Search and Quantum Phase Estimation. However, the review also highlighted a significant gap: very few studies tested quantum machine learning models, which are increasingly important for real-world applications. Furthermore, when researchers introduced bugs to test their detection methods, they relied heavily on artificial mutations—small, systematic changes to the code—rather than using real-world errors found in actual software projects. Only a handful of studies utilized benchmarks containing real-world bugs, suggesting that current testing practices might not fully reflect the challenges developers face in industry.
The way researchers design their test inputs and evaluate the results also showed considerable inconsistency. Most studies used initial quantum states as their primary input, often starting with simple, standard configurations. However, the methods for determining whether a test passed or failed varied widely. Because quantum measurements are probabilistic, researchers must decide how many times to run a program to get a reliable answer. The review found that while many studies used statistical methods to compare their results against expected outcomes, there was no consensus on the best approach. Some relied on comparing probability distributions, while others focused on specific dominant outcomes or the overall properties of the system. This lack of uniformity in how results are analyzed makes it hard to determine if one testing approach is truly more effective than another.
The hardware used to run these experiments was another area of divergence. The majority of studies relied on ideal simulators—classical computers that model how a quantum system should behave in a perfect, noise-free environment. Only a small number of studies used noisy simulators or actual quantum hardware, which are more representative of the current state of technology but introduce significant complexity and cost. The researchers noted that while simulators are useful for early-stage development, relying on them exclusively may not prepare the field for the realities of testing on physical quantum devices, where environmental noise and hardware limitations play a major role. Additionally, the scale of the circuits tested was often quite small, with the median number of qubits in the most complex circuits being around 10, though some studies did explore circuits with up to 20 qubits.
To address these inconsistencies, the authors proposed a set of actionable recommendations for future research. They suggested that studies should include a more diverse range of programs, including real-world benchmarks and quantum machine learning models, to better reflect practical applications. They also emphasized the need for clearer reporting on how test cases are designed, how many times experiments are repeated, and what specific hardware or simulation tools are used. By adopting more standardized practices, the community can improve the reliability and comparability of their findings. The researchers also highlighted the importance of making their tools and data publicly available, noting that while over half of the studies they reviewed shared their artifacts, there is still room for improvement in transparency and reproducibility.
Ultimately, this analysis serves as a roadmap for the maturation of quantum software testing. It reveals that while the field is vibrant and growing, it is still in its early stages, characterized by a diversity of methods that has yet to coalesce into a unified framework. By identifying these gaps and inconsistencies, the study provides a foundation for future work to become more rigorous, comparable, and aligned with the practical needs of the quantum software industry. The path forward involves not just developing new testing techniques, but also agreeing on how to measure their success, ensuring that the promise of quantum computing can be realized with software that is both powerful and trustworthy.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.