QMutBench: A Dataset of Quantum Circuit Mutants

1. Problem Statement

The field of quantum software testing is rapidly evolving, with numerous techniques emerging to automate the testing of quantum programs. However, a critical bottleneck exists: the lack of systematic benchmarks containing faulty quantum programs.

• Scarcity of Real Faults: Real-world faulty quantum programs are scarce, making it difficult to empirically assess the effectiveness of new testing techniques.
• Limitations of Existing Mutation Tools: While mutation analysis (introducing artificial faults to create "mutants") is a standard solution in classical software testing, existing quantum mutation tools (e.g., Muskit) generate massive numbers of mutants without an automated, systematic method to select a representative subset.
• Usability Issues: Previous empirical studies generated large datasets (e.g., over 700,000 mutants) but lacked user-friendly interfaces. Researchers found it impractical to retrieve specific subsets of mutants based on desired characteristics (e.g., difficulty level or specific gate types) without downloading and parsing the entire massive dataset.

2. Methodology

The authors developed QMutBench, a dataset and online interface designed to bridge the gap between raw mutation data and usable benchmarks.

• Data Generation Source:
  • The dataset is built upon an extensive empirical evaluation involving 375 original quantum circuits and 723,079 mutants.
  • Mutants were generated using Muskit, a specialized quantum mutation tool.
  • Three mutation operators were applied across all supported gates and positions: Add (insert a gate), Remove (delete a gate), and Replace (substitute a gate).
• Key Metric: Survival Rate:
  • Instead of using the traditional "mutation score" (percentage of mutants detected), the authors utilize Survival Rate (percentage of mutants not detected).
  • This metric quantifies the difficulty of detecting a fault. It is calculated for groups of mutants sharing specific characteristics (algorithm, qubit count, mutation operator, gate type, and position).
  • The dataset is stratified by survival rate ranges (e.g., 0–10% for easy-to-detect mutants, 90–100% for hard-to-detect/equivalent mutants).
• Interface and Filtering System:
  • An online interface was developed to allow users to filter the dataset dynamically.
  • Selection Criteria: Users can filter by:
    • Original Circuits: Specific quantum algorithms (e.g., Grover, Amplitude Estimation) categorized by output type (dominant vs. non-dominant).
    • Qubit Range: Circuits ranging from 2 to 30 qubits.
    • Survival Rate: Desired difficulty levels for the benchmark.
    • Mutation Characteristics: Specific gates affected, mutation position (as a percentage of circuit depth), and the operator used (Add/Remove/Replace).
• Data Format:
  • All circuits (original and mutated) are provided in OpenQASM format, the de facto standard for quantum circuits, ensuring compatibility with major quantum computing platforms (e.g., IBM Qiskit, Cirq).

3. Key Contributions

1. QMutBench Dataset: A publicly accessible repository containing over 700,000 quantum circuit mutants derived from 375 original circuits.
2. Interactive Selection Interface: A web-based tool that allows researchers to generate custom benchmarks by filtering mutants based on specific technical criteria (survival rate, gate type, algorithm, etc.) without needing to process the raw data manually.
3. Standardized Output: The provision of data in OpenQASM (.qasm) files, organized into a structured .zip format with clear naming conventions (indicating operator, gate, and position), facilitating immediate integration into testing pipelines.
4. Empirical Foundation: The dataset is grounded in a prior extensive study [9] that analyzed the relationship between circuit characteristics and mutant survival rates, providing a scientifically validated basis for benchmark selection.

4. Results and Data Characteristics

• Dataset Composition: The distribution analysis (Fig. 2 in the paper) reveals that the dataset contains a high concentration of mutants with extreme survival rates:
  • 0–10% Survival: Mutants that are very easy to detect.
  • 90–100% Survival: Mutants that are very hard to detect or are equivalent (functionally identical to the original).
  • The dataset also includes a representative number of mutants across intermediate survival rates.
• Usability: The interface successfully enables the retrieval of specific subsets. For example, a user can request "Grover's algorithm with 10 qubits, modified by a 'Remove' operator on a CNOT gate, with a survival rate between 40–50%."
• Download Structure: The downloaded files are organized into Original_programs and Mutated_programs folders, with subfolders for specific algorithms and qubit counts, ensuring traceability.

5. Significance and Future Impact

QMutBench addresses a critical gap in quantum software engineering by providing the necessary infrastructure to evaluate testing techniques systematically.

• Evaluation of Testing Techniques: Researchers can use the benchmarks to measure the mutation score of their testing tools, determining how effectively their tests detect specific types of quantum faults.
• Development of New Techniques: The dataset enables the creation of mutation-guided testing techniques, where test generation is optimized based on the difficulty (survival rate) of specific mutants.
• Debugging and Repair: The dataset serves as a benchmark for evaluating quantum debugging and repair tools, assessing their ability to locate and fix faults in complex circuits.
• Extensibility: The system is designed to be extended. Future work aims to integrate AI-driven recommendation systems (e.g., Large Language Models) to suggest mutations for new circuits not currently in the dataset and to allow users to contribute new circuits directly to the repository.
• Noisy Environment Research: The dataset opens avenues for studying how mutant survival rates change when executed on noisy simulators and real quantum hardware, a crucial step toward practical quantum error correction and testing.

In conclusion, QMutBench transforms a massive, unwieldy collection of mutation data into a structured, accessible, and customizable resource, significantly lowering the barrier to entry for rigorous quantum software testing research.