Technical Summary: Benchmarking Quantum Software Testing with Scalable Quantum Programs

Problem Statement

Quantum software testing (QST) aims to verify that quantum programs behave according to their specifications. However, rigorous empirical evaluation in QST is currently hindered by a lack of suitable benchmarks. Existing studies predominantly rely on small, hard-coded, or fixed-size quantum circuits (low-level programs) often written in assembly languages like OpenQASM. These programs lack scalability and do not reflect modern software engineering practices, such as modularity and the use of high-level abstractions found in Software Development Kits (SDKs) like Qiskit.

Furthermore, existing benchmarks are fragmented. Resources like Bugs4Q contain many classical programs within quantum stacks, while MQT Bench was designed for hardware testing using circuits as inputs rather than programs under test (PUTs). This mismatch limits the ability to design valid test cases, interpret results consistently, and conduct fair, reproducible comparisons across different QST approaches. There is a critical need for a benchmark infrastructure that supports scalable, high-level quantum programs with clear functional specifications and standardized interfaces.

Methodology

The authors propose Qolumbina, a benchmark infrastructure designed to support controlled QST experiments on scalable quantum programs. The construction of Qolumbina follows a systematic pipeline:

1. Collection and Filtering: Programs were curated from 8 open-source repositories with real-world provenance (e.g., official Qiskit repositories, artifacts from prior QST studies). Selection criteria (F1–F3) ensured programs were:

   • Scalable in logic and structure (allowing classical arguments to determine circuit size).
   • Accompanied by functional descriptions or algorithmic principles.
   • Implementing unitary operations in noise-free situations (excluding mid-circuit measurements for this initial scope).
     This process yielded 40 distinct programs.

2. Testability Refactoring: To address heterogeneous interfaces and under-specified constraints, 95% of the original programs underwent moderate refactoring. Adaptations included:

   • Structure Reorganization: Unifying input-output ports and converting fixed variables to configurable arguments.
   • Dependency Decoupling: Replacing external dependencies with local equivalents to ensure long-term usability.
   • Input Validation: Inserting assertions to reject invalid inputs.
   • Cross-language Translation: Translating programs from other frameworks (e.g., Q#) into Qiskit/Python.

3. Documentation and Specifications: The authors generated QST-oriented documentation, including:

   • Formula-based specifications: Mathematical definitions of expected outputs and structures.
   • API Documentation: Auto-generated from docstrings, linked to formal specifications.
   • Code Examples: Demonstrating complex test input construction.
   • Technical Debt Marking: Explicitly noting limitations (e.g., unexposed arguments or unreliable oracles for approximate algorithms).

4. Unit Testing: 220 program-level unit tests were developed and executed to validate that refactored programs conform to specifications, achieving 85% statement coverage.

5. Taxonomy and Metrics: The authors established criteria to characterize programs along:

   • Functionality: Application domains and output characteristics (e.g., deterministic, phase-encoded, approximate).
   • Scale: Development complexity (Lines of Code, Cyclomatic Complexity) and execution complexity (circuit width, size, depth).

Key Contributions

1. Qolumbina Infrastructure: A publicly available benchmark containing 40 scalable Qiskit programs that are closer to practical development than the low-abstraction circuits used in prior research.
2. Construction Pipeline: A methodology for transforming open-source quantum programs into test-ready subjects through systematic selection, refactoring, specification, and standardization.
3. QST-Oriented Taxonomies: A set of criteria for characterizing program functionalities and scales, used to validate the diversity and scalability support of the benchmark.
4. Empirical Validation: Two controlled experiments demonstrating the feasibility of using Qolumbina for execution-cost and fault-detection studies.

Empirical Results

The authors conducted an empirical study addressing three Research Questions (RQs):

• RQ1 (Functional Properties):

  • Real-world Provenance: 70% of the programs are non-pedagogical, and 77.5% are reusable as subroutines, indicating they represent modular components of larger workflows rather than just tutorial examples.
  • Output Diversity: The benchmark covers six distinct output categories, including deterministic, exact distribution, phase-encoded, and approximate outputs. This diversity highlights the need for test oracles tailored to specific output semantics, as standard statistical oracles may fail for phase-encoded programs.

• RQ2 (Scale Analysis):

  • Development Complexity: Programs span a wide range of LOC (4–247) and cyclomatic complexity (2–86), with outliers like the HHL algorithm offering complex structural challenges.
  • Execution Complexity: Unlike fixed-size benchmarks, Qolumbina programs generate circuits where width, size, and depth vary significantly based on classical inputs. The study found that circuit complexity can depend not only on qubit-count arguments but also on other functional arguments (e.g., weight vectors in adders), leading to substantial variations in execution cost.

• RQ3 (Experimental Performance):

  • Execution Cost: Using MSTC and DOSS approaches, the study reproduced findings that mixed-state test suites are more efficient than pure-state ones. Crucially, it extended this to noisy simulations, showing that backend choice (Ideal vs. fake backends like Algiers and Brooklyn) significantly impacts execution time, even in noiseless settings, due to transpilation differences.
  • Fault Detection: When using DOSS as a test oracle, reported failure rates varied significantly across different fake backends. Statistical analysis (Mann–Whitney U test) revealed large effect sizes in failure rates between backends, indicating that backend noise and specific hardware models are experimental variables that influence fault detection results, not just peripheral details.

Significance and Claims

The paper claims that Qolumbina addresses the gap between current research benchmarks and practical quantum software development. By providing scalable, high-level programs with standardized interfaces and specifications, it enables fairer and more reproducible evaluations of QST techniques.

The authors emphasize two key implications for future research:

1. Functionality-Aware Testing: Test oracles and designs must align with the specific intended functionality and output characteristics (e.g., phase vs. probability) of the program, rather than applying a one-size-fits-all approach.
2. Backend as an Experimental Variable: The choice of backend (ideal vs. fake/noisy) materially affects both execution costs and fault detection outcomes. Therefore, backend selection should be treated as a core part of the experimental setting in QST studies, not merely a simulation detail.

The study concludes that while Qolumbina supports scalability analysis and reproduces prior findings, it also reveals new phenomena regarding backend-dependent effects that require further investigation. The authors modestly frame their work as an initial assessment and a foundation for future studies, acknowledging limitations such as the exclusion of mid-circuit measurements and the use of mutation-based faults rather than real-world bugs.