💻 computer science

Auditing Empirical Comparisons in Quantum Software

This paper introduces CLAIMSTAB-QC, a framework for auditing empirical comparisons in quantum software by locking study designs before outcome computation, which reveals a significant materialization gap where most reported claims lack sufficient evidence for direct verification and often yield unresolved or reversed results under strict scrutiny.

Original authors: Boshuai Ye, Peng Liang, Maryam Tavassoli Sabzevari, Arif Ali Khan

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Boshuai Ye, Peng Liang, Maryam Tavassoli Sabzevari, Arif Ali Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are reading a food review that says, "Chef A's burger is tastier than Chef B's." Usually, we assume this is a universal truth about the burgers. But what if Chef A used a secret spice blend, a specific type of bun, and a grill set to a precise temperature, while Chef B used a different bun and a charcoal grill? If you try to taste-test them yourself using your own kitchen tools, you might find that Chef B's burger actually wins.

This is the problem the paper "Auditing Empirical Comparisons in Quantum Software" tackles, but instead of burgers, it's about quantum computers and the software that runs them.

Here is a simple breakdown of what the authors did, using everyday analogies.

1. The Problem: The "Apples vs. Oranges" Trap

In the world of quantum software, researchers often publish papers claiming, "Our tool (Tool A) is faster/better than that tool (Tool B)."

However, quantum software is like a giant, multi-layered sandwich. To make a sandwich, you need bread, filling, sauce, and a specific way of slicing it. In quantum software, these layers are:

The code (the bread).
The compiler (the slicer).
The simulator or hardware (the plate).
The noise and errors (the crumbs).

The authors argue that saying "Tool A is better" is often misleading because the result depends entirely on how the sandwich was made. If you change the bread (the circuit) or the slicer (the compiler settings), Tool A might suddenly look worse than Tool B.

2. The Solution: The "Strict Inspector" (CLAIMSTAB-QC)

The authors built a new framework called CLAIMSTAB-QC. Think of this as a strict food inspector who doesn't just taste the food; they check the recipe card first.

Here is how their "inspection" works:

The Claim Card: When a paper says "A beats B," the inspector writes down exactly what was claimed: the specific ingredients, the specific tools, and the specific rules used.
The Lock: Before the inspector tastes anything, they lock the recipe. They are not allowed to change the ingredients or the tools. They must use exactly what the original paper said.
The Evidence Check: The inspector looks at the paper's "receipt" (the data and code provided).
- Scenario A: The paper provided the exact receipt. The inspector can taste the burger exactly as described.
- Scenario B: The paper said "A is better" but didn't list the ingredients or the temperature. The inspector cannot taste it. They must stop and say, "We cannot verify this claim because the evidence is missing."

3. The Big Discovery: The "Missing Receipt" Gap

The authors tested this framework on 455 claims from 119 different research papers. The results were surprising:

175 claims could be written down as a clear recipe (Claim Cards).
79 claims looked like they could be tested.
53 claims had enough data to set up a test.
BUT... only 8 claims had the complete "receipt" needed to test the claim without guessing or making up missing data.

The Analogy: Imagine a restaurant chain claims their burgers are the best in the city. They hand you a list of 100 locations. You go to 53 of them to check. But when you try to taste the burger, you realize 45 of them didn't tell you what ingredients they used. You can only actually taste and verify the burger at 8 locations.

This is called the "Materialization Gap." Researchers often report the result (the winner) without providing the evidence (the exact settings) needed to prove it.

4. The Results: Who Actually Won?

For the 8 claims that had complete evidence, the authors ran the "strict audit":

2 claims: The original winner was confirmed (The "Sustained" verdict).
4 claims: It was impossible to tell who won because the data was too mixed or the results were too close (The "Unresolved" verdict).
2 claims: The original winner actually lost when tested strictly (The "Reversed" verdict).

The "Reversed" Example: One paper claimed Tool A produced fewer errors than Tool B. When the authors locked the settings and re-ran the test exactly as described, they found that Tool A actually produced more errors. The original claim was only true because of a specific, unreported setting that the authors didn't lock down.

5. The Lesson: "Show Your Work"

The paper concludes that the current way of reporting quantum software comparisons is broken. It's like a math teacher saying, "The answer is 5," but not showing the steps.

The authors suggest that future papers should:

State the comparison clearly.
Provide the exact "receipt" (the specific settings, seeds, and data) needed to lock the test.
Clearly admit where the evidence stops (e.g., "We only tested this on small circuits; we don't know if it works on big ones").

Summary

The paper isn't saying quantum software is bad. It's saying that claims about which software is "better" are often unprovable because the researchers don't share enough details about how they ran the tests.

They built a tool (CLAIMSTAB-QC) to act as a strict auditor. When they used it, they found that most claims couldn't be audited because the "receipts" were missing. For the few that could be audited, the results were mixed: sometimes the original claim held up, sometimes it didn't, and often, it was impossible to tell.

The takeaway: If you want to know if Tool A is truly better than Tool B, you need to see the full recipe, not just the final taste.

Technical Summary: Auditing Empirical Comparisons in Quantum Software

Problem Statement
Empirical papers in Quantum Software Engineering (QSE) frequently report comparative claims (e.g., "Compiler A produces smaller circuits than Compiler B"). However, these "A beats B" conclusions are not intrinsic properties of the tools alone; they are emergent outcomes of a complex stack involving circuit construction, compilation choices, sampling strategies, backend noise assumptions, optimizer stochasticity, and resource budgets. While existing methods address reproducibility (rerunning experiments) and tool testing, they do not directly audit whether the reported comparison itself is supported by the specific evidence exposed in the source paper and artifact.

The core challenge is the materialization gap: a paper may report a clear ordering, but the source often lacks the matched evidence (paired baselines, identical settings, and locked outcome rules) required to audit that specific relation without proxy reconstruction or introducing unsupported assumptions. Consequently, many reported comparisons remain unauditable despite the availability of executable artifacts.

Methodology: CLAIMSTAB-QC Framework
The authors present CLAIMSTAB-QC, a source-bounded framework designed to audit empirical comparisons by treating the reported relation as the audit unit. The framework operates through the following stages:

Claim Card Representation: A reported comparison is formalized as a claim card $C = \langle P, BA, BB, M, R \rangle$ , where $P$ is the problem context, $BA/BB$ are baselines, $M$ is the metric, and $R$ is the reported relation. This fixes the scope before any outcome is computed.
Materialization and Locking: The framework attempts to "materialize" the claim by identifying source-supported evidence. It defines an Audit Space ( $S_{audit} = S_{scope} \times E_{env}$ $S_{a u d i t} = S_{sco p e} \times E_{e n v}$ ) comprising six axis groups: Scope (problem size, instances), L1 (construction/compilation), L2 (sampling/estimation), L3 (backend/noise), ALGO (optimizer stochasticity), and Budget (resource limits).
- A Dry-Run-First procedure inspects these axes. Only variations that are source-supported, metric-affecting, and relation-preserving are activated.
- The design is locked before outcomes are computed. If the source lacks the necessary paired data (e.g., matched circuits under identical seeds), the audit stops at an Evidence Boundary.
Outcome Computation:
- For Scalar-Directional records (strict "better/worse" claims), the framework computes a signed margin for each matched cell.
- It applies a Wilson decision rule to classify the relation as Sustained (lower bound $\ge \tau$ ), Reversed (upper bound $\le 1-\tau$ ), or Unresolved. The threshold $\tau$ is set to 0.95 for strict claims.
- Non-scalar records (aggregate, crossover) use source-supported rules if lockable; otherwise, they are marked as boundaries.
Reporting: The output includes the scoped verdict (S/U/R) or an explicit evidence boundary explaining why the comparison could not be audited (e.g., missing timing protocols, unmatched seeds).

Key Contributions

Framework: A novel QSE framework that shifts the audit focus from "can we reproduce the result?" to "does the source evidence support the reported relation?" using source-bounded records and locked designs.
Empirical Study: An application of the framework to 455 comparative claims from 119 quantum-software papers (2020–2024).
Reusable Artifact: A released package containing claim representations, audit records, locked traces, and missing-evidence annotations to facilitate inspection of empirical QSE comparisons.

Results
The study reveals a significant materialization funnel and gap:

Representation: Of 455 accepted claims, 175 could be represented as claim cards.
Planning: 79 were identified as scalar-directional planning records.
Lockability: 53 yielded lockable audit or diagnostic designs.
Proxy-Free Auditing: Only 8 records exposed enough matched evidence to audit the original comparison without proxy reconstruction (Tier-1).
- Of these 8, 2 were Sustained, 4 were Unresolved, and 2 were Reversed.
- Notably, two "Reversed" cases (EX-C7 and EX-C8) showed that the reported direction was contradicted in 0/80 and 0/80 locked cells, respectively, driven by strict contradictions rather than ties.
Controlled Diagnostics: Experiments on 24 benchmark-relevant comparisons showed that simpler audit protocols (single configuration or single-axis) often preserved apparent directions that were weakened or became Unresolved when the full stack of locked axes (L1–L3, ALGO, BUDGET) was inspected.
Robustness: Verdicts were found to be robust to statistical choices (thresholds, grouping) when evidence was sufficient, but sensitivity to tie-handling and thresholds was explicitly reported as a boundary condition.

Significance and Claims
The paper argues that empirical QSE comparisons are often reported as outcomes without the necessary evidence to audit the comparison itself. The primary significance of this work is the demonstration that performance orderings and their supporting evidence are separable.

The authors conclude that for empirical QSE to advance, sources must report not just who wins, but the matched evidence required to lock the scope of that ordering. The framework provides a mechanism to distinguish between a comparison that is auditable (Sustained/Reversed/Unresolved) and one that is fundamentally unauditable due to missing evidence (Boundary). This shifts the reporting target for future QSE studies: papers should explicitly expose the baselines, settings, instances, budgets, and outcome rules needed to lock the comparison, and clearly mark where evidence boundaries prevent further auditing.