Claim against Measurement: Statistical Artefacts in Quantum Error Mitigation Benchmarks

This paper critically evaluates 81 recent Quantum Error Mitigation (QEM) studies, revealing that widespread statistical shortcomings and unaccounted experimental variables often create misleading benchmarks, and consequently proposes rigorous reporting standards to ensure the validity of QEM performance claims.

The Gist

Imagine you are trying to bake the perfect cake to prove that a new, fancy ingredient (let's call it "Quantum Error Mitigation" or QEM) makes cakes taste better. You want to show the world that your cake is superior to a normal one.

This paper is like a group of food critics who decided to taste-test 81 different recipes claiming to use this new ingredient. They didn't just taste the cakes; they looked at the cookbooks to see how the bakers measured their success.

Here is what they found, explained simply:

1. The "Cookbook" Problem: Not Enough Proof

The critics looked at 81 recent papers (recipes) about this quantum baking technique. They found a major problem: Most bakers were just describing how good the cake looked, rather than proving it statistically.

• The Reality: Only 25% of the bakers used proper statistical tests (like a rigorous taste-test panel with a control group) to prove their cake was actually better.
• The Rest: The other 75% just said, "It tasted better," or showed a graph with error bars, but didn't do the math to prove the difference wasn't just a fluke. It's like saying, "My cake is better," without actually comparing it to the others in a fair way.

2. The "Secret Recipe" Trap: Hidden Ingredients Matter

The authors then tried to bake the same cakes again, but they changed the "hidden" settings that the original bakers didn't write down. They discovered that these hidden choices were active, meaning they completely changed the outcome.

• The Analogy: Imagine a recipe says, "Add sugar." It doesn't say how much.
  • If you add 1 cup, the cake is delicious (a "significant improvement").
  • If you add 5 cups, the cake is a salty, inedible mess (a "significant degradation").
• The Finding: In their study, they changed hidden settings like the "scale factors" (how much they stretched the noise) and the "extrapolation method" (how they guessed the perfect result).
  • In 12% of their test cases, changing these hidden settings turned a "winning" result into a "losing" result.
  • Sometimes, the technique actually made the result worse than doing nothing, but the original paper claimed it was better because they happened to pick the "lucky" settings.

3. The "Wobbly Table" Problem: Time Changes Everything

The second major issue is that quantum computers are like wobbly tables. They drift over time.

• The Analogy: Imagine you are trying to balance a stack of plates on a table.
  • If you try at 9:00 AM, the table is steady, and you balance 10 plates.
  • If you try at 1:00 PM, the table has shifted slightly due to temperature or wear. Now, you can only balance 3 plates.
  • If you try again at 5:00 PM, the table shifts back, and you can balance 9 plates.
• The Finding: The authors ran the exact same experiment over 72 hours (3 days).
  • They found that just by changing the time of day, the "effectiveness" of the technique changed by 3.4 times.
  • One morning, the technique looked amazing. Twelve hours later, it looked mediocre.
  • This created an "Effectiveness Illusion." It looked like the technique was working great, but it was actually just a lucky moment in time.
  • Worse, because the table was wobbly, the 30 times they ran the test didn't count as 30 independent tests. Statistically, it was only as good as 1.8 tests. This makes their "proof" much weaker than they thought.

The Big Conclusion

The authors are not saying that Quantum Error Mitigation is a bad idea or that it doesn't work. They are saying that the way we are currently testing and reporting it is flawed.

Because researchers are:

1. Not using strict statistical math.
2. Hiding their "secret recipe" settings.
3. Ignoring the fact that the hardware drifts over time.

...we might be celebrating "breakthroughs" that are actually just lucky accidents or statistical tricks.

What they propose:
They want a new "Minimum Reporting Standard" for quantum baking. Before you claim your cake is better, you must:

• List every single setting you used (no hidden ingredients).
• Run the test at different times to make sure the table isn't wobbly.
• Use proper statistical math to prove the difference is real, not just a fluke.

In short: The technique might be great, but our current measuring tape is broken. We need to fix the measuring tape before we can trust the results.

Technical Summary

Technical Summary: "Claim against Measurement: Statistical Artefacts in Quantum Error Mitigation Benchmarks"

Problem Statement
Quantum Error Mitigation (QEM) is positioned as a critical bridge between Noisy Intermediate Scale Quantum (NISQ) devices and future Fault Tolerant Quantum Computers (FTQC). However, the empirical evaluation of QEM techniques often lacks rigorous statistical foundations. Current literature frequently relies on descriptive reporting rather than inferential statistics, potentially leading to conclusions that are not statistically supported. Furthermore, QEM benchmarks often fail to account for two compounding sources of artefacts: the sensitivity of results to implicitly assumed parameters (e.g., scale factors, extrapolation methods) and the temporal drift of hardware calibration. These omissions risk conflating genuine mitigation effects with statistical noise or experimental artefacts, thereby overstating the robustness and effectiveness of QEM methods.

Methodology
The authors employ a mixed-method approach combining a systematic literature review with two empirical case studies:

1. Systematic Review: The authors analyzed 81 recent QEM papers (2022–2026) using an eight-criterion framework. The criteria assessed sample size justification, variance reporting, inferential statistical evidence, drift control, overhead quantification, noise model validation, reproducibility, and reporting of negative results.
2. Parameter Space Replication (Case Study 1): Using the Zero-Noise Extrapolation (ZNE) technique with Richardson extrapolation as a representative case, the authors replicated a study by Khan et al. (2024). They formalized the "reproduction parameter space" ($P$) into categories: Hardware/Backend ($H$), Circuit ($C$), Shots & Reps ($Q$), Folding ($F$), Extrapolation ($E$), and Scale Factors ($S$). They systematically swept through 132 configurations by varying unspecified parameters (e.g., scale factors $\{1, 3, 5\}$ vs. $\{1, 1.5, \dots, 3\}$, extrapolation methods, and calibration snapshots) while holding others constant. Statistical significance was assessed using paired t-tests and effect sizes (Cohen's $d$ and Cliff's $\delta$).
3. Longitudinal Drift Study (Case Study 2): To isolate the impact of temporal drift, the authors conducted a 72-hour longitudinal experiment on the 54-qubit IQM Euro-Q-Exa system. They executed the same ZNE configuration at 30-minute intervals over three sessions (two 12-hour days and one 48-hour weekend). They analyzed the autocorrelation of raw expectation values and the variation in ZNE effect sizes ($d$) over time.

Key Contributions

• Systematic Review Findings: The review reveals a significant gap in statistical rigor. Of the 59 papers where statistical evidence was applicable, only 15 (25%) used inferential methods (e.g., hypothesis testing). The majority (42%) reported uncertainty descriptively without testing for statistical significance, and 32% provided no statistical evidence at all. Drift control was addressed in only 30% of papers.
• Active Parameter Identification: The replication study demonstrates that parameters often left unspecified in literature (scale factors, extrapolation methods, calibration snapshots) are "active," meaning their variation can fundamentally alter experimental conclusions. In the 132-configuration sweep, variations shifted outcomes from "statistically significant improvement" to "statistically significant degradation" in specific configurations.
• Drift-Induced Effectiveness Illusion: The longitudinal study shows that temporal hardware drift alone can cause the apparent effectiveness of ZNE to vary by a factor of more than 3.4 (e.g., Cohen's $d$ ranging from 3.3 to 11.3) within a 48-hour window on the same device.
• Effective Sample Size Reduction: The study quantifies how temporal drift violates the independence assumption of standard statistical tests. Autocorrelation in the data reduces the effective number of independent observations ($n_{eff}$) from a nominal 30 repetitions to as few as 1.8, drastically weakening the evidential basis of claims derived from repeated measurements.

Results

• Parameter Sensitivity: In the Khan et al. replication, the choice of scale factors and extrapolation method significantly impacted results. For instance, on a depolarizing noise model, ZNE showed significant improvement in 29/33 configurations, but on real hardware snapshots (IBM Osaka), the improvement was less consistent. Crucially, on the IBM Marrakesh processor with low error rates, ZNE was found to be counterproductive for shallow circuits (TC1), increasing error due to variance amplification outweighing correction.
• Temporal Variability: The longitudinal study confirmed that hardware drift is non-stationary and exhibits different patterns across sessions (e.g., step-changes, gradual declines, overnight shifts). The variation in ZNE effectiveness caused by drift (3.4x) exceeded the variation observed when changing the entire noise model (2.7x).
• Statistical Power: The study highlights that low shot counts and few repetitions risk false negatives for genuine effects and an inability to confirm the absence of improvement. Conversely, high shot counts can inflate effect sizes ($d$) without reflecting genuine robustness if the underlying hardware is unstable.

Significance and Claims
The authors do not claim that QEM methods are intrinsically unsound. Instead, they argue that current evaluation practices make mitigation performance appear more robust than the evidence warrants. The paper asserts that:

1. Evaluation Validity: Without controlling for parameter sensitivity and temporal drift, QEM benchmarks cannot reliably distinguish genuine mitigation effects from statistical or experimental artefacts.
2. Reproducibility Crisis: The "reproducibility risk" is high because documented parameters often represent only a small subset of the full parameter space, and the specific calibration snapshot at the time of execution is a critical, often unreported, variable.
3. Proposed Standards: To address these issues, the authors propose minimum reporting standards for QEM evaluations, including:
   • Explicit documentation of all active parameters (including calibration snapshots).
   • Mandatory inferential statistical testing with effect-size reporting.
   • Robustness checks across a grid of configurations.
   • Longitudinal drift assessment or randomization of execution order to de-confound drift from parameter effects.

The paper concludes that these methodological improvements are necessary to ensure the scientific soundness and practical credibility of QEM research as the field moves toward demonstrating quantum utility.