🧬 biology

Shot saturation and evidence limits in quantum-AI hardware benchmarks

This paper reanalyzes archived quantum-AI hardware benchmarks to demonstrate how shot saturation and incomplete data handling significantly impact error reconstruction and observable interpretation, while highlighting the limitations of current reporting standards in distinguishing measured values from imposed ones.

Original authors: Vikram Lex

Published 2026-09-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Vikram Lex

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

In the race to build machines that think, a new frontier has emerged where the rules of the physical world meet the logic of artificial intelligence. Scientists are testing tiny, fragile processors that operate on the principles of quantum mechanics, hoping they can solve problems that would take today's supercomputers centuries to crack. To know if these machines are working, researchers must measure specific quantities, such as how closely two patterns of data match or how well a circuit can find the best solution to a puzzle. However, these measurements are not taken once; they are repeated thousands of times to build a reliable picture, much like flipping a coin many times to see if it is fair. The challenge lies in the fact that these quantum machines are noisy and imperfect, and the way researchers count their attempts, or "shots," can drastically change what the numbers appear to say. If the counting method is flawed or if the data is incomplete, the results might look promising when they are actually misleading, making it difficult to tell if the machine is truly learning or just stumbling through the noise.

A recent study by Vikram Lex of KarLex AI takes a hard, critical look at a collection of past experiments involving these quantum processors. The researcher did not run new tests on the hardware itself. Instead, he went back to the digital archives of a previous campaign that used cloud-based quantum computers from providers like Rigetti and IonQ. His goal was to re-examine the raw summaries of those experiments to see if the conclusions drawn from them held up under a stricter, more transparent microscope. He focused on three types of tasks: measuring the similarity between vectors, testing a specific type of mathematical matrix used in machine learning, and running an algorithm designed to solve graph problems. By digging into the details of how the data was recorded, he found that the way the experiments were set up and reported often obscured the true performance of the machines.

The first part of the investigation looked at how increasing the number of measurement attempts affected the accuracy of the results. In these experiments, researchers ran a circuit and recorded the outcome thousands of times, then averaged the results to get a single number. The study re-analyzed data where the number of attempts per batch was increased from 256 to 2,048. The expectation was that simply running more shots would smooth out the errors and bring the result closer to the true value. However, the re-analysis revealed a different story. While the random fluctuations in the data did decrease slightly as more shots were added, the overall error remained stubbornly high. The primary source of the mistake was not the lack of data, but a consistent offset in the average value itself. Even with 2,048 shots, the average result was still significantly different from what the ideal, perfect machine should have produced. This suggests that simply asking the machine to try harder or more often would not fix the problem; the issue lay in the fundamental setup of the measurement or the hardware's behavior, which remained unchanged regardless of how many times the test was repeated.

The second area of focus involved a mathematical structure known as a kernel, which is essentially a table of numbers representing how different data points relate to one another. In a complete and useful table, every possible pair of data points should have a measured value. In the archived data from one of the providers, Rigetti, all 45 possible pairs were measured. However, in the data from another provider, IonQ, only 18 pairs were successfully measured before the experiment ran out of computing credits. The remaining 27 pairs were left blank. The study highlighted a critical flaw in how this incomplete data was handled. When the missing entries were treated as zeros, the average value of the entire table dropped dramatically, changing from a value of roughly 0.22 to 0.09. This massive shift showed that the final conclusion depended entirely on how the missing numbers were filled in. Furthermore, the study found that the two providers were not testing the exact same data inputs, making it impossible to fairly compare their hardware performance. Without knowing exactly which data points were used and ensuring every entry was measured, any comparison between the two machines was scientifically invalid.

The final section examined results from an algorithm called QAOA, which is designed to find the best way to split a network of connections into two groups. The researchers had reported their success as a ratio, comparing the quality of the split they found to the total number of connections in the network. The re-analysis uncovered that different providers were using different definitions for the denominator in this ratio. One provider divided the result by the total number of edges in the graph, while another divided by the theoretical best possible score. This meant that a score of 0.5 from one machine and 0.6 from another did not necessarily mean the second machine was better; they were simply using different rulers to measure the same thing. Additionally, the data lacked the detailed records needed to verify if the machines were actually solving the problem as intended or if the results were just a byproduct of how the data was processed. The study concluded that without a standardized way to report these numbers and full access to the raw data, it is impossible to determine which hardware is truly superior.

Ultimately, this re-analysis serves as a cautionary tale for the field of quantum artificial intelligence. It demonstrates that having a large amount of data or a high number of measurement attempts does not guarantee a correct answer if the underlying definitions are unclear or if the data is incomplete. The study found that the dominant errors in these experiments were not random noise that could be fixed by running more tests, but rather systematic issues related to how the experiments were designed and reported. The researchers emphasized that to move forward, the community needs to establish strict standards for what data must be recorded, how missing information should be handled, and how results should be compared. Until these foundational issues are resolved, claims about the performance of quantum hardware will remain difficult to verify, and the true potential of these machines will remain hidden behind a veil of incomplete evidence.

Technical Summary: Shot Saturation and Evidence Limits in Quantum-AI Hardware Benchmarks

Problem Statement
Current benchmarks for quantum artificial intelligence (QAI) often rely on observables common to machine learning pipelines, such as inner products, feature-space overlaps, and expected values of combinatorial objectives. A critical gap exists in verifying whether returned statistics from quantum hardware actually estimate the intended quantities. Distinctions between signed inner products and squared overlaps, or between cuts normalized by edge counts versus optimal cuts, significantly impact numerical conclusions and reproducibility. Furthermore, incomplete data reporting (e.g., missing kernel entries or absent raw histograms) and inconsistent measurement budgets can obscure the true performance of quantum processing units (QPUs). This study addresses the empirical question of what conclusions regarding shot allocation and observable quality survive a reproducible reanalysis of an existing QAI campaign, specifically the Quantum Hybrid Modules (QHM) campaign.

Methodology
The study is a retrospective reanalysis of 66 historical result summaries, source files, and logs from the QHM campaign, involving Rigetti and IonQ hardware accessed via cloud providers. The author distinguishes between recorded metadata and independently reconstructed quantities, avoiding new QPU executions. The analysis focuses on three circuit families:

Signed Overlaps: Four-dimensional real-vector Hadamard tests. The study reconstructs error statistics using stored batch moments (means and standard deviations) across varying shot budgets (256 to 2048 shots per batch). It employs a stationary independent-shot model to decompose root-mean-square error (RMSE) into bias (mean offset) and variance (within-task dispersion).
Quantum Kernels: Compute–uncompute feature-map circuits. The analysis examines the completeness of off-diagonal entries in kernel matrices, specifically comparing a fully measured Rigetti matrix against a partially measured IonQ matrix where 27 of 45 entries were unmeasured due to credit limits. It calculates the algebraic range of the global mean under zero-filling versus missing-value constraints.
QAOA (MAX-CUT): Rigetti submissions for 3-regular graphs. The study normalizes reported cut values by the number of edges ( $|E|$ ) rather than the optimum, and audits the consistency of denominators and bit-string mappings across different providers (Rigetti vs. IonQ).

The methodology explicitly excludes unauthenticated compiled circuits, ordered overlap shots, and complete QAOA histograms, treating reconstructed integer counts as marginal sufficient statistics under a binomial model rather than recovered physical observations.

Key Contributions

Error Decomposition: The paper demonstrates that stored batch moments allow for the exact decomposition of observed overlap error without inventing missing shots. It reveals that for signed-overlap submissions, increasing the shot count reduces within-task dispersion but leaves the dominant error component (the squared mean offset) largely intact.
Impact of Missing Data: An explicit completion mask exposes the sensitivity of kernel statistics to missing values. Filling unmeasured kernel entries with zero drastically alters the observed-entry mean (from 0.2220 to 0.0888), demonstrating that global means are highly dependent on missing-value treatment.
Normalization and Provenance Audits: The study highlights that reported QAOA ratios are often incomparable due to heterogeneous denominators (e.g., dividing by edges vs. optimum) and inconsistent bit-to-vertex mappings. It establishes that without raw histograms and compiled circuit records, claims about noise-induced barren plateaus or specific algorithmic advantages cannot be validated.
Evidence Boundaries: The work defines the limits of reproducible conclusions, distinguishing between descriptive summaries of archived data and claims of physical noise mechanisms or quantum advantage.

Results

Overlap Measurements: For eight signed-overlap submissions, increasing shots from 256 to 2048 changed the reconstructed RMSE only marginally (from 0.1335 to 0.1292). At 2048 shots, 98.2% of the reconstructed error was attributed to the squared offset of empirical batch means, not shot noise. All ideal overlaps lay outside the conditional simultaneous confidence intervals, indicating that a stationary ideal-shot-noise model alone cannot explain the records.
Kernel Completeness: The Rigetti kernel contained 45/45 measured off-diagonal entries, while the IonQ kernel had only 18/45. The algebraic range for the global mean of the IonQ matrix, given the missing entries could be anywhere in $[0, 1]$ , was found to be $[0.0888, 0.6888]$ . This wide interval underscores the inability to rank provider quality without complete data.
QAOA Scores: Across 43 Rigetti QAOA submissions, size-group mean edge-cut fractions ranged narrowly from 0.4782 to 0.5010, hovering near the 0.5 baseline for independent uniform bits. However, heterogeneous parameters and absent raw histograms prevented interpreting these as controlled scaling experiments. IonQ data showed different denominators and bit-order dependencies that altered observable values, preventing direct comparison.
GPU Supplementary Evidence: Earlier GPU-based results (Fashion-MNIST, AG-News, Grover) were retained as exploratory evidence but were not pooled with QPU measurements due to inconsistencies in training routines and missing checkpoints.

Significance and Claims
The paper modestly claims to provide a "quantified reanalysis" and "evidence boundaries" for a specific historical campaign rather than establishing a new benchmark framework or proving quantum advantage. Its primary significance lies in:

Reproducibility: It documents that many conclusions in QAI benchmarks depend on data completeness and definition consistency rather than just hardware performance.
Observable Validation: It argues that before purchasing more shots, researchers must validate that the implemented observable matches the intended one, as increasing shot counts primarily refines a displaced mean rather than correcting systematic offsets.
Data Prerequisites: It establishes that meaningful provider comparisons require input hashes, completion masks, exact cost conventions, physical-to-logical bit maps, and full outcome histograms.
Limitations: The study explicitly states it does not establish quantum advantage, downstream learning improvements, or a general ranking of processor technologies. It serves as a retrospective check that identifies missing evidence needed for stronger hardware claims.

Technical Summary: Shot Saturation and Evidence Limits in Quantum-AI Hardware Benchmarks

More like this