Benchmarking Quantum Computers: Towards a Standard Performance Evaluation Approach

This paper reviews classical and quantum benchmarking methodologies to identify the unique challenges of evaluating quantum processors, assesses existing metrics against quality attributes, and proposes general guidelines to establish a standardized performance evaluation framework akin to SPEC for the quantum computing industry.

The Gist

Here is an explanation of the paper, "Benchmarking Quantum Computers: Towards a Standard Performance Evaluation Approach," translated into simple language with creative analogies.

The Big Picture: The "Car Showroom" Problem

Imagine you walk into a car showroom. One salesperson tells you, "My car is the fastest because it has 500 horsepower!" Another says, "Mine is the best because it gets 50 miles per gallon!" A third claims, "Mine is the safest because it has 10 airbags!"

You are confused. How do you compare them? Are you looking for a race car, a family hauler, or an off-roader? In the early days of classical computers (the ones we use today), manufacturers did the same thing. They invented their own tests to make their machines look good, often cheating by optimizing their computers specifically for those tests. This led to a mess where no one knew which computer was actually "better."

Eventually, the computer industry created a "referee" called SPEC (Standard Performance Evaluation Corporation). They created a standard set of driving tests (like a 100-mile highway drive, a steep hill climb, and a city commute) so everyone could compare cars fairly.

Now, the quantum computing world is in the same mess. We have different types of quantum computers (some use trapped ions, some use superconducting circuits, some use light). They are all very different, and there is no agreed-upon way to say which one is the "best." This paper argues that we need to create our own "SPEC" for quantum computers to stop the marketing hype and start real progress.

---

Part 1: Why We Can't Just Copy-Paste Old Rules

The authors explain that we can't just take the old rules for regular computers and slap them onto quantum computers. It's like trying to use a ruler to measure the temperature of a soup.

The Quantum "Magic" (and the Mess):

• Fragile Qubits: Classical bits are like light switches (On or Off). Quantum bits (qubits) are like spinning coins. They are in a state of "both heads and tails" until you look at them. But if you touch them, or if the room gets too hot, they stop spinning and fall flat. This is called noise.
• The "Reset" Button: Every time a quantum computer finishes a calculation, it has to be reset to zero. It's like a runner who has to walk all the way back to the starting line after every single lap. This takes time and energy.
• The "Black Box" Problem: In a classical computer, you can see exactly what the processor is doing. In a quantum computer, the answer is probabilistic. You run the same test 1,000 times, and you get slightly different results each time. You have to guess the "true" answer based on the pattern.

Because of these weird rules, a simple number like "Speed" doesn't mean the same thing for a quantum computer as it does for a laptop.

---

Part 2: The Five "Golden Rules" for a Good Test

The paper suggests that if we want to test quantum computers fairly, any test (benchmark) must follow five golden rules. Think of these as the rules for a fair Olympic sport:

1. Relevance: The test must actually matter. If you are testing a race car, don't measure how well it parks. For quantum computers, we need tests that actually solve problems we care about (like simulating new medicines), not just random math puzzles that don't mean anything.
2. Reproducibility: If I run the test today, and you run it tomorrow with the same settings, we should get the same result. If the results change wildly because the machine is "moody," the test is useless.
3. Fairness: The test shouldn't favor one type of machine over another just because of how it's written. It shouldn't be like a race where one runner has to run through mud while the other runs on a track.
4. Verifiability: We need to be able to check the work. If a quantum computer says "I solved this," we need a way to prove it didn't just guess. (This is hard in quantum physics because sometimes the answer is too complex for us to check with our current tools).
5. Usability: The test shouldn't be so expensive or complicated that only the biggest companies can afford to run it. It needs to be accessible.

---

Part 3: The "Toolbox" of Metrics (The Scorecards)

The authors reviewed many different ways people are currently trying to score quantum computers. They found that most of these "scorecards" are flawed.

• The "Number of Qubits" Trap: Some people say, "I have 100 qubits, you have 50, so I win!" The authors say this is like saying, "I have a bigger engine, so I win," without checking if the engine actually works. A noisy 100-qubit computer might be worse than a clean 10-qubit one.
• Quantum Volume (QV): This is a popular test that measures how big and complex a circuit a computer can run. It's a good start, but it's like measuring a car only by how fast it can drive in a straight line. It ignores how well it handles turns or bad weather.
• The "Cheating" Problem: Many tests require a supercomputer to check the answer. But if the quantum computer is supposed to be better than a supercomputer, how can we check the answer if the supercomputer can't do it? This creates a paradox.

The paper concludes that no single number can tell us if a quantum computer is good. We need a whole "report card" with many different grades.

---

Part 4: The Roadmap (How to Fix It)

The authors propose a step-by-step plan to fix the industry:

1. Match the Test to the Era: We are currently in the "NISQ" era (Noisy Intermediate-Scale Quantum). This is the "prototype" phase. We shouldn't be testing these machines on problems they can't solve yet (like breaking encryption). We should test them on things they can do, to help engineers improve the hardware.
2. Report "Base" and "Peak" Scores:
   • Base Score: How does the computer perform with standard, no-frills settings? (Like a car in "Eco Mode"). This allows fair comparison.
   • Peak Score: How does it perform when experts tweak every setting to get the absolute best result? (Like a car in "Race Mode"). This shows the machine's potential.
3. Create a "Referee" Organization (SPEQC): The authors propose creating a non-profit organization called SPEQC (Standard Performance Evaluation for Quantum Computers).
   • Think of SPEQC as the "Olympic Committee" for quantum computing.
   • They would create the standard tests.
   • They would make sure everyone follows the rules.
   • They would publish the results so consumers (scientists and companies) know which hardware is actually the best.

The Final Takeaway

The paper is a call to action. It says: "Stop the marketing games. Let's build a fair, standardized way to measure quantum computers so we can actually make them better."

Just as the car industry needed standard crash tests and fuel economy ratings to become a mature industry, quantum computing needs standard benchmarks to move from "science fiction" to "real-world technology." Without these standards, we risk building the wrong things, wasting money, and getting confused by fake progress.

Technical Summary

Here is a detailed technical summary of the paper "Benchmarking Quantum Computers: Towards a Standard Performance Evaluation Approach" by Acuaviva et al.

1. Problem Statement

The rapid development of diverse quantum computing platforms (superconducting, trapped ions, photonic, etc.) and paradigms (digital, annealing, analog) has created a critical lack of standardized methods to fairly compare their performance.

• The "Benchmarketing" Risk: Without rigorous standards, the field risks repeating the mistakes of early classical computing, where manufacturers optimized hardware specifically for specific benchmarks rather than general utility (Goodhart's Law). This leads to misleading results and distorted research priorities.
• Inapplicability of Classical Methods: Directly transferring classical benchmarking strategies (like SPEC or TPC) to quantum computing is flawed due to fundamental differences in physics (e.g., noise, decoherence, measurement collapse) and the lack of a unified architecture.
• Terminological Confusion: The field suffers from inconsistent usage of terms like "benchmarking," "verification," "validation," and "metrics," hindering clear communication and comparison.

2. Methodology

The authors employ a comparative and analytical approach, bridging the gap between mature classical benchmarking theory and the nascent quantum field.

• Classical Analysis: They review the history of classical benchmarking (e.g., Whetstone, Dhrystone, LINPACK, SPEC) to extract fundamental principles, quality attributes, and common pitfalls (e.g., the "ratio game," the need for both base and peak metrics).
• Quantum Characterization: They analyze the intrinsic properties of quantum devices (NISQ era) that prevent naive metric transfer, such as the Input-Process-Output (IPO) model, the exponential cost of state tomography, and platform heterogeneity (different native gate sets, connectivity, and error mechanisms).
• Taxonomy Construction: The paper proposes a unified terminology and a hierarchical classification for quantum benchmarking elements (Techniques, Metrics, Benchmarks, Suites).
• Evaluation Framework: Existing quantum metrics and benchmarks (e.g., Quantum Volume, Q-Score, CLOPS, XEB) are systematically evaluated against a set of five quality attributes derived from classical theory.

3. Key Contributions

A. Unified Terminology and Definitions

The authors propose precise definitions to standardize the field:

• Quantum Benchmark: A test or suite measuring performance for a specific task.
• Quantum Performance Metric: A value characterizing a performance property (e.g., fidelity, speed).
• Verification vs. Testing: Distinguishing between certifying a device works as expected (Verification) and checking if a program meets requirements (Testing).
• Benchmark Frameworks/Suites: Structures for defining families of tests (e.g., Volumetric benchmarks) and collections of benchmarks modeling different behavior spaces.

B. Quality Attributes for Benchmarks and Metrics

Adapting classical standards, the paper defines the essential attributes for a "good" benchmark:

1. Relevance: Correlates with user interests (speed, accuracy, scalability).
2. Reproducibility: Produces consistent results under the same configuration.
3. Fairness: Allows competition without bias (e.g., distinguishing between "Base" and "Peak" configurations).
4. Verifiability: Results can be independently confirmed.
5. Usability: Easy and affordable to run.

Similarly, metrics must be Practical (computable in reasonable time), Repeatable, Reliable (consistent ranking), Linear (proportional to performance), and Consistent (same definition across platforms).

C. Critical Evaluation of Existing Proposals

The paper provides a comprehensive review of current quantum metrics (Table 1 in the paper), assessing them against the quality attributes:

• Static Metrics: (e.g., Number of Qubits, Connectivity) are often practical and repeatable but may lack reliability as standalone performance indicators.
• Non-Static Metrics:
  • Quantum Volume (QV): High consistency but low practicality (exponential measurement cost) and limited reliability for non-square circuits.
  • CLOPS: Measures speed but lacks consistency across platforms with different native gate sets.
  • Q-Score: Application-oriented and practical, but reliability depends on the specific optimization problem class.
  • Cross-Entropy Benchmarking (XEB) & HOG: Reliable and consistent but suffer from poor practicality due to the need for classical simulation of ideal distributions (exponential cost).
• Conclusion: No single metric captures the full behavior space; a suite of complementary metrics is required.

D. Proposed Roadmap and Guidelines

The authors outline five specific guidelines for practitioners:

1. Era-Dependent Benchmarks: NISQ-era benchmarks should focus on hardware development and near-term applications (synthetic/kernel), while future eras (FTQC) should focus on full application suites.
2. Adherence to Quality Attributes: New benchmarks must explicitly state which attributes they satisfy and which they lack.
3. Base and Peak Reporting: Similar to SPEC, benchmarks should report a "Base" score (restricted, reproducible configuration) and a "Peak" score (optimized configuration) to balance fairness with innovation.
4. Metric Quality: Metrics should be practical, repeatable, reliable, and consistent.
5. Use of Suites: Avoid reliance on a single benchmark; use a suite to model the full behavior space.

E. Benchmarking Structure and SPEQC Proposal

• Structured Workflow: A step-by-step process is proposed: Device Verification $\rightarrow$ Benchmark Definition (including transpilation flags) $\rightarrow$ Execution (Base/Peak) $\rightarrow$ Reporting.
• SPEQC Organization: The authors propose the creation of the Standard Performance Evaluation for Quantum Computers (SPEQC), a non-profit organization analogous to SPEC in classical computing. SPEQC would manage standardization, ensure fairness through multi-stakeholder involvement (industry, academia, government), and maintain a flexible framework adaptable to different technological eras (NISQ, PQEC, FTQC).

4. Results and Analysis

• Heterogeneity is a Barrier: The paper demonstrates that rigid numerical thresholds (e.g., "gate time < X microseconds") are inappropriate due to the vast differences in physical platforms (e.g., superconducting vs. trapped ions).
• The "Ratio Game" Danger: The analysis confirms that aggregating metrics without care (e.g., using geometric means of normalized scores) can be manipulated to favor specific architectures, reinforcing the need for transparent reporting of base/peak configurations.
• Verification Gap: A significant finding is the difficulty of verifying results for large-scale quantum computers where classical simulation is impossible. The paper highlights the need for new verification protocols (e.g., interactive proofs, statistical tests) beyond simple cross-validation.

5. Significance

This paper is a foundational work for the maturation of the quantum computing industry.

• Preventing Stagnation: By proposing a standardized framework, it aims to prevent the field from devolving into marketing-driven "benchmarketing," ensuring that performance claims are scientifically rigorous.
• Guiding Development: The guidelines help hardware developers focus on improving actual computational utility rather than gaming specific metrics.
• Facilitating Adoption: A clear, fair, and reproducible evaluation standard is essential for users (enterprises, researchers) to make informed decisions about which quantum hardware to adopt.
• Future-Proofing: The proposed SPEQC framework is designed to evolve alongside the technology, ensuring relevance as the field moves from NISQ to Fault-Tolerant Quantum Computing.

In summary, the paper argues that standardization is not just a bureaucratic necessity but a technical imperative for the survival and growth of quantum computing, providing the community with the vocabulary, metrics, and organizational structure needed to transition from experimental physics to a reliable engineering discipline.