Benchmarking Quantum Computers via Protocols, Comparing IBM's Heron vs IBM's Eagle

This paper employs a protocol-based benchmarking methodology to evaluate and compare IBM's Eagle and Heron quantum processors, demonstrating that the newer Heron architecture offers substantial performance improvements and clearer practical quantum advantages over its predecessor.

The Gist

Imagine you have two giant, incredibly complex Lego sets. One is an older model (let's call it the Eagle), and the other is the brand-new, shiny version (the Heron).

The people who built these sets (IBM) claim the new one is much better. But how do you know if it actually works? You can't just look at the box; you have to try building things with it.

This paper is like a rigorous "stress test" report written by a team of engineers (from the Technion University in Israel) who tried to build specific, tricky structures with both Lego sets to see which one is actually capable of doing the job.

Here is the breakdown of their experiment in simple terms:

1. The Problem: "Gate" vs. "Protocol"

Usually, when people test computers, they check tiny, individual parts (like checking if a single Lego brick is the right color). This is called "gate-level" testing.

The authors say, "That's not enough!" They wanted to see if the computers could actually do something useful. So, instead of checking single bricks, they asked the computers to perform specific "recipes" or protocols. Think of these as instructions like:

• Do-Nothing: Just hold a state steady (like balancing a cup of water on your head while walking).
• Teleportation: Moving information from one side of the chip to the other without physically carrying it (like magic).
• Super-dense Coding: Sending two messages using only one channel (like squeezing a whole suitcase into a backpack).

2. The New "Transmit" Test

When they started testing the older Eagle computer, they hit a wall. It was so broken that even the simplest "Do-Nothing" test failed. The computer couldn't even hold a cup of water steady.

So, the authors invented a simpler test called "Transmit."

• The Analogy: Imagine you have a bucket of water (the information). You have to pass it down a line of people (the qubits) from Person A to Person B.
• The Test: If the water spills too much along the way, the computer fails.
• The Result: Even this simple test was too hard for the old Eagle. It was like asking a toddler to run a marathon.

3. The "Optimal Lookup" Workflow (The Filter)

The authors didn't just test the whole computer at once. They realized that even a bad computer might have a few good "islands" of performance.

They created a filtering process (like a talent show):

1. Round 1 (Corner-to-Corner): They tested small 12-brick sections. If a section failed, they threw it out.
2. Round 2 (Max Length): They tested longer paths within the surviving sections.
3. Round 3 (All Lengths): They tested every possible path.

Only the sections that passed every round were considered "Quantum Capable." This helped them find the "best parts" of the computer, ignoring the broken parts.

4. The Showdown: Eagle (Brisbane) vs. Heron (Kingston)

The Old Guard: IBM Eagle (Brisbane)

• The Verdict: It struggled mightily.
• The Analogy: The Eagle is like an old, rusty bicycle. You can maybe get it to roll a few feet on a flat sidewalk (the "Transmit" test), but if you try to ride it up a hill (Teleportation) or carry a heavy load (Entanglement Swapping), it falls apart.
• The Result: Out of 18 sections, only a tiny few could do the simplest tasks. None of them could work together as a team (pairs) to do complex tasks. It was mostly useless for anything advanced.

The New Challenger: IBM Heron (Kingston)

• The Verdict: It was a game-changer.
• The Analogy: The Heron is like a brand-new, high-performance sports car. Not only can it drive on the sidewalk, but it can also race on the track, carry heavy cargo, and handle sharp turns.
• The Result:
  • Stability: It passed the tests consistently.
  • Scalability: When they tested pairs of sections working together, the Heron succeeded where the Eagle failed completely.
  • Complexity: It could actually perform the "magic" tricks like Teleportation and Entanglement Swapping with high accuracy.

5. The "Protocol Vector" (The Heat Map)

The authors created a colorful map (a "Protocol Vector") to visualize the results.

• Red areas = Broken/Useless.
• Green/Blue areas = Working/Quantum.
• The Eagle Map: Mostly red, with a few tiny green spots.
• The Heron Map: A large, connected island of green. This means you can actually pick a chunk of the Heron chip and run a real program on it.

The Big Takeaway

The paper concludes that while the older IBM computers (Eagle) are essentially "toys" that struggle with basic physics, the new generation (Heron) has finally crossed the line into being practical tools.

They aren't perfect yet, but for the first time, there are large, reliable sections of the chip that can actually perform the complex quantum magic needed for real-world applications. It's the difference between a broken toy car and a real vehicle that can actually drive you somewhere.

In short: The new Heron chip is a massive leap forward, proving that quantum computers are finally moving from "science fiction experiments" to "functional machines."

Technical Summary

Here is a detailed technical summary of the paper "Benchmarking Quantum Computers via Protocols: Comparing IBM's Heron vs IBM's Eagle."

1. Problem Statement

As quantum hardware rapidly advances, the field faces a critical challenge: objectively evaluating the capabilities and error rates of new processors. Traditional gate-level benchmarking often fails to provide a transparent, intuitive assessment of whether a specific quantum processor (or sub-regions within it) can demonstrate a practical quantum advantage. The authors argue that a clear, realistic understanding of current performance is essential to guide research priorities. Specifically, they aim to determine if specific sub-chips within IBM's architectures can reliably execute complex quantum protocols, moving beyond simple gate fidelity metrics to protocol-level viability.

2. Methodology: Protocol-Based Benchmarking

The authors utilize and extend a methodology defined in their prior work [1], which evaluates hardware based on the successful execution of well-defined quantum protocols rather than individual gate errors.

A. The Protocols

The study employs six specific protocols, ordered by complexity:

1. Transmit: A newly defined, fundamental protocol where a state is moved from Alice to Bob via swap gates. It serves as a baseline for "quantumness" with a fidelity threshold of $2/3$.
2. Do-nothing: The standard baseline where a state is initialized and measured without active manipulation (threshold: $2/3$).
3. Bell-state transfer: Transferring a maximally entangled pair across the circuit (threshold: $0.5$).
4. Teleportation: Standard quantum teleportation.
5. Super-dense coding: Transmitting two classical bits using one qubit and entanglement.
6. Entanglement swapping: Swapping entanglement between non-adjacent qubits.

B. The "Optimal Lookup" Workflow

To efficiently assess large chips (127+ qubits) without exhausting computational budgets, the authors developed a staged workflow:

1. Sub-chip Definition: The chip is divided into repeating rectangular sub-chips (12 qubits each for both Eagle and Heron).
2. Staged Filtering:
   • c2c (Corner-to-Corner): Tests paths between opposite corners of a rectangle. Only rectangles passing this proceed.
   • M-L (Maximal Lengths): Tests all maximal length paths within a rectangle. (Note: This stage is skipped for pairs of rectangles as it loses cyclical symmetry).
   • A-L (All Lengths): The final, rigorous stage testing every minimal inner path between qubits in the sub-chip.
3. Progression: A sub-chip must pass the threshold in the current stage to advance to the next. If a single rectangle passes A-L, it is tested as part of a pair (two neighboring rectangles) starting again at the c2c stage.
4. Output: The result is a Protocol Vector, a visual map of the chip showing which sub-chips are capable of which protocols.

3. Key Contributions

• Introduction of the "Transmit" Protocol: Due to the inability of older hardware (Eagle-r3) to pass even the "Do-nothing" protocol, the authors defined "Transmit" as a simpler, fundamental baseline to refine resolution on poorly performing hardware.
• Temporal Benchmarking of IBM Hardware: The study uniquely captures a "before and after" snapshot of the IBM Brisbane (Eagle-r3) chip. The authors observed significant, unannounced improvements in the chip's performance between May and September 2025, categorizing them as "Old" and "Modified" Brisbane.
• Comparative Architecture Analysis: A direct, protocol-level comparison between the older Eagle-r3 (Brisbane) and the newer Heron-r2 (Kingston) architectures.
• Scalability Assessment: The methodology specifically tests scalability by evaluating not just single rectangles, but pairs of rectangles, to see if quantum advantage holds as the sub-chip size doubles.

4. Key Results

A. IBM Eagle-r3 (Brisbane)

• Performance: The "Modified" Brisbane showed improvement over the "Old" version, but performance remained limited.
• Single Rectangles:
  • Transmit: 10 out of 18 rectangles passed the A-L stage.
  • Do-nothing: Only 6 rectangles passed.
  • Complex Protocols: Teleportation passed in 4 rectangles; Bell-state transfer in 1. Super-dense coding and Entanglement swapping failed completely (no rectangles passed the initial c2c stage).
• Pairs (Scalability): Zero pairs of rectangles passed the assessment for Do-nothing, Teleportation, or any complex protocol. Only 6 pairs passed the basic Transmit protocol.
• Conclusion: The Eagle architecture lacks the coherence required for multi-qubit protocols beyond the most basic state transfer.

B. IBM Heron-r2 (Kingston)

• Performance: Demonstrated a substantial leap in capability compared to Eagle.
• Single Rectangles:
  • Transmit: 13 rectangles passed.
  • Do-nothing: 11 rectangles passed.
  • Complex Protocols: Teleportation (11), Bell-state transfer (12), Entanglement swapping (10), and Super-dense coding (9) all had successful sub-chips.
• Pairs (Scalability):
  • Do-nothing: All possible pairs passed.
  • Teleportation: 19 pairs passed.
  • Entanglement Swapping: Only 2 pairs failed.
  • Super-dense coding: 4 pairs passed.
• Conclusion: The Heron architecture demonstrates genuine quantum advantage and scalability, with large contiguous sub-chips capable of executing complex protocols.

C. Quantitative Comparison

• Global Scoring: Using a weighted scoring formula (accounting for yield and average fidelity), Heron (Kingston) significantly outperformed Eagle (Brisbane) across all protocols.
  • Example: Bell-state transfer score: Kingston (0.422) vs. Brisbane (0.028).
  • Example: Entanglement swapping score: Kingston (0.309) vs. Brisbane (0).
• Optimal Sub-chips: The "Optimal Lookup" workflow identified a large, high-fidelity sub-chip on Kingston suitable for complex calculations, whereas Brisbane offered no such viable sub-chip for multi-qubit protocols.

5. Significance and Limitations

Significance:

• Hardware Validation: The study provides empirical evidence that the transition from Eagle to Heron represents a genuine architectural improvement in coherence and error rates, not just a qubit count increase.
• Methodological Utility: The "Protocol Vector" approach offers a practical tool for users to identify specific "islands" of high performance within a noisy quantum processor, which is more useful than average chip-wide metrics.
• Dynamic Hardware: The paper highlights the fluid nature of cloud-based quantum hardware, where unannounced improvements (as seen in Brisbane) can drastically alter benchmarking results.

Limitations:

• Temporal Instability: Both chips exhibited performance fluctuations over time (e.g., results varied by week), making binary pass/fail classifications sensitive to the specific calibration window.
• Data Integrity: Some experiments (particularly on Kingston pairs) suffered from system cancellations, requiring data aggregation from different dates, which introduces potential consistency errors.
• Hardware Availability: The study was constrained by maintenance windows and the retirement of the Sherbrooke chip before full assessment.

Conclusion

The paper concludes that while the older Eagle-r3 architecture struggles to demonstrate quantum advantage beyond the most basic state transfer, the newer Heron-r2 architecture successfully executes complex protocols like teleportation and entanglement swapping on significant sub-chips. The protocol-based benchmarking method effectively isolates these capabilities, proving that Heron is a viable platform for practical quantum advantage, whereas Eagle remains limited to foundational research tasks.