FTPrimitiveBench: A Benchmark Suite For Logical Computation Under Hardware-Motivated and Biased Noise Models

This paper introduces FTPrimitiveBench, a systematic benchmarking suite that evaluates how logical quantum computing primitives interact with diverse, hardware-motivated noise models beyond the standard uniform depolarizing assumption, thereby enabling reproducible studies for hardware-aware fault-tolerant architecture co-design.

The Gist

Imagine you are trying to build a super-advanced computer that uses the laws of physics (quantum mechanics) to solve problems no regular computer can touch. The biggest problem with these machines is that they are incredibly fragile. The slightest vibration, heat, or electromagnetic wave causes their information to scramble. This is called "noise."

To fix this, scientists use Quantum Error Correction (QEC). Think of this like a team of bodyguards protecting a VIP. Instead of relying on one person (one qubit) to hold the secret, they spread the secret across a whole team (many physical qubits). If one bodyguard gets distracted or makes a mistake, the others can figure out what happened and fix it without losing the secret.

However, there's a catch. Most computer simulations assume that all bodyguards are equally likely to make mistakes, and that mistakes happen randomly and evenly. In the real world, this isn't true. Some bodyguards are more tired than others, some make mistakes more often in one direction than another, and sometimes they all get distracted at the same time.

This paper introduces FTPrimitiveBench, a new "stress test" tool designed to see how well these error-correcting teams perform when the noise is messy, uneven, and realistic—just like real hardware.

Here is a breakdown of what they did and what they found, using simple analogies:

1. The Problem: The "Perfect Weather" Assumption

For a long time, researchers tested their error-correction codes by assuming the weather was always "perfectly uniform rain." They assumed every part of the computer had the exact same chance of getting wet.

• The Reality: Real hardware is more like a storm where it's pouring in one corner, drizzling in another, and the wind is blowing sideways. Some parts of the computer are "biased" (they make one specific type of mistake more often), and some parts are "noisy" (they make mistakes at different rates).
• The Risk: If you design your bodyguard team assuming it's raining evenly, but the wind is actually blowing hard from the East, your team might fail because they aren't positioned to handle the wind.

2. The Solution: FTPrimitiveBench (The "Real-World Simulator")

The authors built a software suite called FTPrimitiveBench. Think of this as a flight simulator for quantum computers, but instead of just simulating smooth flights, it lets you program specific, messy weather patterns.

It allows researchers to:

• Create "Biased" Noise: Imagine a storm where 90% of the rain is falling from the North. The tool can simulate this.
• Create "Measurement" Noise: Imagine the bodyguards' radios are staticky and hard to hear, even if they are standing still. The tool can simulate this.
• Create "Uneven" Noise: Imagine some bodyguards are on a shaky bridge (unstable) while others are on solid ground. The tool can simulate this.

3. The Experiments: Testing Different "Moves"

The researchers tested four specific "moves" (logical operations) that a quantum computer needs to make to do math. They saw how these moves performed under the messy weather conditions.

A. Logical Memory (The "Hold Still" Test)

• The Move: Just holding a piece of information steady without moving it.
• The Result: When the noise was biased (e.g., mostly "Z" errors), they found that changing the shape of the bodyguard team helped. If the noise came mostly from the North, they made the team taller than it was wide. This "asymmetric" shape protected the information much better than a square shape.
• Analogy: If you know the wind only blows from the North, you build a tall, narrow wall to block it, rather than a square wall.

B. The Hadamard Gate (The "Spin" Test)

• The Move: This is a move that swaps the roles of the bodyguards. It's like telling the team, "Now, the people who were guarding the North are guarding the East, and vice versa."
• The Result: This move destroyed the advantage of the asymmetric shape. Because the move swaps the directions, the "North wind" suddenly becomes an "East wind" halfway through the operation.
• Analogy: You built a perfect wall for North wind, but then you rotated the whole building 90 degrees. Now the wall is useless against the wind. The paper found that this specific move is very sensitive to noise and doesn't benefit from the "shape-shifting" tricks that worked for memory.

C. Lattice Surgery (The "Merge" Test)

• The Move: This is when two separate teams of bodyguards join hands to perform a complex task together.
• The Result: When the radios (measurements) were noisy, the teams needed to talk to each other more times to get it right. The paper found that if the radios are bad, you need to repeat the conversation (add more rounds of checking) to be sure you heard correctly.
• Analogy: If you are trying to pass a message across a noisy room, shouting it once isn't enough. You have to shout it ten times and wait for confirmation. The tool showed exactly how many times you need to shout based on how bad the noise is.

D. The Phase Gate (The "Twist" Test)

• The Move: A subtle adjustment to the information.
• The Result: This move behaved similarly to the "Merge" test. It was sensitive to how many times they checked the message (redundancy).

4. Key Discoveries

• Shape Matters (But Only Sometimes): If you have a biased noise problem (like a one-sided wind), changing the shape of your code (making it rectangular instead of square) can drastically improve performance. However, if your computer needs to perform a "spin" move (Hadamard), that shape advantage disappears because the move mixes everything up.
• Decoders Need to Know the Weather: A "decoder" is the brain that figures out what went wrong. The paper found that if the brain knows the noise is biased, it can fix errors much better. But if the noise becomes extremely biased, a simpler brain works just as well as a complex one.
• Unevenness is Okay (Mostly): The researchers tested what happens if every single bodyguard has a slightly different error rate (some are clumsy, some are sharp). Surprisingly, as long as the "brain" (decoder) knows about these differences, the system is very robust. It doesn't fall apart just because the hardware is a bit inconsistent.

Summary

FTPrimitiveBench is a new tool that stops researchers from pretending quantum computers live in a perfect, uniform world. It lets them test their designs against the messy, uneven, and biased reality of actual hardware.

Their main takeaway is that one size does not fit all. A design that works great for "holding still" (memory) might fail miserably when the computer tries to "spin" (Hadamard). To build a reliable quantum computer, engineers need to design their error-correction strategies specifically for the type of noise their hardware produces, and they need to be ready to adjust their plans depending on which "move" the computer is trying to make.

Technical Summary

Technical Summary: FTPrimitiveBench

Problem Statement

The pursuit of fault-tolerant quantum computing (FTQC) requires rigorous evaluation of how error-correcting codes and logical operations perform under realistic physical noise conditions. While standard benchmarks often rely on the uniform depolarizing noise model (where every fault location has an identical error rate $p$), this assumption fails to capture the complex, heterogeneous, and biased characteristics of actual quantum hardware. Real-world devices exhibit:

• Asymmetry: Dominant error channels (e.g., $Z$-biased dephasing in neutral atoms or measurement-dominated errors in superconducting circuits).
• Heterogeneity: Variations in error rates across qubits, gate types, and spatial locations due to calibration drifts and fabrication imperfections.
• Correlations: Spatio-temporal error distributions that deviate from independent, identically distributed (i.i.d.) assumptions.

Existing simulators and benchmark suites often lack a unified framework to systematically explore how these structured noise features interact with specific logical primitives (e.g., memory, lattice surgery, gates). Furthermore, fair comparison across different studies is hindered by non-standardized modeling assumptions (e.g., inclusion or exclusion of idle errors). There is a critical need for a benchmarking suite that aligns noise models with target hardware while maintaining simulation tractability to enable accurate performance estimation and hardware-aware co-design.

Methodology

The authors introduce FTPrimitiveBench, a systematic benchmarking approach designed to decouple noise-model specification from logical circuit generation. The framework operates on the rotated surface code and focuses on core logical Clifford primitives.

1. Noise Model Interface

FTPrimitiveBench establishes a flexible interface for injecting stochastic Pauli error channels into stabilizer circuits. It supports four levels of granularity for parameter assignment:

• Global: Uniform parameters across all components and rounds (recovering standard baselines).
• Spatial: Parameters vary across qubits/interactions but remain fixed in time (static heterogeneity).
• Temporal: Parameters vary round-to-round but are shared across components (drift/fluctuations).
• Spatio-Temporal: Full variation across both space and time.

The framework models three classes of physical noise:

• Gate Errors: General Pauli channels for single- and two-qubit gates, supporting bias and correlation.
• SPAM Errors: Basis-dependent state preparation and measurement errors.
• Idling Errors: Accumulated errors during wait times, calculated using $T_1/T_2$ coherence parameters via a Pauli-twirled approximation.

2. Built-in Noise Families

To facilitate controlled comparative studies, FTPrimitiveBench includes four pre-packaged noise families:

• Uniform Depolarizing: The standard baseline.
• Pauli-Biased: Models dominant error axes (e.g., $Z$-bias) with a bias factor $\eta$.
• Measurement-Biased: Specifically rescales measurement/reset error rates to model readout-dominated regimes.
• Non-Uniform: Applies Gaussian perturbations to error rates to simulate spatial and spatio-temporal heterogeneity.

3. Primitive Generation

The suite provides high-level generators for four fundamental logical primitives, outputting Stim circuits with detector annotations and logical observables:

• Logical Memory: Preserving a logical state over $t$ syndrome-extraction rounds.
• Transversal Hadamard ($H_L$): Swapping $X$ and $Z$ stabilizers via transversal gates.
• Lattice Surgery: Entangling operations via joint parity measurements ($M_{XX}$ or $M_{ZZ}$) involving merge and split phases.
• Logical Phase Gate ($S_L$): Implemented via lattice surgery and $Y$-basis measurement on an ancilla.

4. Evaluation Pipeline

The framework uses Stim for efficient stabilizer simulation and PyMatching (Minimum-Weight Perfect Matching) for decoding. The evaluation sweeps code distances ($d \in \{3, 5, 7, 9, 11\}$) and physical error rates, reporting both absolute Logical Error Rates (LER) and Relative LER (structured noise vs. uniform baseline).

Key Contributions

1. Flexible Noise Modeling: A unified interface supporting custom specifications and structured noise families (bias, measurement bias, non-uniformity) that can be applied consistently across different primitives.
2. Standardized Primitive Generation: Automated generation of Stim circuits for logical memory, lattice surgery, transversal Hadamard, and the $S$ gate, ensuring detector and observable consistency.
3. Reproducible Benchmarking: A workflow that pairs noise models with primitive construction, enabling direct comparative studies of decoders and simulators under matched hardware assumptions.
4. Open Source: The suite is fully open-sourced on GitHub.

Key Results

The evaluation reveals that structured noise affects logical primitives in qualitatively distinct ways:

• Impact of $Z$-Bias:

  • Memory & Lattice Surgery: Asymmetric patches ($d_Z > d_X$) substantially improve performance under $Z$-biased noise by suppressing the dominant fault chains.
  • Transversal Hadamard: This primitive exchanges $X$ and $Z$ channels mid-circuit, effectively averaging the bias. Consequently, the geometric advantage of asymmetric patches is significantly diminished, and the Hadamard does not preserve the input bias.
  • Decoder Performance: Correlated Minimum-Weight Perfect Matching (MWPM) offers a clear advantage over uncorrelated matching under uniform depolarizing noise. However, this advantage narrows as the channel becomes strongly $Z$-biased, as the off-diagonal correlations ($Y$ errors) that correlated matching exploits become rare.

• Impact of Measurement Bias:

  • Temporal Redundancy: Under measurement-biased noise, the optimal number of syndrome-extraction rounds increases with the bias factor. Lattice surgery performance is highly sensitive to round count, highlighting that temporal redundancy is a critical architectural knob invisible to uniform-depolarizing analyses.
  • Non-Monotonicity: The relative LER penalty peaks at intermediate physical error rates (near threshold) rather than at low error rates.

• Impact of Non-Uniform Noise:

  • Robustness: When decoder priors are matched to the underlying per-component error rates, the relative LER for all primitives remains close to the uniform-depolarizing baseline across various variance levels ($\sigma$) and code distances. This indicates the rotated surface code is largely robust to spatial and spatio-temporal heterogeneity.
  • Sampling Effects: Minor deviations below unity in relative LER at small distances are attributed to sampling stochasticity in the perturbation draws rather than a systematic failure mode.

Significance and Claims

The paper claims that FTPrimitiveBench provides a principled basis for moving beyond homogeneous logical-memory benchmarks to analyze active logical computation. Its significance lies in:

• Standardization: It enables reproducible comparative studies of QEC protocols and decoder performance by standardizing the relationship between noise-model specification and primitive construction.
• Hardware-Software Co-Design: By linking hardware characterization (noise profiles) directly to logical-level performance analysis, it provides a practical infrastructure for optimizing fault-tolerant architectures.
• Insight into Primitive Sensitivity: It demonstrates that the benefits of noise-aware design (e.g., asymmetric patches) are not universal; they depend heavily on the specific logical operation (e.g., bias-preserving memory vs. bias-mixing Hadamard).

The authors position FTPrimitiveBench not as an exhaustive mapping of the design space, but as a tractable infrastructure layer that allows researchers to extend studies to new codes, decoders, and noise models without rewriting the underlying simulation pipeline.