🤖 machine learning

Reinforcement Learning for Syndrome Extraction

This paper presents a reinforcement learning and importance sampling approach for quantum error correction syndrome extraction that significantly outperforms existing tools like AlphaSyndrome and PropHunt by reducing logical error rates across all scales, achieving up to a 97.8% improvement for distance-15 surface codes.

Original authors: John Zhuoyang Ye, Aarav Pabla, Jens Palsberg

Published 2026-09-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: John Zhuoyang Ye, Aarav Pabla, Jens Palsberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Quantum computers promise to solve problems that would take today's machines thousands of years, from cracking complex codes to simulating new medicines. But these machines are incredibly fragile. The tiny particles they use to store information, called qubits, are easily disturbed by heat, vibration, or even stray electromagnetic waves. A single mistake can corrupt an entire calculation. To make quantum computing practical, scientists have developed a safety net called quantum error correction. Instead of relying on one perfect qubit, they group many unreliable physical qubits together to form a single, more reliable "logical" qubit. This system works by constantly checking for mistakes. It does this by measuring specific patterns of the qubits without looking at the data itself, a process that produces a set of clues called a "syndrome." If the syndrome is empty, everything is fine. If the syndrome shows a pattern, it signals that an error has occurred, and the computer can correct it before the mistake spreads.

The challenge lies in how these checks are performed. To measure a syndrome, the computer must perform a sequence of operations, connecting the qubits in a specific order. While the math allows for many different ways to arrange these connections, not all arrangements are equally safe. In a noisy environment, the order in which these operations happen determines how easily a small mistake can spread from one qubit to many others, potentially causing a catastrophic failure that the error correction cannot fix. Finding the safest arrangement is like searching for a needle in a haystack that grows exponentially larger with every additional qubit. For a modest-sized quantum code, there are more possible arrangements than there are atoms in the observable universe. Previous attempts to find the best arrangement have struggled to balance speed with quality; methods that found good solutions were too slow to be useful for large systems, while faster methods often produced schedules that were still prone to errors.

In a new study, researchers at the University of California, Los Angeles, have developed a tool called FastSched that solves this problem by combining two powerful techniques. They trained a computer program using reinforcement learning, a method where an artificial intelligence learns by trial and error, much like a child learning to ride a bike. The program was tasked with choosing the order of operations for each check, one by one. To make this learning process efficient, the researchers used a statistical trick called importance sampling. Normally, finding a logical error in a well-designed circuit is so rare that a computer would need to run millions of simulations just to see a single failure. This makes learning extremely slow. The researchers instead artificially increased the likelihood of errors during the training phase, allowing the program to see mistakes frequently and learn how to avoid them. They then mathematically adjusted the results to reflect what would happen under normal, low-error conditions. This approach allowed the AI to explore the vast landscape of possible schedules and identify the safest paths without getting stuck or wasting time.

The results of this approach are significant. When tested against the best existing tools, FastSched consistently produced schedules with much lower error rates. On average, it reduced the logical error rate by nearly 26 percent compared to one leading tool and by over 71 percent compared to another. The improvement became even more dramatic as the quantum codes grew larger. For a specific type of quantum code with a distance of 15—a measure of its size and robustness—the new tool reduced the error rate by nearly 98 percent compared to the previous state-of-the-art method. This means that for the same amount of hardware, the new schedules could make the computer vastly more reliable. The researchers verified these findings through extensive simulations using realistic noise models derived from actual hardware data. They found that the tool remained effective even as the physical noise in the system decreased, a scenario where finding errors becomes even harder for traditional methods.

The study also addressed a critical trade-off that has limited previous work. Earlier methods either produced high-quality schedules that took too long to compute or fast schedules that were not reliable enough for serious applications. FastSched manages to be both fast and accurate, scaling effectively to larger and more complex quantum codes. The researchers demonstrated that their method works across a variety of different quantum code structures, not just one specific type. By focusing on the specific order of operations and how errors propagate through the system, they were able to find arrangements that prevent small mistakes from becoming large disasters. This work does not claim to have solved all the problems of quantum computing, but it provides a crucial piece of the puzzle. It offers a practical way to design the control sequences that will be necessary to run large-scale, fault-tolerant quantum computers in the future. The code and results are now available for other scientists to use and build upon, potentially accelerating the timeline for reliable quantum computation.

Technical Summary: Reinforcement Learning for Syndrome Extraction

Problem Statement

Quantum error correction (QEC) relies on syndrome extraction to detect and correct errors in logical qubits. A critical subtask is determining the optimal syndrome extraction schedule, which specifies the order of CNOT gates within each stabilizer check and the timing of these gates.

The search space for these schedules is vast. For a code with $m$ checks where check $i$ involves $w_i$ data qubits, the number of possible CNOT orderings is $\prod_{i=1}^m w_i!$ . For example, a distance-5 rotated surface code has approximately $3.1 \times 10^{24}$ possible order combinations. While the ideal behavior of a check is independent of CNOT order, in a noisy environment, the order dictates how ancilla faults propagate to data qubits (creating "hook errors"). Consequently, different schedules yield vastly different Logical Error Rates (LER).

The core challenge is a search problem: find a schedule that minimizes the LER. Previous state-of-the-art tools face a tradeoff between solution quality and scalability:

AlphaSyndrome uses Monte Carlo Tree Search (MCTS) to find high-quality schedules but scales poorly to large code distances.
PropHunt uses Maximum Satisfiability (MaxSAT) to scale to larger codes but often produces schedules with significantly higher LERs compared to hand-designed or MCTS-optimized ones.

Furthermore, evaluating schedules is computationally expensive because logical errors are rare events. Direct Monte Carlo sampling requires an infeasible number of shots to accurately estimate LERs for high-quality schedules.

Methodology: FastSched

The authors introduce FastSched, a tool that combines Reinforcement Learning (RL) for search and Importance Sampling (IS) for evaluation to overcome the quality-scalability tradeoff.

1. Reinforcement Learning Formulation

The schedule synthesis problem is modeled as a Markov Decision Process (MDP):

State ( $s_i$ ): Encodes the partial schedule, including the history of chosen CNOT orders for previous checks, the code structure (support matrix, check types), and the current check index.
Action ( $a_i$ ): Selects a specific CNOT ordering for the current check $i$ . The action space is a fixed table of permutations (all permutations for small checks; a subset of 32 for larger ones).
Reward: The terminal reward is derived from the estimated LER of the complete schedule. To encourage exploration, an exploration bonus based on the inverse square root of visit counts is added.
Algorithm: The authors use Proximal Policy Optimization (PPO) with an actor-critic architecture. The actor proposes gate orders, and the critic predicts the value of the state. Generalized Advantage Estimation (GAE) is used to assign credit to earlier gate-order decisions based on the final LER.

2. Importance Sampling for Evaluation

To address the rarity of logical failures, FastSched employs Importance Sampling during the evaluation phase:

Amplification: Instead of sampling errors under the target noise model $P$ , the system samples under an amplified distribution $Q$ where error probabilities are increased (capped at 0.45) to make logical failures more frequent.
Weighting: Each sampled failure is weighted by the likelihood ratio $W(x) = P(x)/Q(x)$ to provide an unbiased estimator of the LER under the original noise model $P$ .
Unbiasedness: The paper proves that for a fixed sampling budget, this estimator is unbiased.
Adaptive Stopping: The system uses the Effective Sample Size (ESS) of failure contributions to determine when to stop sampling a candidate schedule, balancing information gain against computational cost.

3. Post-Training Selection and Refinement

After training, the system selects a final schedule through a multi-stage process:

Candidate Pool: The training process maintains a pool of up to 300 distinct schedules with their estimated LERs.
Re-evaluation: A shortlist of candidates is re-evaluated with larger sample budgets (and potentially different amplification factors) to distinguish the best performers.
Local Improvement: An optional local search phase tests single-check CNOT order changes on the best candidates. Changes are accepted if the estimated LER improvement exceeds a combined standard error threshold.

Key Contributions

RL Formulation: A novel RL approach for schedule synthesis that learns gate-order policies to minimize LER, outperforming previous state-of-the-art methods.
Importance Sampling Evaluator: A schedule evaluator based on importance sampling and increasing sample budgets, which significantly reduces the computational cost of evaluating rare logical errors.
Theoretical Guarantee: A proof that the importance sampling estimator is unbiased under a fixed sampling budget.

Results

The authors evaluated FastSched against AlphaSyndrome and PropHunt across various quantum codes (surface codes, color codes, hyperbolic codes, and LDPC codes) using the Brisbane noise model.

Average Reductions:
- Compared to AlphaSyndrome: FastSched reduced the LER by an average of 25.9% across 11 circuits.
- Compared to PropHunt: FastSched reduced the LER by an average of 71.7% across 8 circuits.
Scalability (Distance 15): For a rotated surface code with distance $d=15$ $d = 15$ , FastSched achieved a 97.8% reduction in LER compared to AlphaSyndrome.
- FastSched LER: $3.62 \times 10^{-9}$
- AlphaSyndrome LER: $1.68 \times 10^{-7}$
Scalability (Search Time): Under matched time caps (10–30 minutes), FastSched consistently found schedules with lower LERs than AlphaSyndrome across distances $d=3$ to $d=15$ . In contrast, PropHunt searches often returned no optimized schedules within the time limits for larger distances.
Low-Noise Regime: As physical error rates decreased, the advantage of FastSched grew. For a surface code at $d=5$ with physical noise reduced to 1/32 of the baseline, FastSched's LER was 391.3 times lower than AlphaSyndrome's.

Significance and Claims

The paper claims that FastSched successfully addresses the schedule-synthesis problem without forcing a tradeoff between solution quality and scalability. By combining RL for efficient search and importance sampling for efficient evaluation, the tool can explore the vast space of CNOT orderings to find schedules that significantly outperform both hand-designed and previously automated methods.

The authors emphasize that the key to this success is making rare logical failures useful for learning. Noise amplification allows the RL agent to receive frequent feedback signals during training, while importance weights ensure these signals accurately reflect the performance at the target (low) noise level. The results suggest that combining RL with importance sampling is a viable and superior strategy for syndrome-extraction scheduling, particularly as quantum codes scale to larger distances and lower physical error rates.