⚛️ quantum physics

Designing Quantum Error Correcting Codes to fit decoders via Reinforcement Learning

This paper presents a reinforcement learning framework using Proximal Policy Optimization to co-design Bivariate Bicycle quantum error correcting codes and their decoders, optimizing the code generation process to maximize decoder performance under depolarizing noise.

Original authors: Omer S. Sella, Robert Pinsler, Thomas Heinis

Published 2026-08-18

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Omer S. Sella, Robert Pinsler, Thomas Heinis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Building a computer that uses the strange laws of quantum mechanics to solve problems is a race against time and noise. In these machines, information is stored in tiny particles called qubits, which are incredibly fragile. Even a whisper of heat or a stray magnetic field can corrupt the data they hold, turning a calculation into garbage. To stop this, scientists use a technique called quantum error correction. They take many physical qubits and link them together to form a single, more stable unit of information, known as a logical qubit. This process is like weaving a safety net; if one strand breaks, the others hold the structure together. However, for this net to work, the machine must constantly check for mistakes and fix them faster than new ones appear. This requires two things working in perfect harmony: the code that defines how the qubits are linked, and the decoder, a fast computer program that figures out what went wrong and how to fix it. If the code is too complex for the decoder to handle, or if the decoder is too slow, the entire system fails.

For years, researchers have treated the design of these codes and the design of the decoders as separate tasks. They would create a code, then try to find a decoder that could read it, or vice versa. But this approach often misses the mark because the performance of a decoder depends heavily on the specific structure of the code it is reading. A decoder that works well with one type of code might struggle with another, even if both codes look similar on paper. The central question driving this new work is simple: if we know exactly how a specific decoder works, can we design a code specifically to make that decoder perform at its absolute best? The researchers at Imperial College London and Microsoft Research set out to answer this by treating the creation of a code not as a static blueprint, but as a journey of small, sequential choices.

To solve this, the team turned to a branch of artificial intelligence called reinforcement learning. Imagine a video game where a player learns to win not by being told the rules, but by trying different moves, seeing what happens, and slowly figuring out which path leads to the highest score. In this study, the "player" is an artificial agent, and the "game" is the construction of a quantum error-correcting code. The agent starts with a blank slate and makes a series of tiny adjustments to a mathematical structure that defines the code. After each small change, the agent tests the new code against a specific decoder to see how well it handles simulated noise. If the code performs better, the agent receives a reward and learns to repeat that kind of change. If it performs worse, the agent learns to avoid it. Over thousands of attempts, the agent learns a strategy for building codes that are perfectly tuned to the decoder it is paired with.

The researchers focused on a specific family of codes known as bivariate bicycle codes. These are a type of quantum code that can be described using four simple mathematical polynomials. Instead of searching through billions of possible arrangements of qubits, the agent only had to decide which coefficients in these four polynomials to flip. This made the search space manageable. The agent was trained using a method called Proximal Policy Optimization, which ensures that the learning process is stable and efficient. The goal was to maximize a single number that represented the code's overall health: the area under the curve that plots how often the code fails at different levels of noise. A larger area meant the code was more robust, keeping errors low even when the environment was noisy.

The results showed that this approach works. The agent successfully learned to generate new codes that outperformed existing, manually designed benchmarks for the same decoder. In one specific test case involving a code with 108 physical qubits, the agent found a configuration that achieved a higher performance score than the best previously known code for that size. The study did not just find a single lucky code; it produced a policy, or a set of rules, that could be used to generate many such high-performing codes. The researchers also discovered that the agent could start with a random, messy code and, through a series of small, deliberate steps, refine it into a highly efficient structure. This suggests that the relationship between a code and its decoder is not fixed, but can be optimized through a process of iterative improvement.

A key part of the study involved teaching the agent to understand the structure of the codes it was building. The researchers designed a special neural network component that could look at the mathematical description of a code and predict how well it would perform before it was even fully tested. This component acted as a shortcut, helping the agent learn faster by giving it a head start on what a good code looks like. They tested this by training the component on small codes and then seeing if it could help the agent design much larger codes it had never seen before. The results were mixed but promising; while the component could predict the number of information units in a code quite accurately across different sizes, predicting the exact error performance was harder. This tension suggests that while the agent can learn the structural rules of good codes, the final performance depends on subtle details that are difficult to generalize.

The paper also highlights the limitations of the current approach. The training was done using simulations of noise, not on actual quantum hardware. While the simulations are based on realistic models of how errors occur, the real world is often messier. The researchers noted that their method currently assumes a specific type of noise that affects all qubits equally, but real quantum devices might have different error patterns. Furthermore, the study focused on a fixed decoder architecture. In the future, the researchers hope to expand this method to design both the code and the decoder at the same time, creating a system where both parts evolve together to solve problems more efficiently. They also pointed out that this method could be applied to other types of quantum codes and different noise environments, potentially helping to build more reliable quantum computers for the future.

Ultimately, this work demonstrates that the design of quantum error-correcting codes does not have to be a static, manual process. By using artificial intelligence to explore the vast landscape of possible codes, researchers can find solutions that are better suited to the specific tools they have available. The study shows that when a code is co-designed with its decoder, the result is a system that is more resilient to errors. This is a significant step forward in the quest to build scalable quantum computers, proving that the right combination of code and decoder can turn a fragile quantum state into a robust tool for computation. The findings suggest that the path to fault-tolerant quantum computing may lie not just in better hardware, but in smarter, more adaptive software that learns to protect the information it carries.

Technical Summary: Designing Quantum Error Correcting Codes to Fit Decoders via Reinforcement Learning

1. Problem Statement

The design of Quantum Error Correcting Codes (QECCs) and their corresponding decoders are traditionally treated as separate problems. However, the performance of a specific decoder—particularly those relying on heuristics and approximations of Maximum Likelihood (ML) decoding, such as Belief Propagation (BP) combined with Ordered Statistics Decoding (OSD)—is heavily dependent on the structural properties of the code.

The central problem addressed is: Given a fixed decoder architecture and a specific noise model, how can one engineer a QECC that maximizes the decoder's performance? The authors argue that code design and decoder design are not independent, especially when the decoder must operate under strict time constraints (latency limits) typical of fault-tolerant quantum computing. The goal is to co-design the stabilizer sets of the code and the decoder to achieve optimal logical error rates under a depolarizing channel.

2. Methodology

2.1 Formulation as a Sequential Decision Problem

The authors frame the code design problem as a Markov Decision Process (MDP) solved using Reinforcement Learning (RL), specifically Proximal Policy Optimisation (PPO).

State ( $s$ ): The current state is defined by the parity check matrices $H_X$ and $H_Z$ of a Bivariate Bicycle (BB) code. These matrices are derived from four polynomials: $a(x), b(x), a(y), b(y)$ .
Action ( $a$ ): An action consists of modifying the coefficients of these polynomials. Specifically, the agent selects one coefficient in one of the four polynomials and flips its binary value (XOR with 1). This allows the agent to iteratively "edit" an existing code.
Episode Structure: Episodes begin with random polynomials (up to 3 non-zero coefficients) and proceed for a fixed number of steps ( $T=30$ ). To prevent the agent from getting stuck in local optima or "mediocre" codes due to sampling noise at low error rates, episodes are truncated after $O(l+m)$ steps.

2.2 Reward Function

The reward is a single scalar derived from the decoder's performance.

Monte Carlo Evaluation: For a given code and decoder, the system simulates errors across a geometric range of physical error rates ( $p \in [0.001, 0.1]$ ).
Logical Error Rate Curve: The simulation produces a curve of logical error rates versus physical error rates.
Integration: The reward is calculated as the area under the curve of $(1 - \text{logical\_error\_rate})$ versus physical error rate. This integrates performance across the entire range.
Logical Qubit Constraint: To prevent the agent from optimizing for trivial codes (e.g., codes with zero logical qubits), the reward is penalized if the number of logical qubits $k$ falls below a minimum threshold $k_{min}$ . If $k < k_{min}$ , the reward is a deterministic function of the deficit; otherwise, the integrated performance metric is used.

2.3 Policy Architecture

The neural network architecture is designed to handle the variable size of codes while leveraging the specific structure of BB codes.

Multi-Branch Input: The network processes three distinct inputs:
1. MLP Branch: Flattened parity check matrices $H_X, H_Z$ (providing full visibility of the code structure).
2. Encoder Branch: A Transformer-based encoder that processes the polynomial coefficients. This branch uses cyclic positional encoding (Fourier basis) to respect the cyclic nature of the BB code construction ( $x^l = y^m = I$ ). This branch is designed to be size-transferable.
3. Logical Qubit Features: Explicit inputs regarding the number of logical qubits $k$ to help the agent immediately recognize invalid or suboptimal code dimensions.
Fusion and Heads: These branches are fused and passed to separate "Actor" (policy) and "Critic" (value) heads. The Actor outputs a categorical distribution over possible coefficient flips for each of the four polynomials.

2.4 Pretraining and Warm-Start

To improve sampling efficiency, the authors employ a "warm start" strategy.

Surrogate Model: The encoder branch is pre-trained in a supervised manner on a large dataset of evaluated BB codes. The pretraining task is multi-task: predicting the logical error rate curve and the number of logical qubits $k$ directly from the polynomial coefficients.
Transfer: The weights of this pre-trained encoder are frozen and used to initialize the RL agent's encoder. This provides the agent with a latent representation of code performance before RL training begins.

3. Key Contributions

Co-Design Framework: A novel RL framework that treats QECC design as a sequential decision-making problem tailored to a specific decoder, rather than a static search for optimal code parameters.
Generative Policy for BB Codes: The development of a generative model that produces Bivariate Bicycle codes by iteratively modifying polynomial coefficients, allowing for the optimization of existing codes.
Size-Transferable Encoder: The design of a Transformer-based encoder that utilizes cyclic positional encoding, enabling the model to learn representations that are potentially transferable across different code sizes ( $l, m$ ), addressing the scalability issues of standard MLPs.
Surrogate-Assisted RL: The integration of a supervised pretraining step (predicting performance curves and logical qubit counts) to guide the RL agent, significantly improving the efficiency of the search process.

4. Results

Performance Improvement: The trained policy successfully discovers codes that outperform published reference BB codes (e.g., for parameters $l=9, m=6$ ) in terms of the integrated reward metric.
Learning Dynamics: Analysis of training runs shows that the agent learns to avoid the "penalty region" (codes with insufficient logical qubits) and converges toward high-performing codes. The "80% band" of evaluation rewards tightens over time, indicating the agent consistently finds better codes.
Extrapolation Limits: While the pre-trained encoder shows success in predicting logical qubit counts ( $k$ ) and error curves for seen code sizes, the paper notes a tension in extrapolating to unseen sizes. Rank correlations between $k$ -prediction and curve-prediction errors become negative for unseen sizes, suggesting that while the encoder learns structural features, perfect generalization of performance prediction across vastly different code dimensions remains challenging.
Efficiency: The use of the warm-start strategy and the specific action space (flipping single coefficients) allows the agent to navigate the search space more effectively than random search or standard evolutionary strategies, though the computational cost of Monte Carlo evaluations remains the primary bottleneck.

5. Significance and Claims

The paper modestly claims to present "progress" in answering how to engineer better, stack-integrated QECC-decoder pairs given a fixed decoder. It does not claim to solve the general problem of QECC design for all hardware or noise models.

Specific Utility: The work demonstrates that for a fixed decoder (BP+OSD) and a specific noise model (depolarizing), an RL agent can iteratively improve BB codes.
Future Direction: The authors position this work as a step toward a broader vision where a generative policy could optimize compiled quantum algorithms in lock-step with changes to the underlying hardware backend and its specific error characteristics.
Limitations: The authors acknowledge that the current approach is limited to the code-capacity noise model (independent depolarizing noise) and does not yet account for circuit-level noise, lattice surgery constraints, or the simultaneous optimization of decoder parameters (though they propose this as future work).

In summary, the paper establishes a proof-of-concept that Reinforcement Learning can be effectively used to co-design quantum error correcting codes and decoders, moving beyond static code selection to dynamic, performance-driven code generation.