Post-Error Correction for Quantum Annealing Processor using Reinforcement Learning

Technical Summary: Post-Error Correction for Quantum Annealing Processor using Reinforcement Learning

Problem Statement
Finding the ground state of the Ising spin-glass is a fundamental NP-hard problem in condensed matter physics with broad applications in combinatorial optimization, including the travelling salesman problem and protein folding. While quantum annealing (QA) is a paradigm well-suited for solving Quadratic Unconstrained Binary Optimization (QUBO) problems, commercially available devices (specifically the D-Wave 2000Q) suffer from inherent errors. These include thermal noise causing decoherence and programming control errors in coupling coefficients ($J_{i,j}$) and biases ($h_i$). Consequently, QA often fails to find the true low-energy ground states, necessitating a post-processing error correction scheme to lower the energy of the states output by the annealer.

Methodology
The authors propose a post-processing framework called Simulated Annealing with Reinforcement (SAwR). This approach combines deep reinforcement learning (RL) with classical simulated annealing (SA) to correct errors in QA outputs without requiring the overhead of multiple physical qubits per logical qubit (unlike traditional quantum error correction).

1. Reinforcement Learning Formulation:

   • State ($s$): Represents the Ising instance, including the spin configuration $\sigma$, coupling strengths $\{J_{i,j}\}$, and external magnetic fields $\{h_i\}$.
   • Action ($a$): Flipping a single spin $i$ (changing $\sigma_i$ from $+1$ to $-1$ or vice versa). Each spin can be flipped at most once per episode.
   • Reward ($r$): The change in energy ($\Delta E$) resulting from the spin flip.
   • Objective: The agent learns a policy to maximize the expected return, effectively learning a strategy to flip spins sequentially to reach a lower energy state.

2. Model Architecture (DIRAC-inspired):
   The system utilizes a Graph Neural Network (GNN) based on the DIRAC architecture to encode the Ising instance.

   • Encoding: The Ising instance is treated as a graph where couplings are edge weights and spins/biases are node weights. A modified Spin Glass Neural Network (SGNN) performs message passing via edge-centric and node-centric updates to generate low-dimensional vector embeddings for each node (spin) and the global state.
   • Decoding: A feed-forward neural network takes the concatenation of the state embedding and a specific action embedding (node) to output a Q-value, estimating the expected future reward for flipping that specific spin.

3. The SAwR Procedure:
   The algorithm begins with standard Simulated Annealing. As the temperature lowers, the Metropolis-Hastings criterion (which probabilistically accepts energy-increasing moves) is replaced by a single pass of the trained RL agent. The agent selects spins to flip based on the highest Q-values, recording the energy state after each step. The final solution is defined as the lowest energy state encountered during this procedure.

Key Contributions

• Novel Post-Processing Scheme: Introduction of SAwR, which integrates deep RL with simulated annealing to correct QA errors.
• Scalability: The method demonstrates excellent scalability. The model is trained on small Chimera instances (specifically $C_3$, 72 spins) and successfully deployed on significantly larger instances ($C_{16}$, 2048 spins) without retraining, leveraging the graph-level encoding capabilities of the GNN.
• Architecture Adaptation: Adaptation of the DIRAC architecture to the specific constraints and topology of the D-Wave 2000Q Chimera graph.

Experimental Results
The authors evaluated SAwR against standard Simulated Annealing and a single-pass spin-flipping procedure (the RL model acting alone) on 500 random Ising instances across four Chimera sizes ($C_4$ to $C_{16}$).

• Success Probability: SAwR showed a slightly lower probability of finding a lower energy state compared to standard Simulated Annealing. This difference was consistent across all system sizes, suggesting a systemic rather than random variance.
• Improvement Magnitude: On average, SAwR was able to find a better low-energy state than standard SA, though the difference was not statistically significant.
• Single-Pass Performance: The standalone RL agent (single pass of spin flipping) performed poorly, achieving only approximately a 1% success rate, highlighting the necessity of combining RL with the broader search capabilities of Simulated Annealing.

Significance and Claims
The paper positions SAwR as a "proof-of-concept" demonstrating that reinforcement learning can be applied to correct errors in quantum annealer outputs. The authors explicitly acknowledge that the performance of SAwR on the Chimera graph is currently inferior to typical algorithms like standard Simulated Annealing.

The significance lies not in immediate outperformance, but in the demonstration of scalability and the viability of using RL for post-error correction. The authors note that the complexity of the real-world Chimera graph (compared to the regular lattices used in prior DIRAC studies) likely contributes to the performance gap. They conclude that while the current results are modest, using reinforcement learning for post-error correction remains an "open and promising avenue of research," particularly as D-Wave systems evolve toward more complex topologies like Pegasus and Zephyr. Future work is suggested to explore architectural changes (e.g., separating encoder and decoder training) and alternative RL algorithms (e.g., asynchronous methods or Monte Carlo Tree Search).