Replay-buffer engineering for noise-robust quantum circuit optimization

This paper introduces ReaPER+, OptCRLQAS, and a lightweight transfer scheme to overcome key bottlenecks in deep reinforcement learning for quantum circuit optimization by engineering replay buffers for noise robustness, amortizing expensive evaluations, and reusing noiseless trajectories, thereby achieving significant gains in sample efficiency, wall-clock time, and solution accuracy across various quantum benchmarks.

The Gist

Imagine you are trying to teach a robot to build a complex Lego structure (a quantum circuit) that solves a difficult puzzle. The robot learns by trial and error, just like a human learning to ride a bike. Every time it tries a new arrangement of blocks, it gets a score: "Good job!" or "Try again."

This paper is about how to make that robot learn faster, smarter, and without getting confused by a noisy, messy environment.

The authors found that the robot's "memory" (called a Replay Buffer) was the weak link. Here is how they fixed it using three clever tricks, explained with everyday analogies:

1. The "Smart Diary" (ReaPER+)

The Problem:
Imagine the robot keeps a diary of its past attempts.

• Old Method A: It only reads the entries where it made a huge mistake. This is great for learning quickly at first, but if the robot is just having a bad day (noise), it might keep obsessing over mistakes that weren't actually its fault.
• Old Method B: It only reads the entries where the result was reliable and steady. This is safe, but the robot learns very slowly because it ignores the exciting, high-energy moments where it almost got it right.

The Solution:
The authors created ReaPER+, which is like a smart diary that changes its reading habits over time.

• Early in training: The robot is a beginner. The diary focuses on the "big mistakes" (high error) to help it learn the basics quickly.
• Later in training: The robot is getting better. The diary shifts focus to "reliable successes." It stops obsessing over random noise and focuses on the high-quality lessons that actually matter.
• The Result: The robot learns 4 to 32 times faster than before and builds much more compact, efficient circuits.

2. The "Batch Cooking" Strategy (OptCRLQAS)

The Problem:
In the quantum world, checking if a circuit works is incredibly expensive and slow. It's like baking a cake to see if the recipe is good, but every time you bake a cake, it takes 2 hours and costs $100.
In the old method, the robot would bake a tiny bit of a cake, check the taste, bake a little more, check again, and so on. This is a waste of time and money.

The Solution:
They introduced OptCRLQAS, which is like batch cooking.

• Instead of checking the taste after every single ingredient added, the robot adds a whole bunch of ingredients (10 changes) first.
• Then, it bakes the whole cake once and checks the taste.
• The Result: They cut the time it takes to learn by 67.5%. They got the same delicious cake (solution quality) but spent way less time in the kitchen.

3. The "Ghost Training" Transfer (Noise-Aware Transfer)

The Problem:
Quantum computers today are "noisy." It's like trying to learn to ride a bike on a bumpy, windy road.
Usually, when scientists move from a perfect simulator (a smooth, windless road) to a real noisy quantum computer, they throw away all the practice the robot did on the smooth road and start from zero. That's like telling a pro cyclist, "You practiced on a smooth track? Forget it. Start over on this bumpy dirt road."

The Solution:
They created a lightweight transfer scheme.

• They let the robot practice on the smooth, perfect road (noiseless simulator) and save its "muscle memory" (the replay buffer).
• When they move to the bumpy road (real noisy hardware), they don't throw the memory away. They just drop the robot onto the bumpy road with that memory already loaded.
• The Result: The robot doesn't have to relearn everything from scratch. It reaches the goal 85-90% faster and makes fewer mistakes, even on the bumpy road.

The Big Picture

The paper argues that to make quantum computers useful, we don't just need better hardware; we need better learning strategies.

By treating the robot's memory as a primary tool—making it read the right lessons at the right time, batching its expensive tests, and carrying over its "smooth road" experience to the "bumpy road"—the authors have made quantum circuit optimization significantly faster, cheaper, and more robust against errors.

In short: They taught the robot how to study smarter, not harder.

Technical Summary

1. Problem Statement

Deep Reinforcement Learning (RL) has emerged as a promising framework for optimizing quantum circuits (both compilation and architecture search). However, its practical application faces three fundamental bottlenecks:

1. Inefficient Experience Reuse: Standard replay buffers often ignore the reliability of Temporal-Difference (TD) targets, leading to unstable learning or slow convergence.
2. Computational Bottleneck in Curriculum RL: Curriculum-based Quantum Architecture Search (QAS) triggers a full, expensive quantum-classical evaluation (variational optimization) at every environment step, making scaling to larger qubit counts (e.g., 12-qubits) prohibitively slow.
3. Noise Gap and Data Waste: When transitioning from noiseless simulators to noisy hardware, existing workflows typically discard all noiseless trajectories and retrain from scratch. This is inefficient because the "noiseless-to-noisy" gap represents a shift in computational substrate that is physically unavoidable but makes experience reuse critical.

2. Methodology

The authors propose a framework treating the replay buffer as a primary algorithmic lever, introducing three core components:

A. ReaPER+ (Annealed Replay Rule)

The authors introduce ReaPER+, a dynamic sampling strategy that transitions between two existing prioritization methods:

• PER (Prioritized Experience Replay): Prioritizes transitions with high TD errors ($\delta_t$). Effective for early exploration but can amplify noisy targets.
• ReaPER (Reliability-adjusted PER): Discounts transitions where downstream TD errors indicate unreliable targets. Effective for stability in later training but converges slowly initially.
• Mechanism: ReaPER+ uses an annealing exponent $\omega_\tau$ that increases linearly over training steps $\tau$. The priority $\Psi$ is defined as:
  $$\Psi^{(+,\tau)}_t = R_t^{\omega_\tau} (\delta^+_t)^\alpha$$
  where $R_t$ is the reliability score.
  • Early Training ($\omega \approx 0$): Behaves like PER, focusing on high-error transitions to drive exploration.
  • Late Training ($\omega \to 1$): Shifts toward ReaPER, prioritizing transitions that are both informative (high error) and reliable (low downstream noise), ensuring stable convergence.

B. OptCRLQAS (Amortized Curriculum Learning)

To address the computational cost of QAS, the authors introduce OptCRLQAS, an amortized variant of the standard CRLQAS algorithm.

• Standard Approach: Evaluates the circuit cost after every single gate modification (step).
• OptCRLQAS: Accumulates $m$ local architectural edits (e.g., $m=10$) before triggering a single expensive quantum-classical evaluation.
• Benefit: This reduces the number of variational optimizations per episode by a factor of $m$, significantly cutting wall-clock time while providing a stronger learning signal by evaluating blocks of changes rather than isolated, often indistinguishable, single steps.

C. Lightweight Buffer Transfer

The authors propose a transfer learning scheme that reuses noiseless trajectories to warm-start learning in noisy environments.

• Mechanism: The replay buffer $B_{src}$ collected from a noiseless source environment is directly copied to initialize the target buffer $B_{tgt}$ in the noisy environment.
• Constraints: This is done without transferring network weights or using $\epsilon$-greedy pretraining. It relies on the fact that the state and action spaces remain identical; only the transition dynamics and reward statistics change due to noise.
• Rationale: High-value trajectories found in the noiseless setting remain informative for the noisy setting, providing better initial coverage and accelerating the discovery of high-quality circuits.

3. Key Contributions

1. ReaPER+: A novel annealed replay rule that outperforms fixed PER, fixed ReaPER, and uniform replay, achieving 4–32× gains in sample efficiency across quantum compilation and QAS benchmarks.
2. OptCRLQAS: An efficient curriculum learning method that reduces wall-clock time per episode by up to 67.5% on 12-qubit problems without degrading solution quality.
3. Noise-Aware Transfer: A lightweight transfer scheme that reduces the steps required to reach chemical accuracy by 85–90% and improves final energy error by up to 90% compared to training from scratch in noisy settings.
4. Domain Agnosticism: Validation on the classical LunarLander-v3 benchmark confirms that the ReaPER+ annealing principle is not specific to quantum reward structures but generalizes to standard continuous control tasks.

4. Results

The methods were evaluated on quantum compilation (1-qubit and 2-qubit unitary synthesis) and Quantum Architecture Search (molecular ground-state preparation for $H_2O$ and $BeH_2$).

• Quantum Compilation:
  • 1-qubit: ReaPER+ achieved the highest success probability (89.3% at 0.99 tolerance) and the shortest circuit lengths compared to PER, ReaPER, and PPO.
  • 2-qubit: ReaPER+ reached a fidelity of 0.9920 in $2.5 \times 10^4$ episodes, a 32× reduction in episodes compared to PPO and a 4× reduction compared to fixed ReaPER.
• Quantum Architecture Search (QAS):
  • Sample Efficiency: ReaPER+ consistently found circuits with lower energy error and fewer gates (specifically CNOTs) than non-RL baselines (DQAS, GQAS, TF-QAS) and other RL baselines.
  • Scalability: Using OptCRLQAS, the authors successfully trained on 12-qubit $H_2O$ problems. OptCRLQAS reduced the average wall-clock time per episode by 67.5% (3× faster) compared to standard CRLQAS.
• Noise-Robust Transfer:
  • On 6-, 8-, and 12-qubit molecular tasks, transferring noiseless buffers to noisy settings reduced steps to chemical accuracy by up to 88.2% (at 12-qubits).
  • The transfer advantage grows with system size, demonstrating that experience reuse becomes increasingly critical as quantum systems scale.
• Classical Validation:
  • On LunarLander-v3, ReaPER+ achieved a 9% higher normalized cumulative-return AUC and solved the task 9.3% faster than fixed ReaPER, confirming the generalizability of the annealing strategy.

5. Significance

This paper establishes that replay-buffer engineering is a decisive factor in making RL viable for scalable, noise-robust quantum circuit optimization. By treating the buffer not just as a storage container but as a dynamic algorithmic component, the authors address the critical inefficiencies of sample usage, computational cost, and noise sensitivity.

• Practical Impact: The proposed methods enable the optimization of larger quantum circuits (up to 12 qubits) that were previously computationally intractable due to evaluation costs.
• Theoretical Insight: The work bridges the gap between noiseless simulation and noisy hardware, proving that "from-scratch" retraining is unnecessary and that lightweight buffer transfer is a superior strategy for the noisy intermediate-scale quantum (NISQ) era.
• Generalizability: The success of ReaPER+ on classical benchmarks suggests that these principles of annealed prioritization may benefit deep RL applications beyond quantum computing.