Reinforcement Learning for Adaptive Composition of Quantum Circuit Optimisation Passes

Technical Summary: Reinforcement Learning for Adaptive Composition of Quantum Circuit Optimisation Passes

Problem Statement
Quantum software development kits (QSDKs) provide suites of highly optimised circuit passes (e.g., Clifford subcircuit optimisation, ZX-calculus rewriting, Pauli-gadget optimisation). However, the order in which these passes are applied is typically left to the user or defined by general-purpose default sequences. This presents two primary challenges:

1. Expertise Barrier: Recognising when a specific pass is beneficial often requires exceptional expertise in circuit construction and optimisation.
2. Phase Ordering Problem: The quality of the resulting circuit is highly sensitive to the order of application. General-purpose default sequences often miss optimisation opportunities specific to individual circuits, while designing bespoke sequences manually is time-consuming and error-prone.

Existing machine learning approaches often rely on fixed-size observation spaces (limiting input to specific features) or use classical simulation on simplified circuit representations, which can reduce prediction accuracy. Furthermore, search-based approaches for generating bespoke sequences often suffer from poor scalability as the action space and circuit size increase.

Methodology
The authors propose a Reinforcement Learning (RL) framework using Proximal Policy Optimisation (PPO) to train an agent that iteratively selects and applies a sequence of optimisation passes tailored to a specific quantum circuit.

• Graph Representation: The quantum circuit is encoded as a graph where nodes represent gates, inputs, and outputs, and edges represent qubit wires. This representation supports arbitrary connectivity and arbitrary circuit sizes, unlike 3D tensor representations. Node features include gate types (ZZPhase, Rz, PhasedX), parameters, Clifford flags, and the index of the most recently applied pass to prevent idempotent repetition.
• Action Space: The agent selects from a set of global optimisation passes available in PyTKET (version 2.9.3), including:
  • KAKDecomposition (two-qubit subcircuit resynthesis)
  • CliffordResynthesis and CliffordSimp
  • ZXGraphlikeOptimisation
  • GreedyPauliSimp
  • ThreeQubitSquash
  • DoNothing (terminates the sequence)
    After each pass, the circuit is rebased to the Quantinuum native gateset.
• Architecture: The agent employs a Graph Neural Network (GNN) with Graph Convolutional layers (using modified GINConv operators) to process the circuit graph. The network outputs action logits (via an actor) and state values (via a critic). The model uses mean pooling to handle variable circuit sizes.
• Reward Function: The reward for a single action is defined as the fraction of two-qubit gates removed relative to the initial count ($r_t = (n_{t-1} - n_t) / n_0$). A small penalty is applied for actions other than DoNothing to encourage efficiency.
• Training: The model is trained on a synthetic dataset of over 400,000 circuits (including Random-SU4/SU8, IQP, QAOA, and Clifford-mixed classes) with qubit counts ranging from 3 to 7. Training utilizes PPO with hyperparameters optimised via Tree-structured Parzen Estimator.

Key Results
The trained agent was evaluated on a diverse test set of 40,412 circuits (in-distribution) and compared against several PyTKET default sequences (FullPeepholeOptimise, QuantinuumDefaultTwo, QuantinuumDefaultThree) and individual high-performing passes.

• Performance: The RL agent achieved a mean cumulative reward (fraction of two-qubit gates removed) of 57.7% and a median of 56.7%. This outperformed the next best default sequence (QuantinuumDefaultTwo), which achieved a mean of 41.8% and a median of 50.0%.
• Adaptability: The agent successfully generated bespoke sequences for different circuit classes. For example, in Ordered-Clifford-SU4 circuits, the agent correctly identified a sequence involving CliffordSimp followed by KAKDecomposition in 99.2% of cases, demonstrating the ability to delay immediate rewards to enable better subsequent optimisations.
• Generalisability: The model generalised to "out-of-distribution" circuits larger than those seen in training (up to 12 qubits and 450 two-qubit gates), maintaining performance comparable to in-distribution results. It also demonstrated efficacy on very large circuits (100 qubits).
• Efficiency vs. Search: When compared to beam-search-based optimisers (which generate bespoke sequences via exhaustive or greedy search), the RL model produced comparable or superior two-qubit gate reduction but with significantly lower execution time. While search-based methods scale poorly with action space size and search depth, the RL model's inference time scales only with the sequence depth, making it more scalable.

Significance and Claims
The paper claims that this approach resolves the difficulty of manually orchestrating optimisation passes by automating the selection of bespoke sequences. The significance of the work lies in:

1. Superiority over Defaults: Demonstrating that a learned, circuit-specific policy significantly outperforms general-purpose default sequences in reducing two-qubit gate counts (a proxy for noise).
2. Scalability: Showing that RL-based selection is more computationally efficient than search-based methods for generating bespoke pass sequences, particularly as circuit complexity increases.
3. Reduced Expertise Requirement: Lowering the barrier to entry for effective circuit optimisation, as the agent does not require human experts to determine the optimal pass order.
4. Extensibility: The methodology is not limited to the specific passes used; the action space can be expanded to include passes from other libraries without altering the core approach.

The authors note that while the approach improves upon global rewrites, it remains subject to the Phase Ordering Problem inherent in global transformations. They suggest future work could involve agents that select specific circuit regions for rewriting (combining global and local optimisation) and expanding training sets to include real user circuits.