⚛️ quantum physics

Approximate Quantum State Preparation Through Proximal Policy Optimization

This paper proposes a deep reinforcement learning framework based on Proximal Policy Optimization to efficiently search for optimal quantum circuits that approximate target states with high fidelity ( $10^{-14}$ ) while minimizing gate counts across various multi-qubit scenarios.

Original authors: Marco Mordacci, Michele Amoretti

Published 2026-07-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marco Mordacci, Michele Amoretti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Quantum Lego Challenge

Imagine you are trying to build a specific, incredibly complex sculpture out of Lego bricks. Now, imagine that instead of just snapping blocks together, every time you add a piece, the entire structure shifts, spins, and changes color in ways that defy normal physics. This is the world of Quantum Computing. Unlike the computers in your pocket that use simple on/off switches (bits), quantum computers use "qubits" that can exist in many states at once. To make these machines do anything useful, scientists first have to perform a tricky dance called Quantum State Preparation (QSP). Think of this as the "setup" phase: you start with a blank slate (all zeros) and need to twist and turn the qubits until they form a very specific, desired pattern.

The problem is that as you add more qubits, the number of ways you can arrange them explodes. It's like trying to find the perfect Lego recipe in a universe where the number of possible recipes grows faster than you can count. If you get the recipe wrong, the quantum computer won't work. For years, humans have tried to write these recipes by hand, but it's like trying to solve a maze that keeps changing shape. This is where Reinforcement Learning comes in. It's a type of artificial intelligence that learns by trial and error, kind of like a dog learning to sit for a treat. The AI tries an action, gets a "reward" if it's close, and a "punishment" if it's far off, slowly figuring out the best path without a human holding its hand.

The Paper's Story: Teaching a Robot to Build Quantum Circuits

In this paper, Marco Mordacci and Michele Amoretti from the University of Parma propose a new way to solve this Lego puzzle. They built a digital "agent" (a smart robot brain) using a specific type of reinforcement learning called Proximal Policy Optimization (PPO). Their goal was simple but tough: teach this agent to build a quantum circuit that creates a specific target state, using as few "gates" (the quantum equivalent of Lego bricks) as possible, while getting the result as close to perfect as math allows.

The agent works step-by-step. It looks at the current state of the quantum system and decides to add one new gate. The gates it can choose from are like a basic toolkit: three types of single-qubit rotations (think of them as spinning a single brick in different directions: $R_x$ , $R_y$ , and $R_z$ ) and one "entangling" gate called CNOT (which links two qubits together, like snapping two bricks so they move as one). Every time the agent adds a gate, the computer runs a quick, mini-training session to tweak the angles of the rotations to make the result as close to the target as possible. If the result gets closer, the agent gets a reward; if it gets further away or uses too many gates, it gets a penalty.

The researchers tested this "robot architect" on a range of challenges, starting with 2 qubits and going up to 5. They asked the agent to build famous, pre-defined patterns (like the "Bell," "GHZ," "W," and "Dicke" states, which are like standard, well-known Lego models) and also completely random, messy patterns.

The results were impressive, especially for smaller systems. For 2-qubit and 3-qubit tasks, the agent often found highly efficient solutions, though not always the absolute theoretical minimum. For example, when asked to build a "Bell state" (a simple two-qubit connection), the agent did not find the exact theoretical solution (which requires a Hadamard gate, a tool not in its toolkit). Instead, it approximated the effect using a rotation gate ( $R_y$ ) to achieve the target. When tackling random 2-qubit states, it consistently found solutions using 7 gates, which matches what other scientists have calculated is the minimum needed. However, the paper notes that in some cases, the agent does not converge to the optimal solution; it might identify the best gates but overuse them due to insufficient penalization, requiring manual simplification later. Despite these quirks, the agent was so precise that it could achieve an error rate as low as $10^{-14}$ , meaning the result was almost indistinguishable from the perfect target.

However, the story gets a bit more complicated as the puzzles get bigger. When the researchers tried 4-qubit and 5-qubit systems, the agent still learned, but it struggled to find the absolute shortest path every time. For random 4-qubit states, the agent usually built circuits with between 47 and 70 gates (averaging around 63), whereas the "optimized" versions (after some manual cleanup) could get down to about 53. For 5 qubits, the search space became so huge that the agent couldn't finish the job perfectly in the time allowed, though it showed it could still learn the basics.

The authors also discovered a clever trick to help the agent when it got stuck. By using a "success buffer"—a memory bank that saved the best 10 circuits the agent had ever found and occasionally retrained on them—the robot could find better solutions faster. They also found that by changing how they calculated the "reward," specifically using a logarithmic scale to make tiny improvements feel like big wins, the agent could keep learning even when it was already 99.999% accurate.

In the end, the paper suggests that this AI-driven approach is a powerful new tool for designing quantum circuits. It doesn't just copy human ideas; it explores the space of possibilities on its own. While it hits a wall of complexity as the number of qubits grows (which is expected given the exponential nature of the problem), the framework successfully automates the design process for small-to-medium quantum systems, achieving errors as low as $10^{-14}$ . The authors note that future work will focus on making the training even faster and testing these ideas on even larger quantum systems, hoping to one day let these AI architects design the complex circuits needed for real-world quantum computers.

Technical Summary: Approximate Quantum State Preparation Through Proximal Policy Optimization

Problem Statement
Quantum State Preparation (QSP) is a fundamental task in quantum computing, essential for applications such as quantum machine learning, Hamiltonian simulation, and quantum chemistry. The objective is to find a unitary operator $U$ that transforms an initial state $|0\rangle^{\otimes n}$ into a target state $|\psi\rangle$ . While exact QSP is theoretically possible, the search space for the optimal quantum circuit grows exponentially with the number of qubits, making the identification of the most efficient circuit non-trivial. In approximate QSP, the goal is to construct a state within a specific tolerance $\epsilon$ while minimizing circuit depth (gate count). Existing methods, including variational quantum circuits and manual design, often struggle to balance approximation accuracy with circuit complexity, particularly as system size increases.

Methodology
The authors propose a novel framework for approximate QSP based on Quantum Architecture Search (QAS) utilizing Deep Reinforcement Learning (RL). The core of the approach is an agent trained via Proximal Policy Optimization (PPO) within an Actor-Critic architecture.

Agent and Environment: The agent sequentially constructs a Parameterized Quantum Circuit (PQC) gate-by-gate. The environment is a quantum simulator (implemented via PennyLane) that evaluates the fidelity between the current state and the target state.
Action Space: At each step, the agent selects an action from a discrete set comprising single-qubit rotation gates ( $R_x(\theta), R_y(\theta), R_z(\theta)$ ) and the entangling CNOT gate. The action space includes all admissible gate-wire combinations for an $n$ -qubit system.
Hybrid Optimization Loop: Upon the agent appending a new gate, the environment triggers a short, internal training phase to optimize the rotation angles ( $\theta$ ) of the circuit. This minimizes a cost function based on the loss $L = 1 - F(\psi, \phi)$ , where $F$ is the fidelity between the target state $\psi$ and the current state $\phi$ . Early stopping is employed to reduce computational overhead.
Reward Function: To guide the agent toward high-precision approximations, a logarithmic reward function is used. This amplifies small reductions in error (distance) when fidelity is already high, preventing the agent from becoming insensitive to incremental improvements. A penalty factor is applied to discourage the generation of unnecessarily deep circuits.
Observation: The agent receives a tuple containing the current quantum state, the number of gates used, and the difference between the target and current states.

Key Contributions

Novel Algorithm: The paper introduces a QSP algorithm specifically based on proximal policy approximation, leveraging PPO to stabilize training and improve sample efficiency compared to standard policy gradient methods.
Comprehensive Evaluation: The framework is rigorously assessed on both predefined entangled states (Bell, GHZ, W, Dicke) and completely random states across system sizes ranging from 2 to 5 qubits.
High-Precision Approximation: The system demonstrates the capability to achieve approximation errors (trace distance) as low as $10^{-14}$ .

Results
Experiments were conducted on a high-performance computing facility using up to 16 parallel environments. The results vary by qubit count:

2-Qubit Systems: The agent successfully identified optimal or near-optimal circuits for Bell states and random states. For random states, the agent consistently found solutions with 7 gates (the theoretical minimum) after approximately 4,000 episodes.
3-Qubit Systems: For predefined states like GHZ and W, the agent achieved optimal circuits. For random 3-qubit states, the agent typically oscillated between 17 and 20 gates. While the theoretical minimum is 17 gates, the agent often found solutions that could be simplified to the optimal count by merging redundant gates. A modified penalty configuration helped the agent converge to the 17-gate solution faster in some runs.
4-Qubit Systems: For W and Dicke states, the agent found solutions with 11–16 gates (W) and 10–12 gates (Dicke). A "success buffer" mechanism was introduced to stabilize training by sampling from the best 10 circuits found. For random 4-qubit states, the agent showed learning trends but had not yet consistently converged to precise solutions within the tested episode limits.
5-Qubit Systems: The framework was extended to 5 qubits to demonstrate scalability. While the agent could identify solutions due to a large gate limit, the model was not fully trained in this scenario, highlighting the challenges of exponential search space growth.

Significance and Claims
The paper claims that the proposed PPO-based framework is effective for automating the design of quantum circuits for state preparation, achieving high-fidelity approximations ( $10^{-14}$ ) while simultaneously minimizing gate counts. The authors note that the agent successfully identifies optimal solutions for many 2- and 3-qubit cases. However, they modestly acknowledge that performance deteriorates as the number of qubits increases, primarily due to the exponential growth of the search space and increased circuit complexity.

The authors conclude that while the current method involves an internal training loop for rotation angles, future work will aim to remove this internal training to achieve more stable results and faster agent training. Additionally, they plan to test the approach on larger qubit systems and compare its circuit depth performance against existing QSP methods.

The Quantum Lego Challenge

The Paper's Story: Teaching a Robot to Build Quantum Circuits

More like this