⚛️ quantum physics

Quantum control of solid state qubits via Reinforcement Learning

This paper presents a simulation-based analysis demonstrating that model-free reinforcement learning effectively optimizes pulse-level control for state preparation and gate design in NV nuclear spins within a complex, noisy spin-bath environment, outperforming or matching other state-of-the-art black-box optimization tools under challenging conditions of sparse rewards and minimal observability.

Original authors: Dennis Herb, Mirko Rossini, Sabine Wölk, Joachim Ankerhold, Dominik Maile, Alessio Belenchia

Published 2026-10-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dennis Herb, Mirko Rossini, Sabine Wölk, Joachim Ankerhold, Dominik Maile, Alessio Belenchia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the race to build useful quantum computers, scientists face a persistent enemy: noise. Unlike classical computers, which rely on stable electrical currents, quantum machines use fragile quantum states that are easily disturbed by their surroundings. Even the slightest interaction with the environment can scramble the information a quantum computer is trying to process. To make these machines work, researchers must learn to steer these delicate states with extreme precision, a task known as quantum control. The challenge is particularly acute in solid-state devices, where the quantum bits, or qubits, are embedded in a noisy environment of other atoms and spins that act like a chaotic crowd, constantly jostling the qubits and causing errors. For these devices to become reliable tools for sensing or computing, scientists need a way to design control pulses that can navigate through this noise and reach a desired target state with high accuracy.

A team of researchers at Ulm University and the German Aerospace Center has tackled this problem by teaching a computer program to learn how to control a specific type of quantum system: a nitrogen-vacancy center in a diamond. This system consists of a central electron spin that acts as a handle, which scientists can manipulate with microwave pulses, and nearby nuclear spins that serve as the actual memory or qubits. The difficulty lies in the fact that the nuclear spins are not isolated; they are surrounded by a "spin-bath," a collection of other nuclear spins in the diamond that create a complex, fluctuating magnetic environment. Traditionally, designing control pulses for such a system requires a perfect mathematical model of the environment, which is often impossible to know in advance. Instead, the researchers used a method called reinforcement learning, a type of artificial intelligence where an agent learns by trial and error. In this setup, the agent acts as a controller that sends microwave pulses to the electron spin, hoping to indirectly influence the nuclear spins to perform a specific task, such as creating a quantum gate or preparing an entangled state. The agent does not see the full quantum state of the system; it only receives a simple reward signal at the end of each attempt, telling it how well it performed.

The researchers simulated this learning process in a highly realistic environment that included the complex interactions of the spin-bath, using a sophisticated technique to model how the noise affects the system over time. They trained the agent to find the best sequence of pulse durations and phases to achieve two goals: flipping a single nuclear spin to create a logic gate, and entangling two nuclear spins to create a shared quantum state. The results showed that the agent could successfully learn to control the system even when it was surrounded by noise, provided it was trained in a simulated environment that accounted for that noise. Crucially, the agent discovered strategies that were different from those found by other standard optimization methods. For instance, to counteract the noise, the agent learned to use stronger external magnetic fields and shorter pulse sequences, effectively stiffening the system against the chaotic jostling of the surrounding spins. This approach allowed the agent to find solutions that remained robust even when the physical conditions, such as the strength of the microwave pulses or the exact positions of the atoms, varied slightly.

To ensure these findings could eventually be used on real hardware, the researchers tested the agent's ability to adapt to uncertainty. They employed a technique called domain randomization, where the agent was trained in a simulated world where key physical parameters were constantly changing. This forced the agent to learn a flexible strategy rather than memorizing a single perfect solution. When the researchers then tested these trained agents in new, slightly different environments, the agents were able to maintain high performance. Furthermore, they demonstrated that the agent could be "fine-tuned" on a real-world-like system using only simple, single-shot measurements, which are the kind of data actually available in a laboratory. This means the agent can start with a general strategy learned in simulation and then quickly adjust to the specific quirks of a physical device without needing a complete, perfect model of the noise.

The study also compared this learning approach to other powerful mathematical tools used for optimization, such as Bayesian optimization and evolutionary strategies. While these traditional methods could find good solutions in idealized settings, they struggled significantly when faced with the complex, noisy reality of the spin-bath. The reinforcement learning agent, by contrast, consistently found higher-fidelity solutions that were more robust to the imperfections of the physical system. In one specific test involving the creation of an entangled state between two nuclear spins, the agent achieved fidelities above 90 percent, whereas the other methods fell far short. The researchers found that for these longer, more complex tasks, the agent needed to account for higher-order interactions within the noise bath, a level of detail that simpler models often miss. By successfully navigating this complex landscape, the agent proved that it could discover novel control protocols that human-designed sequences or other algorithms might overlook.

Ultimately, this work provides a blueprint for how to control real quantum devices in the noisy era of current technology. It shows that by combining realistic simulations with learning algorithms that can handle sparse, imperfect information, scientists can develop control strategies that are both high-performing and adaptable. The ability to train an agent in a noisy simulation and then fine-tune it on a real device using simple measurements suggests a practical path forward for building reliable quantum sensors and computers. The researchers emphasize that while these results are currently based on simulations, the methods used are designed to work with the limited data available in actual experiments. This approach offers a promising way to tame the inherent noise of solid-state quantum systems, turning a major obstacle into a manageable factor in the design of future quantum technologies.

Technical Summary: Quantum Control of Solid State Qubits via Reinforcement Learning

Problem Statement
The paper addresses the challenge of designing high-fidelity quantum control strategies for Noisy Intermediate-Scale Quantum (NISQ) devices, specifically focusing on the indirect control of nuclear spins in a Nitrogen-Vacancy (NV) center in diamond. The core difficulty lies in the presence of a complex, structured spin-bath (composed of $^{13}$ C nuclear spins) that induces non-Markovian decoherence and gate errors. Traditional model-based optimal control methods often fail in this regime because they require precise knowledge of the bath's microscopic structure and interactions, which is difficult to characterize experimentally. Furthermore, standard Reinforcement Learning (RL) approaches often assume access to the full quantum state or process propagators at each step, an assumption that is unrealistic for physical devices where measurement is destructive and resource-intensive (requiring full tomography). The authors aim to develop a model-free RL framework that operates under "minimally observable" conditions, utilizing sparse, single-shot measurement rewards, while accounting for the microscopic complexity of the spin-bath.

Methodology
The authors employ a model-free Reinforcement Learning approach, specifically the Proximal Policy Optimization (PPO) algorithm, to design microwave (mw) pulse sequences. The setup involves an RL agent that indirectly controls a register of nuclear spins by applying pulses to the central NV electronic spin.

System Modeling: The physical system is modeled using the generalized Cluster Correlation Expansion (gCCE). This technique allows for the inclusion of the microscopic structure of the spin-bath and intra-bath interactions (dipolar couplings) without the prohibitive computational cost of exact diagonalization of the full system-bath Hilbert space. The gCCE approximates the bath by constructing clusters of the register and selected bath spins, treating the remaining bath spins in a mean-field fashion.
Indirect Control: The agent acts on the NV electronic spin via mw pulses (parameterized by duration, phase, and Rabi frequency) and free evolution periods. The goal is to steer the nuclear spin register to a target state (e.g., a Hadamard gate or a Bell state) despite the noise from the surrounding bath.
Observability and Rewards: To mimic realistic experimental constraints, the agent operates in a partially observable environment. It does not receive the full quantum state. Instead, it receives a "sparse reward" only at the end of an episode (a sequence of pulses). The reward is derived from single-shot (or few-shot) Pauli measurements, providing an unbiased estimator of the state or process fidelity. The authors compare several reward strategies, including exact fidelity (simulation only), Direct Fidelity Estimation (DFE), Hofmann bounds, and 0-fidelity.
Sim-to-Real Transfer: To ensure robustness against parameter uncertainties (e.g., fluctuations in Rabi frequencies, magnetic fields, or spin positions), the authors employ Domain Randomization (DR). During training, physical parameters are randomized at each episode. This allows the agent to learn a stochastic policy that generalizes well to unseen environments, facilitating zero-shot transfer or fine-tuning on real hardware.

Key Contributions

Model-Free Control in Structured Baths: The work demonstrates that model-free RL agents can successfully navigate the complex optimization landscape of an NV center coupled to a structured, non-Markovian spin-bath without prior knowledge of the bath's specific configuration.
Sparse Reward Strategies: The paper validates the use of single-shot fidelity estimators as rewards for RL training. This bridges the gap between simulation and experimental reality, showing that agents can learn high-fidelity control sequences using only the limited information available from realistic quantum measurements.
Robustness via Domain Randomization: The authors show that training with domain randomization produces policies that are robust to variations in physical parameters (Rabi frequency, spin positions). This approach enables the transfer of simulated solutions to environments with different characteristics, a critical step for experimental deployment.
Benchmarking: The RL agents are benchmarked against other state-of-the-art black-box optimization methods, specifically Bayesian Optimization (BO) and Covariance Matrix Adaptation Evolution Strategy (CMA-ES).

Results

Gate and State Preparation: The RL agents successfully designed pulse sequences for single-qubit Hadamard gates and two-qubit Bell state preparation.
- Agents trained with the spin-bath (using gCCE) achieved significantly higher fidelities in the presence of the exact bath compared to agents trained in an idealized, bath-free environment.
- For the Hadamard gate task, bath-trained agents achieved fidelities around 96% (exact bath) compared to ~88% for bath-agnostic agents.
- For the Bell state preparation (a more complex, longer-sequence task), RL agents outperformed BO and CMA-ES significantly, especially in the presence of the spin-bath. While BO and CMA-ES achieved fidelities of approximately 70.96% and 70.78% respectively in the exact bath, RL agents trained with gCCE1 and gCCE2 bath models achieved fidelities of 81.50% and 82.77% respectively.
Learning Dynamics: Agents trained with single-shot rewards required more training episodes than those with exact fidelity rewards but still converged to high-fidelity solutions. The use of domain randomization further improved the robustness of these solutions when tested against parameter variations outside the training distribution.
Policy Characteristics: The learned policies often utilized shorter pulse sequences and higher magnetic fields to suppress transverse spin dynamics in the bath. The agents also learned to set the duration of unnecessary pulses to zero, effectively optimizing the sequence length.
Fine-Tuning: The study demonstrated a "pre-training + fine-tuning" pipeline. Agents pre-trained in simulation (with or without bath models) could be fine-tuned on a specific target environment (simulated with different parameters) using single-shot rewards, rapidly restoring high fidelities.

Significance and Claims
The paper claims that Reinforcement Learning offers a viable and competitive pathway for quantum optimal control in complex, noisy solid-state systems where precise system characterization is unavailable. By combining model-free learning with realistic reward schemes (single-shot measurements) and sim-to-real transfer techniques (domain randomization), the authors provide a "blueprint" for experimental implementation.

The significance lies in the demonstration that RL agents can discover tailored control strategies that account for the specific, complex structure of the spin-bath, outperforming traditional black-box optimization methods in these challenging scenarios. The authors emphasize that their approach does not require the characterization of the bath, making it suitable for real-world devices where such characterization is difficult or impossible. While the study is theoretical and simulation-based, the authors argue that the methodology lays the groundwork for deploying robust quantum control strategies on actual NISQ devices, particularly for NV centers and potentially other solid-state qubit platforms.

Technical Summary: Quantum Control of Solid State Qubits via Reinforcement Learning

More like this