Reinforcement Learning for Robust Calibration of Multi-Qudit Quantum Gates

1. Problem Statement

High-dimensional quantum systems (qudits, where $d > 2$), particularly qutrits ($d=3$), offer advantages over qubits in terms of Hilbert space size and circuit depth. However, controlling them is significantly more challenging due to:

• Spectral Crowding: The dense energy level structure increases the risk of leakage to non-computational states.
• Parameter Sensitivity: Gate fidelity is highly sensitive to device parameters (transition frequencies, coupling strengths) which vary due to fabrication imperfections and slow temporal drifts.
• Limitations of Current Methods:
  • Optimal Control Theory (OCT): Methods like GRAPE (Gradient Ascent Pulse Engineering) can design high-fidelity pulses for a nominal (ideal) model but fail when the actual device deviates from this model (model mismatch).
  • Pure Deep Reinforcement Learning (DRL): While DRL is model-free, it struggles in high-dimensional continuous action spaces (e.g., optimizing thousands of time-slice pulse segments). It often fails to converge to high-fidelity solutions from scratch and is computationally inefficient compared to gradient-based OCT.

The core challenge is to achieve robust, high-fidelity gates across an ensemble of devices with varying parameters without requiring computationally expensive re-optimization for every single device instance.

2. Methodology: Hybrid OCT + DRL Framework

The authors propose a hybrid framework that leverages the strengths of both OCT and DRL, assigning them complementary roles:

A. The OCT Baseline (Open-Loop)

• Role: Generate a high-fidelity "nominal" control pulse for an idealized system model.
• Implementation: Uses GRAPE to optimize control pulses ($\epsilon_1, \epsilon_2$) for a target two-qutrit Controlled-Phase gate ($CZ_3$) on a nominal Hamiltonian.
• Outcome: This pulse serves as a strong initialization and an upper-bound benchmark for fidelity on the ideal device.

B. The DRL Calibration Stage (Closed-Loop/Adaptive)

• Role: Learn residual corrections to the nominal OCT pulse to compensate for static parameter mismatches (fabrication variability and drift).
• Formulation: The problem is framed as a Contextual Bandit:
  • Context (Observation): A normalized vector of device parameter deviations ($\delta\omega_1, \delta\omega_2, \delta g$) relative to the nominal model.
  • Action: Instead of outputting raw pulse shapes, the agent outputs coefficients for a truncated discrete cosine basis. This drastically reduces the action space dimensionality (from $N=1600$ time slices to $K=20$ modes per drive).
  • Reward: The incremental fidelity gain ($F_{RL} - F_{OCT}$) achieved by applying the corrected pulse to the specific noisy device instance.
• Algorithms Evaluated: Soft Actor-Critic (SAC), Twin-Delayed DDPG (TD3), DDPG, and Proximal Policy Optimization (PPO).

3. Key Contributions

1. Hybrid Architecture: Demonstrates that DRL should not replace OCT for pulse synthesis but should act as a calibration layer. By restricting DRL to learning low-dimensional residual corrections, the framework bypasses the "curse of dimensionality" that plagues pure DRL in quantum control.
2. Contextual Bandit Formulation: Introduces a device-aware calibration strategy where the agent learns a generalizable mapping from parameter offsets to pulse corrections, rather than re-optimizing from scratch for each device.
3. Cosine-Basis Parametrization: Utilizes a discrete cosine basis to enforce smoothness and reduce the action space, ensuring the learned corrections are physically realizable and computationally efficient.
4. Comprehensive Algorithm Comparison: Systematically evaluates SAC, TD3, DDPG, and PPO, showing that off-policy methods (TD3, DDPG, SAC) generally outperform PPO in this specific continuous control setting.

4. Key Results

The study was conducted on a two-qutrit superconducting transmon system targeting a $CZ_3$ gate.

• Nominal Device Performance:

  • OCT: Achieves near-unit fidelity ($1 - 10^{-7}$).
  • Pure DRL: Fails to reach high fidelity, plateauing around $0.45-0.48$, confirming that DRL cannot outperform gradient-based OCT on ideal models.
  • Hybrid (OCT + DRL): All agents preserve the high fidelity of the nominal OCT pulse (SAC, TD3, DDPG > 0.99), proving the calibration layer does not degrade performance on ideal devices.

• Robustness to Static Noise (Single Device):

  • When applied to a device with parameter offsets (simulating fabrication error), the nominal OCT fidelity drops to $\approx 0.92$.
  • Hybrid DRL: Successfully recovers fidelity to near-unity ($>0.99$ for SAC/TD3/DDPG, $\approx 0.95$ for PPO) by applying small, structured corrections.

• Ensemble Robustness (100 Devices):

  • OCT Only: Average fidelity drops to 0.824 with a large standard deviation ($\sigma \approx 0.138$), indicating poor transferability.
  • Hybrid DRL:
    • SAC: Achieves 0.963 average fidelity with $\sigma \approx 0.044$.
    • TD3/DDPG: Achieve $\approx 0.962$ with similar low variance.
    • PPO: Achieves 0.926.
  • Conclusion: The hybrid approach reduces the fidelity variance by an order of magnitude, making gate performance uniform across an ensemble of devices.

• Robustness to Imperfect Estimation:

  • The framework remains robust even when the input context (parameter estimates) contains up to 10% noise. Performance degrades only when estimation errors become very large ($>25\%$), indicating the method relies on coarse parameter knowledge.

• Pulse Structure:

  • The learned corrections are small (few percent of amplitude) and smooth, confirming that the agent performs fine-tuning rather than radical pulse redesign.

5. Significance and Implications

• Scalability: This approach offers a scalable path for calibrating multi-qubit and multi-qudit processors. Instead of running expensive OCT re-optimizations for every new chip or drifting parameter, a single trained DRL policy can instantly adapt the nominal pulse to specific device instances.
• Practical Hardware Integration: The method is compatible with existing superconducting control electronics (smooth, low-bandwidth corrections) and integrates naturally with standard characterization routines (spectroscopy) that provide the necessary parameter offsets.
• Theoretical Insight: The paper clarifies the distinct roles of model-based and model-free control: OCT defines the theoretical limit for ideal systems, while DRL provides the adaptability required for real-world, imperfect hardware.
• Future Outlook: The framework is extendable to larger qudit systems, open-system dynamics (decoherence), and online adaptation for time-dependent drifts, positioning it as a critical component for the next generation of quantum hardware calibration.