⚛️ quantum physics

Benchmarking Optimization Algorithms for Automated Calibration of Quantum Devices

This paper presents a comprehensive benchmark of optimization algorithms for automating quantum device calibration, demonstrating through simulated experiments that the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) outperforms other methods, including Nelder-Mead, across both low- and high-dimensional parameter regimes.

Original authors: Kevin Pack, Shai Machnes, Frank K. Wilhelm

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kevin Pack, Shai Machnes, Frank K. Wilhelm

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have just bought a brand-new, incredibly complex musical instrument. It's not just a guitar; it's a quantum computer. To make it play a perfect note (perform a calculation), every single string, peg, and piece of wood needs to be tuned to the exact right tension. If even one is slightly off, the music sounds terrible, or worse, the instrument breaks.

This process of tuning is called calibration.

The Problem: Tuning by Hand is a Nightmare

Right now, tuning these quantum instruments is done by human experts. They tweak knobs, run tests, tweak more knobs, and repeat.

It takes forever: It can take weeks or even months to tune a machine with just a few dozen "strings" (qubits).
It's fragile: The machine drifts out of tune while you are trying to tune it, like a guitar string stretching while you're tightening it.
It doesn't scale: If we want to build a quantum computer with thousands of strings, humans will never be able to tune them fast enough. We need a robot tuner.

The Solution: Let the Computer Tune Itself

The authors of this paper asked: Can we write a computer program that automatically tunes the machine?

To do this, the program needs a goal. In math terms, this is called a "loss function." Think of this as a scorecard.

If the machine plays a note perfectly, the score is 0 (perfect).
If it's slightly off, the score is 10.
If it's terrible, the score is 1000.

The program's job is to wiggle the knobs until the score is as close to 0 as possible. But here's the catch: the "scorecard" is noisy. Sometimes the machine gives a bad score just because of random static, not because the knobs are actually wrong. It's like trying to tune a radio in a storm; the static makes it hard to hear if you're actually on the right station.

The Race: Which Algorithm Wins?

The researchers tested different "tuning strategies" (algorithms) to see which one could find the perfect settings the fastest and most reliably. They imagined two scenarios:

The Simple Song (Low-Dimensional): A tune with only a few knobs to turn (like a DRAG pulse).
The Symphony (High-Dimensional): A complex tune with 82 knobs to turn simultaneously (like a PWC pulse).

They pitted the contenders against each other:

Nelder-Mead: The old-school veteran. Reliable for simple tasks but gets confused easily.
Simulated Annealing: Inspired by how metals cool down. It's good at escaping bad spots but can be slow.
Differential Evolution: Tries to evolve solutions like nature does, but struggled in this test.
CMA-ES (The Star Player): A sophisticated strategy that learns the "shape" of the problem as it goes.

The Results: The "Smart Navigator" Wins

The researchers ran hundreds of simulations, acting as a massive stress test. Here is what they found:

The "Local Trap" Problem: Imagine you are in a mountainous landscape trying to find the deepest valley (the perfect tune). Many algorithms get stuck in a small dip thinking it's the bottom, not realizing there is a much deeper valley nearby.
- Analogy: It's like stopping at a puddle thinking you've found the ocean.
- CMA-ES was the only one smart enough to realize, "Hey, this puddle isn't the bottom," and keep climbing to find the real ocean.
The Noise Factor: Because the "scorecard" was noisy (static), many algorithms got confused and stopped improving. CMA-ES was the most noise-resistant. It could ignore the static and keep moving toward the true goal.
The Complex Symphony: When they added more knobs (82 parameters), the simple algorithms slowed down or gave up. CMA-ES handled the complexity like a pro, finding the best settings faster and more accurately than anyone else.

The Verdict

The paper concludes that if you want to automate the tuning of quantum computers, CMA-ES is the best tool for the job.

Why? It's robust against noise, it doesn't get stuck in "local traps," and it works well whether you have a few knobs or hundreds.
The Catch: Even the best tuner needs a good map. The researchers noted that the scorecard (the loss function) matters just as much as the tuner. If your scorecard is bad, even the best algorithm won't help.

In a Nutshell

Tuning a quantum computer is like trying to find the perfect recipe for a cake in a dark room where the scales are broken. You have to guess, taste, and adjust.

Some people just guess randomly (Random Search).
Some people follow a strict checklist (Nelder-Mead).
CMA-ES is like a master chef who tastes the batter, remembers what happened last time, and intuitively knows exactly how much sugar to add next, even if the scales are acting up.

The paper proves that this "master chef" approach (CMA-ES) is the most reliable way to get our future quantum computers playing perfect music.

1. Problem Statement

The calibration of Noisy Intermediate-Scale Quantum (NISQ) devices is a critical bottleneck. As the number of qubits increases, manual calibration becomes infeasible due to:

Time Consumption: Manual tuning can take hours to weeks.
System Drift: Temporal instabilities (e.g., coherence time fluctuations) can degrade calibration effectiveness before it is completed.
Scalability: The hierarchical nature of manual calibration does not scale to systems with thousands of qubits.
Complexity: Advanced control pulses (e.g., DRAG, PWC) offer higher fidelities but introduce high-dimensional parameter spaces that are difficult to tune manually.

The authors argue for automated calibration using classical optimization algorithms to replace manual procedures. However, the choice of the optimization algorithm is non-trivial because:

Real-world data is noisy (stochastic).
The optimization landscape often contains multiple local minima.
The system is a "black box" (no analytical gradient available).
The dimensionality of the problem varies from low (simple pulses) to high (complex pulses).

2. Methodology

The authors conducted a comprehensive benchmarking study within a simulated environment designed to mimic real experimental conditions.

A. Simulation Framework

System Model: A single-qubit system modeled as a Duffing oscillator (3-level system) with anharmonicity.
Loss Function: The ORBIT (Optimized Randomized Benchmarking) protocol was used as the loss function.
- It generates random Clifford gate sequences.
- The loss is defined as the negative log of the infidelity: $l(x) = \ln(1 - \text{fidelity})$ .
- Stochasticity: The loss function is noisy because it averages over a finite number ( $N$ ) of random sequences, mimicking experimental sampling noise.
Pulse Types: Two scenarios were tested to evaluate dimensionality scaling:
1. DRAG Pulse: Low-dimensional (few parameters), standard for reducing leakage.
2. Piecewise Constant (PWC) Pulse: High-dimensional (82 parameters), representing complex control shapes where each time-step amplitude is a tunable parameter.
Initial Conditions: A "worst-case" scenario was simulated where all parameters started with a 5% detuning from the optimal values, mimicking a rough initial calibration.

B. Algorithms Benchmarked

The study focused on gradient-free (black-box) optimization algorithms, as gradients are unavailable in experimental settings. The candidates included:

CMA-ES (Covariance Matrix Adaptation Evolution Strategy): State-of-the-art evolutionary strategy.
Nelder-Mead: Classic simplex method.
1+1-ES: Simple evolutionary strategy.
Powell's Method: Direction set method.
Simulated Annealing (SA): Probabilistic global search.
Differential Evolution (DE): Population-based evolutionary strategy.
Bayesian Optimization (mentioned as a candidate but not the primary focus of the final results).

C. Hyperparameter Tuning

To ensure a fair comparison, the authors first optimized the hyperparameters (e.g., population size, initial standard deviation) for each algorithm using CMA-ES. This prevented bias where an algorithm might fail simply due to poor default settings.

3. Key Contributions

Comprehensive Benchmarking: A rigorous comparison of state-of-the-art gradient-free optimizers specifically tailored for quantum device calibration, covering both low and high-dimensional regimes.
Realistic Simulation: The use of the ORBIT protocol as a stochastic loss function and the inclusion of system drift/detuning in the initial conditions provides a more realistic assessment than standard analytical benchmarks.
Hyperparameter Sensitivity Analysis: The study explicitly demonstrates that optimizing the hyperparameters of the optimizers themselves is crucial for valid benchmarking.
Identification of CMA-ES Superiority: Empirical evidence establishing CMA-ES as the most robust algorithm for this specific domain.

4. Results

The performance was evaluated based on convergence speed (function evaluations) and final infidelity (quality of the solution).

Low-Dimensional (DRAG Pulse):
- CMA-ES was the only algorithm that consistently converged to the global optimum.
- Algorithms like Nelder-Mead, 1+1-ES, and Simulated Annealing improved rapidly initially but stagnated at sub-optimal infidelity levels, likely getting trapped in local minima.
- Differential Evolution failed to converge beyond a certain threshold.
- Powell's method plateaued early.
- Observation: The median performance of non-CMA-ES algorithms was significantly worse than their mean, indicating that while some runs succeeded, most failed to escape local minima.
High-Dimensional (PWC Pulse):
- CMA-ES again achieved the lowest final infidelity with fewer function evaluations than all competitors (except 1+1-ES in terms of iteration count, but CMA-ES reached a better final value).
- Surprisingly, simpler algorithms (Nelder-Mead, SA) performed better in the high-dimensional setting than in the low-dimensional one, suggesting the PWC landscape may have fewer local minima or is easier to navigate than the DRAG landscape.
- Differential Evolution remained the worst performer.
Robustness: CMA-ES demonstrated superior robustness to noise and the ability to escape local extrema, which is critical for the stochastic nature of ORBIT-based calibration.

5. Significance and Conclusions

Recommendation: The authors strongly recommend CMA-ES for the automated calibration of quantum devices. It offers the best trade-off between convergence speed, final fidelity, and robustness to noise.
Hyperparameter Importance: The study highlights that the choice of hyperparameters is as critical as the choice of the algorithm itself.
Loss Function Dominance: A key insight is that the loss function (e.g., ORBIT vs. other metrics) has a more decisive impact on the success of calibration than the choice of the optimizer. Future research should focus on optimizing the loss function definition.
Future Directions:
- Investigating hybrid approaches (e.g., starting with Nelder-Mead for speed and switching to CMA-ES for precision), though the authors note the difficulty of transferring state information between fundamentally different algorithms.
- Developing optimizers specifically tailored for quantum system calibration.
- Refining the "model learning" stage of their C3 software toolset using the collected experimental data.

In summary, this paper provides a data-driven roadmap for automating quantum device calibration, identifying CMA-ES as the current gold standard for handling the noise, dimensionality, and local minima inherent in the task.