🔬 physics

Convolutional neural network versus persistence for forecasting syndrome rate threshold crossings

This paper demonstrates that while short-term threshold crossings in quantum error correction are predictable under slow calibration drift, a convolutional neural network offers no accuracy advantage over a simple last-cycle persistence baseline, highlighting the necessity of trajectory-disjoint evaluation and robust temporal baselines for assessing machine learning monitors.

Original authors: Ashton Reddy

Published 2026-10-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ashton Reddy

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Quantum computers promise to solve problems that are impossible for today's machines, but they are incredibly fragile. The tiny particles that hold information, called qubits, are easily disturbed by heat, vibration, or even stray electromagnetic waves. To keep these machines running, scientists use a technique called quantum error correction. Instead of relying on a single, perfect qubit, they spread information across several physical qubits and constantly check them for mistakes. They do this by measuring specific patterns, known as syndromes, which reveal if an error has occurred without destroying the delicate data being stored. However, these machines are not static; the conditions inside them change over time. Just as a musical instrument goes out of tune, the components of a quantum processor drift, causing error rates to rise and fall unpredictably. If these shifts go unnoticed, the computer could produce wrong answers without anyone realizing the machine has degraded.

A researcher at Phillips Exeter Academy, Ashton Reddy, asked a practical question: can we predict when these error rates will spike before they cause a failure? The idea is to use the history of the machine's own error checks to forecast the future. If a computer could warn operators that a recalibration is needed, or flag a stretch of computation as unreliable, it would be a major step forward. To test this, Reddy simulated a simple quantum system—a three-qubit code designed to catch bit-flip errors—and let the noise inside it drift slowly over time. The goal was to see if a machine learning model, specifically a type of artificial intelligence called a convolutional neural network, could look at the last eight cycles of error data and predict whether the next cycle would exceed a dangerous threshold.

The study involved generating fifteen separate, long-running simulations of this drifting noise, totaling six thousand cycles of data. To ensure the test was fair, the researcher split the data so that entire simulations were kept separate; the model was trained on some, tested on others, and never saw the same simulation twice. This prevented the model from simply memorizing patterns that happened to repeat within a single run. The model was fed eight consecutive snapshots of error probabilities and asked to guess if the very next snapshot would show a high error rate. The results were clear and somewhat surprising. The neural network did learn to spot the warning signs, correctly identifying nearly all the moments when the error rate spiked. It caught about ninety-eight percent of the high-error events. However, it also sounded the alarm too often, raising false alarms on many quiet cycles.

When compared to a much simpler method, the complex neural network did not win. The simpler method, called a persistence baseline, operates on a straightforward logic: it assumes the next moment will look exactly like the most recent one. If the last cycle was quiet, it predicts the next will be quiet; if the last was noisy, it predicts the next will be noisy. In this specific scenario, where the noise changes slowly, this simple guess worked better than the advanced AI. The simple method achieved an accuracy of ninety-one percent, while the neural network reached about ninety percent. The neural network was better at catching the rare, dangerous spikes, but it made more mistakes overall by predicting danger when there was none. The study suggests that for slow, steady changes in quantum hardware, the most recent data point carries almost all the information needed to make a prediction, rendering the complex pattern-matching of a neural network unnecessary for this specific task.

The research highlights a crucial lesson for anyone building artificial intelligence to monitor quantum computers. If data is split carelessly, a model might appear brilliant simply because it has seen parts of the same pattern before, a flaw known as data leakage. By keeping entire simulations separate, this study ensured that the results reflected genuine predictive power. It also showed that before deploying a sophisticated AI, scientists must check if a simple, rule-based approach can do the job just as well. In this case, the complex tool offered no advantage in overall accuracy over the simple baseline. While the neural network's ability to catch almost every spike is useful in situations where missing an error is far worse than a false alarm, the study concludes that for forecasting slow drifts, the simplest temporal predictor remains the most reliable benchmark. The work serves as a reminder that in the race to build smarter quantum monitors, the most effective tool is not always the most complicated one.

Technical Summary: Convolutional Neural Network versus Persistence for Forecasting Syndrome Rate Threshold Crossings

Problem Statement
Quantum processors are currently limited by noise, including gate, measurement, and coherence errors that vary over time due to calibration drift. While Quantum Error Correction (QEC) mitigates these errors via syndrome measurements, the reliability of the hardware itself fluctuates. The paper investigates whether machine learning can forecast an upcoming rise in error rates—specifically, whether the rate of nonzero syndromes will exceed a fixed threshold—based on recent syndrome statistics. This capability could facilitate proactive recalibration or flag unreliable computation periods. The study focuses on a simplified setting: a simulated three-qubit bit-flip code under slowly drifting noise, asking if a Convolutional Neural Network (CNN) can outperform a simple temporal baseline in predicting these threshold crossings.

Methodology
The study utilizes a dataset of 15 independent noise trajectories, each comprising 400 cycles (6,000 cycles total).

Data Generation: Simulations were performed using Python, Qiskit, and Qiskit Aer with density-matrix simulation. The noise model includes depolarizing errors on gates and symmetric bit-flip readout errors. Crucially, the noise parameters drift over time: the two-qubit error probability ( $p_2$ ) and readout probability ( $p_m$ ) increase linearly with cycle number and include Gaussian random walk components.
Task Formulation: The input consists of eight consecutive cycles of syndrome probabilities (four probabilities per cycle: 00, 01, 10, 11). The target is a binary classification: whether the nonzero-syndrome rate of the next cycle ( $t$ ) exceeds a fixed threshold of 0.08.
Data Splitting: To prevent data leakage caused by the high temporal correlation of sliding windows, the dataset was split by whole trajectory. Nine trajectories were used for training, three for validation, and three for testing. This ensures no cycle or feature from a training trajectory appears in the test set.
Models:
- CNN: A 2D convolutional network with two convolutional layers (32 and 64 filters, $(3,2)$ kernels), max pooling, a dense layer, and dropout. It was trained for 10 epochs using Adam optimization.
- Baselines: Two baselines were established: a "training-majority" baseline (always predicting the majority class) and a "last-cycle persistence" baseline (predicting the next cycle's status based solely on whether the current cycle's rate exceeded the threshold).

Key Results
On the held-out test set (three trajectories, 1,176 windows):

Accuracy: The CNN achieved 89.63% accuracy, while the last-cycle persistence baseline achieved 91.24%. The CNN did not outperform the simple persistence model in overall accuracy.
Recall vs. Precision: The CNN demonstrated higher recall for threshold crossings (98.0%) compared to persistence, identifying 633 of 646 crossings. However, this came at the cost of more false alarms (109 vs. 50 for persistence).
Balanced Accuracy: The CNN's balanced accuracy was 88.71%, slightly lower than persistence's 91.18%. Performance varied across individual test trajectories, with persistence matching or exceeding CNN accuracy in all three cases.
ROC AUC: The CNN achieved a high ROC AUC of 0.9826, indicating strong ranking capability, though this did not translate to superior accuracy at the fixed decision threshold.

Key Contributions and Significance
The paper's primary contribution is methodological rather than algorithmic, highlighting critical pitfalls in evaluating machine learning for QEC monitoring:

Trajectory-Disjoint Evaluation: The study demonstrates that splitting data by sliding windows (which share 7 of 8 cycles) leads to data leakage and inflated accuracy. Splitting by whole trajectory is necessary to obtain a valid estimate of generalization to new noise histories.
Necessity of Simple Baselines: The results show that for slow drift scenarios, the most recent cycle contains sufficient signal to predict the next cycle. Consequently, complex models like CNNs offer no accuracy advantage over a trivial "persistence" baseline.
Evaluation Framework: The author argues that assessing ML monitors for QEC requires both trajectory-disjoint splits and comparison against temporal baselines to distinguish genuine predictive skill from temporal autocorrelation.

Limitations and Scope
The author explicitly notes that the study is limited to a simulated, small-scale setting. The target is the observed syndrome rate, not decoded logical failure. The noise model omits thermal relaxation, crosstalk, and coherent drift. The author cautions that in regimes with abrupt changes or faster drift, or in stateful QEC experiments with logical failure, the persistence baseline might fail, potentially allowing more complex models to demonstrate value. The study concludes that while next-cycle crossings are predictable under slow drift, the CNN provides no accuracy benefit over the simplest temporal predictor in this specific context.

Technical Summary: Convolutional Neural Network versus Persistence for Forecasting Syndrome Rate Threshold Crossings

More like this