Bayesian quantum sensing using graybox machine learning

Technical Summary: Bayesian Quantum Sensing Using Graybox Machine Learning

Problem Statement
Quantum sensors offer transformative potential in spatial resolution and sensitivity for applications ranging from materials science to healthcare. However, their practical deployment is hindered by the gap between idealized theoretical models and real-world experimental conditions. In realistic environments, sensor performance is degraded by unmodeled effects, including state preparation imperfections, complex environmental interactions (noise), external parameter dependencies (e.g., temperature, vibrations), and non-ideal control fields (pulse distortions).

These inaccuracies are particularly detrimental to Bayesian inference, a probabilistic framework used for optimal parameter estimation and adaptive control. Bayesian methods rely on a "likelihood" function to update beliefs about unknown parameters based on measurement data. If the underlying model of the sensor does not faithfully represent the experimental reality, the likelihood function becomes inaccurate, leading to suboptimal control decisions, numerical instability, and degraded estimation accuracy. Purely analytical ("whitebox") models often fail to capture these complex non-idealities, while fully data-driven ("blackbox") deep learning models require excessive training data and lack physical interpretability, making them impractical for many quantum sensing tasks.

Methodology
The authors propose a "graybox" (GB) modeling strategy that integrates a physics-based system model with a data-driven description of experimental imperfections. This approach is experimentally validated on a solid-state open quantum system: a single nitrogen-vacancy (NV) center in diamond used to estimate a static magnetic field via Ramsey interferometry.

1. Theoretical Framework:

   • Whitebox (WB) Baseline: The authors first establish a standard physics-based model derived from the Ramsey experiment Hamiltonian. This model accounts for known errors like state preparation infidelity and readout imperfections but assumes ideal control pulses and simplified noise models.
   • Graybox Architecture: The proposed GB model combines a "blackbox" (BB) neural network with "whitebox" (WB) physical layers.
     • Inputs: Ideal pulse parameters (delay $\tau$, phase $\phi$, Larmor frequency $f_B$), external parameters ($\vec{\chi}$), and detector calibration coefficients ($\pi_0, \pi_1$).
     • Blackbox Component: A deep neural network (8 layers) processes the inputs to predict the matrix parameterization of a noise operator ($\hat{V}_Z$). This component learns the unknown deviations caused by noise, imperfect pulses, and environmental factors directly from data.
     • Whitebox Component: The network output is integrated with physical layers that reconstruct the ideal Ramsey unitary evolution and apply the noise operator to compute the expectation value $\langle Z \rangle$.
     • Output: The final layer calculates the probability of a detector click ($P_{cl}$) using the calibrated readout model, effectively generating a highly accurate likelihood function $P(r|f_B)$.

2. Experimental Protocol:

   • Data Acquisition: Experiments were conducted on a single NV center at room temperature. A dataset of approximately 10,000 points was collected, comprising randomized Ramsey sequences with varying delays and phases.
   • Training: The GB model was trained using a loss function based on the logarithm of the Mean Squared Error (MSE) between the predicted click probability and the experimentally measured probability. The dataset was split 90/10 for training and testing.
   • Bayesian Estimation: The trained GB model was embedded into a Bayesian inference loop. Starting with a uniform prior, the algorithm iteratively updated the posterior distribution of the unknown Larmor frequency based on measurement outcomes, using the GB-derived likelihood function.

Key Contributions

• First Experimental Demonstration: This work presents the first experimental implementation of a graybox modeling strategy for a solid-state open quantum system. Previous applications of this framework were limited to numerical simulations or photonic circuits.
• Hybrid Modeling Framework: The authors demonstrate a rigorous integration of noise operator formalism with machine learning, allowing the model to capture complex, non-Markovian dynamics and pulse distortions that are difficult to characterize analytically.
• Improved Likelihood Functions: By training the model on prior experimental data, the GB approach generates a likelihood function that is significantly more accurate than purely analytical models, directly addressing the failure modes of standard Bayesian inference in noisy environments.

Results
The performance of the GB model was benchmarked against a standard whitebox model using the same experimental data for magnetic field estimation.

• Accuracy: The GB model yielded a mean squared error (MSE) improvement of several orders of magnitude compared to the whitebox model. The distribution of MSE for the GB model centered around $10^{-5}$ (MHz)$^2$, whereas the whitebox model centered around $10$ (MHz)$^2$.
• Convergence: While the variance of the estimates indicated that the Bayesian procedure converged for both models, the whitebox model frequently converged to inaccurate values due to model mismatch. The GB model successfully converged to the true frequency in 156 out of 159 test cases.
• Robustness: Even in worst-case scenarios where the GB model did not fully converge, the estimation error remained significantly lower than that of the whitebox approach.

Significance
The paper claims that this work establishes a framework to improve the performance of quantum sensors in realistic environments. The significance lies in demonstrating that graybox models can bridge the gap between theoretical physics and experimental reality without the prohibitive data requirements of fully blackbox approaches.

• Adaptive Control: The authors emphasize that accurate predictive models are critical for real-time adaptive protocols. Even modest model inaccuracies can lead to unstable control loops; the GB approach mitigates this risk.
• Scalability: The method is presented as broadly applicable to various quantum sensing platforms beyond single-spin systems. The authors suggest that the computational requirements for more complex pulse sequences (e.g., multi-pulse dynamical decoupling) could be managed by training smaller, modular blackbox components for base units.
• Future Integration: The work lays the groundwork for integrating graybox models with real-time Bayesian experiment design and reinforcement learning, where the differentiable nature of the model allows for the optimization of information-theoretic quantities like Fisher information.

The authors remain modest regarding the data acquisition time, noting that collecting the 10,000-point dataset took several weeks due to calibration and drift, though they suggest that optical improvements could reduce this to under an hour. They conclude that this approach provides a key building block for scalable, interpretable, and high-performance quantum sensing.