🔬 physics

Flow-Based Surrogates for High-Dimensional Likelihoods in Experimental Neutrino Physics

This paper demonstrates that a hybrid normalizing flow architecture can effectively model high-dimensional, non-Gaussian likelihoods in experimental neutrino physics, achieving near-perfect sampling efficiency and faithful reproduction of complex systematic uncertainties compared to traditional Gaussian approximations.

Original authors: Mathias El Baz, Lorenzo Giannessi, Adrien Blanchet, Federico Sánchez

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mathias El Baz, Lorenzo Giannessi, Adrien Blanchet, Federico Sánchez

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet, deep underground laboratories of the world, scientists are chasing a ghost. Neutrinos are elusive subatomic particles that zip through the Earth by the trillions every second, rarely interacting with anything. To understand them, physicists build massive detectors and fire beams of these particles across hundreds of miles, watching how they change or "oscillate" from one type to another. These changes hold the keys to some of the universe's deepest mysteries, such as why there is more matter than antimatter. But to see these subtle shifts clearly, researchers must first understand the noise that obscures them. Before the particles even reach the distant detector, they pass through a near detector, which measures the beam's initial properties. This measurement is crucial for calibrating the experiment, but it is complicated by a vast web of uncertainties—factors like the exact energy of the beam or how the particles interact with atoms. These uncertainties are not simple errors; they are complex, interwoven variables that can twist the data in unpredictable ways. If scientists cannot accurately model these uncertainties, their measurements of the neutrino's behavior will be flawed, potentially hiding the very answers they seek.

For decades, the standard way to handle this complexity has been to simplify the problem. When researchers analyze their data, they often replace the messy, real-world distribution of these uncertainties with a smooth, symmetrical bell curve, known as a Gaussian approximation. This mathematical shortcut is easy to use and fast to calculate, but it assumes that the uncertainties behave in a perfectly predictable, linear fashion. In reality, the physics of neutrino interactions is often jagged and irregular. The relationships between different variables can curve, and the distribution of possible values can be lopsided, with long tails that the simple bell curve cannot capture. When scientists force this complex reality into a simple shape, they risk introducing a bias that skews their final results. The challenge has been to find a way to carry the full, messy complexity of the initial measurement into the final analysis without needing to keep the entire original dataset or run the original, time-consuming computer simulations every time.

A team of researchers at the University of Geneva and other institutions has developed a new solution to this problem, using a type of artificial intelligence known as a normalizing flow. Instead of trying to force the data into a simple shape, they trained a flexible computer model to learn the exact shape of the uncertainty distribution. Imagine the uncertainty landscape as a rugged terrain with hills, valleys, and strange, winding paths. The old method tried to flatten this terrain into a smooth, round hill. The new method builds a digital map that preserves every ridge and dip of the original landscape. The researchers tested this approach using a realistic simulation of a near-detector experiment, similar to those used in the T2K experiment in Japan. They created a scenario with 110 different parameters representing various sources of uncertainty, ten of which were designed to create complex, non-linear distortions that a simple bell curve would fail to describe.

The results showed that the new method could faithfully reproduce the complex, jagged structure of the true uncertainty distribution. When the researchers compared the output of their new model against a gold-standard reference method called Markov Chain Monte Carlo, which is known for its accuracy but is extremely slow to compute, the two matched almost perfectly. The new model captured the skewed shapes and curved correlations that the traditional bell curve missed. In contrast, the standard approximation failed to reproduce these features, producing a symmetric shape that did not reflect the reality of the data. This difference mattered significantly when the researchers used the models to predict the neutrino flux, or the number of particles expected to arrive. The traditional method overestimated the uncertainty by about 3.5 percent compared to the new model, while the new model's predictions aligned with the accurate reference. This suggests that relying on the old, simplified method could lead scientists to believe their measurements are less precise than they actually are, or worse, to draw incorrect conclusions about the physics.

Beyond accuracy, the new method offers a practical advantage in speed and portability. Once the model is trained, it becomes a compact, self-contained object that can be shared and used by other scientists without needing access to the original, massive datasets or the complex software used to generate them. It can be sampled millions of times in a fraction of the time it takes the old methods. In their tests, the new model generated samples nearly twenty times more efficiently than the traditional approach, allowing researchers to explore the full range of possibilities without getting bogged down by computational limits. The researchers found that the model could handle the high-dimensional complexity of the problem, where dozens of variables interact in subtle ways, while remaining fast enough to be used in real-time analysis. This creates a bridge between the detailed, messy reality of the experiment and the clean, usable models needed for global scientific collaboration.

The significance of this work lies in its ability to preserve the integrity of scientific data as it moves from one stage of an experiment to the next. In modern physics, experiments are often combined, with results from one detector feeding into the analysis of another, or with data from different experiments being merged to find a clearer picture. If the uncertainty information is simplified too early, the subtle details that define the physics are lost. By providing a way to carry the full complexity of the uncertainty through the entire analysis chain, this new approach ensures that the final results are as faithful to the data as possible. The researchers demonstrated that this technique works effectively in a controlled simulation, offering a promising path forward for future experiments where precision is paramount. It represents a shift from forcing nature to fit our mathematical conveniences to building tools that can adapt to the true, intricate shape of the universe.

Technical Summary: Flow-Based Surrogates for High-Dimensional Likelihoods in Experimental Neutrino Physics

Problem Statement
Precision long-baseline neutrino experiments rely on near-detector data to constrain systematic uncertainties on the unoscillated neutrino flux, a prerequisite for accurate oscillation-parameter measurements at the far detector. These constraints involve high-dimensional likelihoods with numerous nuisance parameters (often $O(10^2)$ ) that exhibit complex, non-Gaussian features, including asymmetric marginals, curved correlations, and non-Gaussian tails. Current propagation methods face a methodological gap:

Post-fit Gaussian approximations: Compact and portable but assume quadratic log-likelihoods, failing to capture non-linear dependencies and physical boundaries, potentially introducing bias.
Markov Chain Monte Carlo (MCMC): Faithful to the target distribution but not a closed-form function, making them difficult to insert into downstream likelihoods for gradient-based optimization or joint fits without rerunning the original analysis.
Full likelihood implementations: Accurate but computationally heavy and dependent on specific software infrastructure, limiting portability.

The paper addresses the need for a portable, closed-form, and faithful surrogate for these high-dimensional constrained likelihoods that can be sampled, evaluated, and propagated to downstream analyses (e.g., far-detector predictions or joint experiment fits) without access to original data or the upstream fitting machinery.

Methodology
The authors propose using Normalizing Flows (NF) as explicit surrogate models for the systematic likelihood. The approach utilizes a hybrid architecture designed to balance expressiveness with computational efficiency in high-dimensional spaces ( $D \approx 110$ ):

Hybrid Architecture: The parameter space is partitioned into two blocks based on the expected linearity of the systematic effects:
- High-dimensional "easy" block ( $\eta_B$ ): Modeled using coupling layers (affine coupling blocks). This allows for fast, single-pass sampling and density evaluation.
- Low-dimensional "difficult" block ( $\eta_A$ ): Contains the non-linear parameters (e.g., interaction-model uncertainties) expected to generate strong non-Gaussianities. This is modeled using conditional autoregressive spline flows (Rational Quadratic Neural Spline Flows, RQ-NSF). The autoregressive structure is conditioned on $\eta_B$ to preserve cross-correlations while restricting the expensive autoregressive computation to the smaller dimension.
- The joint density is factorized as $q_{NF}(\eta_A, \eta_B) = q_{AR}(\eta_A | \eta_B) q_{CL}(\eta_B)$ .
Training Strategy:
- Initialization: The flow is initialized using the post-fit Gaussian approximation (best-fit point and covariance matrix from a standard fit) as a proposal distribution. This provides an efficient starting dataset.
- Loss Function: The model is trained to minimize a symmetric Kullback–Leibler (KL) divergence between the target likelihood $p$ and the surrogate $q_\theta$ . To handle the high variance of importance weights in high dimensions, the authors employ an iterative, asynchronous training scheme.
- Asynchronous Loop: CPU workers sample from a lagged flow checkpoint (the proposal), evaluate the expensive target likelihood, and push batches to a FIFO buffer. A GPU trainer continuously optimizes the flow parameters on this aggregated dataset. This prevents the training distribution from being dominated by outdated proposals and stabilizes the optimization.

Key Contributions

Surrogate Model: Development of a hybrid normalizing flow architecture specifically tailored for high-dimensional systematic likelihoods in neutrino physics, combining the speed of coupling layers with the expressiveness of autoregressive splines.
Portability: Demonstration that the trained flow acts as a closed-form, continuous density model. It is samplable, pointwise evaluable, and differentiable, allowing it to be inserted directly into downstream likelihoods for gradient-based optimization.
Benchmarking: Creation of a realistic, controlled near-detector likelihood replica inspired by the T2K experiment, featuring 110 systematic parameters (100 linear, 10 non-linear) and non-Gaussian features derived from event-by-event weights.

Results
The method was validated against a reference MCMC sampling (Metropolis-Hastings) and a standard post-fit Gaussian approximation:

Fidelity in Parameter Space: The NF surrogate accurately reproduces the one-dimensional marginal distributions and two-dimensional projections of the MCMC reference. It successfully captures asymmetric tails, displaced modes, and curved correlations that the Gaussian approximation fails to represent.
Flux Predictions: When propagating systematic uncertainties to unoscillated flux predictions, the NF-based uncertainty bands agree with the MCMC reference within $\approx \pm 1\%$ . In contrast, the Gaussian approximation overestimates the uncertainty width by $\approx 3.5\%$ and underestimates the median event rate by $\approx 0.2\%$ , indicating a bias in the Gaussian surrogate.
Sampling Efficiency: The trained NF achieves a relative effective sample size (rESS) of 98%, compared to only 5% for the Gaussian proposal. This represents a $\approx 20$ -fold improvement in sampling efficiency for importance sampling tasks.
Computational Cost: Once trained, the surrogate is extremely fast, generating $\approx 38$ million samples per GPU-hour. This is orders of magnitude faster than evaluating the original likelihood ( $\approx 10^4$ evaluations per CPU-hour). The computational cost scales primarily with the size of the non-linear block ( $D_A$ ), not the total dimension.

Significance
The paper claims that this approach fills a critical methodological gap in particle physics statistical analysis. By converting the output of a complex fit into a portable, explicit density model, the method enables:

Faithful Propagation: Systematic constraints can be propagated to far-detector analyses or joint fits without losing non-Gaussian information or requiring access to original data.
Efficiency: It eliminates the need to store large MCMC chains or rerun expensive upstream fits for downstream tasks.
Accuracy: It avoids the biases introduced by restrictive quadratic approximations while remaining computationally tractable for gradient-based inference.

The authors position this as a practical tool for the next generation of precision oscillation measurements, where the ability to carry the full complexity of a likelihood is as essential as the precision of the fit itself.

Technical Summary: Flow-Based Surrogates for High-Dimensional Likelihoods in Experimental Neutrino Physics

More like this