⚛️ high-energy experiments

Finite Asimov Sample Construction in Unbinned Neural Simulation-Based Inference

This paper proposes a method to construct a finite, weighted reference sample that globally maximizes the likelihood for learned density ratios, demonstrating that such a small dataset can accurately reproduce expected test statistic scans and distributions typically requiring millions of simulated events in neural simulation-based inference.

Original authors: Rafael Coelho Lopes de Sa, Jay Sandesara

Published 2026-09-15

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Rafael Coelho Lopes de Sa, Jay Sandesara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes world of particle physics, scientists often act as cosmic detectives trying to understand the universe by smashing particles together at incredible speeds. When these collisions occur, they produce a chaotic spray of new particles, and researchers must sift through this debris to find rare signals hidden within a massive sea of background noise. To know if they have truly discovered something new, they rely on a statistical tool called an "expected sensitivity" calculation. This process involves creating a perfect, imaginary version of an experiment—a theoretical ideal known as an Asimov dataset—that represents what the data would look like if the laws of physics were exactly as predicted. By comparing real observations against this perfect ideal, scientists can determine if a signal is strong enough to be considered a discovery or if it is just a random fluctuation. However, building this perfect ideal has traditionally been a computationally exhausting task, requiring the generation of millions of simulated events to smooth out the natural randomness of the data, a process that consumes vast amounts of computer time and memory.

A team of researchers at the University of Massachusetts Amherst and the University of Wisconsin–Madison has found a way to bypass this heavy computational burden. They developed a method to construct a finite, weighted reference sample that acts as a highly efficient stand-in for the massive datasets usually required. Instead of relying on millions of individual simulated events to approximate the perfect ideal, their new approach uses a learned mathematical relationship to assign specific weights to a much smaller set of reference points. In a demonstration using a model inspired by high-energy physics, the researchers showed that a sample consisting of just 256 weighted events could reproduce the results of a scan that typically requires two million simulated events. This technique allows the generating parameters of the model to globally maximize the likelihood, meaning the small sample sits exactly where the perfect theoretical model predicts it should, rather than drifting due to the random fluctuations inherent in large simulations.

The researchers tested this idea using a toy model with a five-dimensional observable space, which mimics the complexity of real particle collision data. They trained a neural network to estimate the ratio between the signal they were looking for and a standard reference distribution. By applying this learned ratio to a fixed set of reference points, they assigned weights that effectively concentrated the statistical power of millions of events into a tiny, manageable group. When they ran their simulations, they found that the expected test statistic scans obtained with their 256-event sample were nearly identical to those obtained with the massive two-million-event sample. The results were stable and precise, with the small sample stabilizing much faster than the conventional method, which continued to show significant variations even with millions of events. This suggests that the new method can provide the same level of confidence in expected sensitivity calculations without the need for the enormous computational resources previously thought necessary.

Crucially, the researchers verified that their simplified model agreed with independent simulator-based experiments, confirming that the density ratios learned by the neural network were accurate. They generated hundreds of thousands of pseudo-experiments from both their weighted sample and independent simulator banks, finding that the distributions of discovery statistics matched closely. This agreement indicates that the method is not just a mathematical trick but a robust way to model reality, provided the underlying ratios are well-trained. While the study does not eliminate the need for large simulations to train and validate the initial neural networks, it dramatically reduces the size of the sample needed to build the expected test statistic scans used in the final analysis. This addresses a major bottleneck in the field, potentially saving hundreds of CPU hours and gigabytes of memory for experiments like those conducted by the ATLAS collaboration. The work demonstrates that by carefully choosing how to weight a small number of data points, scientists can achieve the precision of a massive dataset with a fraction of the effort, making the path to discovery more efficient without sacrificing accuracy.

=== SUMMARY ===

Technical Summary: Finite Asimov Sample Construction in Unbinned Neural Simulation-Based Inference

Problem Statement
In frequentist analysis utilizing Neural Simulation-Based Inference (NSBI), specifically Neural Ratio Estimation (NRE), calculating expected sensitivity often requires approximating an unbinned Asimov dataset. Traditionally, this is achieved using large weighted Monte Carlo (MC) samples. However, this approach presents two significant challenges:

Finite Sample Fluctuations: Even with large samples, finite MC fluctuations displace the likelihood maximum from the true generating parameters, leading to instability in expected test statistic scans.
Computational Cost: Evaluating likelihoods on millions of events for repeated scans is computationally expensive. For instance, the ATLAS experiment reports likelihood maximizations in this regime requiring $O(10\text{--}20)$ CPU hours and $O(100\text{--}500)$ GB of memory.

The authors address the need for a method that allows the integration sample to be chosen for scan accuracy rather than merely for approximate score cancellation, directly tackling the issue of finite-sample displacement.

Methodology
The paper proposes a construction for a finite weighted reference sample where the generating parameters globally maximize the weighted likelihood. The method relies on learned density ratios between a target distribution $p(x; \theta)$ and a parameter-independent reference distribution $p_{ref}(x)$ .

Ratio Estimation: NRE is used to estimate the ratio $r_{\psi}(x; \theta) \approx p(x; \theta) / p_{ref}(x)$ via a classifier $D_{\psi}$ .
Fixed Reference Points: A fixed set of reference points $X_M = \{(x_m, \omega_m)\}_{m=1}^M$ is drawn from $p_{ref}$ , where $\omega_m$ are weights summing to 1. These points remain fixed throughout the analysis.
Normalization and Intensity: For any parameter value $\theta$ , the method computes normalizers $Z_s(\theta)$ for each process component $s$ using the fixed points and learned ratios. The intensity function $h(x; \theta)$ is constructed by combining the expected yields $\lambda_s(\theta)$ and the ratios, normalized by $Z_s(\theta)$ .
Asimov Weight Assignment: At a pre-determined generating point $\theta_A$ , Asimov weights $w_m^A$ are assigned to the fixed points based on the finite intensity masses $v_m(\theta_A)$ .
Likelihood Construction: A weighted log-likelihood function $\ell_{A,M}(\theta)$ is defined using these weights. Crucially, the reference distribution cancels out in the profile negative log-likelihood ratio statistic, preserving the maximizers.

Key Contributions

Theorem 2.1 (Finite Sample Asimov Closure): The authors prove that if the intensity masses are strictly positive and the auxiliary constraints are maximized at the generating parameters, the generating point $\theta_A$ is a global maximizer of the constructed weighted log-likelihood for any sample size $M$ . This ensures that the Asimov dataset exactly recovers the generating parameters, eliminating the displacement caused by finite MC fluctuations.
Algorithmic Efficiency: The construction allows for the use of a very small number of integration points (e.g., $M=256$ ) to generate stable expected sensitivity scans, decoupling the scan accuracy from the need for massive sample sizes.
Handling of Systematics: The framework accommodates nuisance parameters and auxiliary constraints, with specific notes on how normalization systematics (signal strength) differ from shape systematics regarding the requirement for updating normalizers.

Results
The authors validate the method using a toy model inspired by high-energy physics (HEP) with a five-dimensional observable space, correlated Gaussian mixtures, and Gaussian detector smearing.

Stability and Accuracy: In scans at a signal strength of $\mu_A = 1$ $μ_{A} = 1$ , an Asimov dataset with only 256 weighted events closely reproduced the expected test statistic scans obtained with 2 million simulated reference events.
- The mean discovery statistic $q_{0,A}$ for $M=256$ was 3.20 (std dev 0.11), compared to 3.24 for the large reference sample.
- Conventional samples with $M=256$ failed to locate the minimum correctly (often hitting the boundary), and even at $M=2 \times 10^6$ , conventional fits showed a standard deviation of 0.09 across repetitions.
Distribution Agreement: The test statistic distributions derived from the small corrected Asimov samples agreed with independent simulator-based pseudo-experiments ( $10^5$ trials). This agreement confirmed that the learned density ratios were well-trained.
Computational Gain: The method significantly reduces the computational burden for building expected test statistic scans, requiring only a few hundred integration points comparable to the bins in a large binned analysis.

Significance and Claims
The paper claims that this construction addresses a major computational bottleneck in NSBI workflows, such as those used in the ATLAS experiment. By ensuring that the generating parameters globally maximize the likelihood on a finite sample, the method provides stable and accurate expected sensitivity scans with orders of magnitude fewer events than previously required.

However, the authors maintain a modest and rigorous stance regarding the limitations:

Validation Dependency: The agreement with the simulator depends entirely on the accuracy of the learned ratios. The method does not correct for errors in the learned ratios themselves; explicit validation against independent simulators remains necessary.
Training Requirements: The approach does not reduce the simulation size required to train and validate the neural estimators. The large datasets are still needed for the initial learning phase.
Asymptotic Nature: The calculation of discovery significance relies on asymptotic approximations (Wald assumptions) which are separate from the exact finite- $M$ theorem regarding the location of the maximum.
Tail Behavior: Residual differences in sparsely populated tails may arise from ratio errors, finite simulation banks, or the asymptotic approximation, necessitating simulator validation and potential calibration.

In summary, the paper demonstrates that a finite sample Asimov dataset, constructed via consistent ratio normalization, can replace massive weighted MC samples for the specific purpose of generating expected test statistic scans, provided the underlying density ratios are accurately learned and validated.

Technical Summary: Finite Asimov Sample Construction in Unbinned Neural Simulation-Based Inference

More like this