⚛️ high-energy experiments

The Search Budget of the BSM Resonance Program

This paper quantifies the statistical trials factor of the ATLAS BSM resonance program by calculating that its current published record requires a local significance of 6.55σ for a 5σ global discovery, while a fully combinatorial scan would only marginally increase this requirement to 7.11σ, and further argues that two-stage unblinding is a superior safeguard against spurious signals in machine-learning searches where trials factors are difficult to enumerate.

Original authors: Theresa Reisch, Tobias Golling

Published 2026-08-21

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Theresa Reisch, Tobias Golling

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, high-energy collisions of the Large Hadron Collider, physicists are hunting for new particles that could rewrite the laws of nature. These searches often involve looking for a "bump"—a small, unexpected rise in the number of events at a specific mass—hidden within a smooth, predictable background of ordinary particle interactions. The challenge is that nature produces a staggering variety of these bumps, and researchers must scan a massive landscape of possible masses and particle combinations to find them. The more places they look, the higher the statistical bar they must clear to claim a discovery, because random fluctuations can mimic a signal if one looks hard enough in enough different spots. This is known as the "look-elsewhere effect," and it acts as a natural filter, ensuring that only the most robust signals are accepted as real discoveries.

A team of researchers at the University of Geneva has now mapped out exactly how much ground the ATLAS experiment has covered in its search for these new resonances, and what it would cost to scan every possible corner of the data. By counting the number of independent places the experiment has looked, they calculated the precise statistical threshold required to turn a local hint into a global discovery. Their analysis reveals that the current published searches, which cover a wide range of theoretical models, have already looked in roughly 7,900 distinct places. To claim a discovery today based on this existing record, a signal would need to be strong enough to correspond to a local significance of 6.55 standard deviations, rather than the traditional 5.

The study then asked a bolder question: what if the experiment scanned every single mass that could be built from up to four reconstructed particles, without being guided by specific theories? This hypothetical "combinatorial" scan would cover a much wider territory, amounting to 360,000 distinct looks. While this is a massive increase in the number of places to check, the statistical penalty is surprisingly small. The extra effort required to clear the bar for a discovery in this fully open scan would only raise the local significance requirement from 6.55 to 7.11. In other words, scanning 46 times more territory costs less than half a standard deviation in sensitivity. This finding suggests that the cost of being completely open-minded is far lower than many physicists feared, and that the program is not yet limited by the sheer number of places it has looked, but rather by the number of events available to fill the histograms.

However, the paper also identifies a new kind of risk that arises when using advanced machine-learning tools to find these bumps. Unlike traditional scans where the number of places looked is known and countable, some modern algorithms act as imperfect estimators that can occasionally produce false alarms. The authors found that if a machine-learning network flags a signal, the rate of these false alarms can be much higher than standard statistics would predict, effectively inflating the number of "looks" in a way that cannot be counted. To solve this, they propose a two-stage safety procedure. In the first stage, the algorithm scans a small fraction of the data to identify potential candidates. Only those specific candidates are then "unblinded" and checked against the remaining, unseen data. This method acts as a filter, ensuring that any signal that does not repeat in the second stage is discarded as a fluke. Their simulations show that this two-step approach is actually more sensitive than trying to correct for the errors in a single, massive scan, provided the machine learning tool has a known rate of mistakes.

The researchers also examined the specific landscape of known models to see how much of the territory remains unexplored. They found that while the published ATLAS program has scanned the vast majority of mass spectra motivated by current theories, there are still 51 specific combinations of particles that have been predicted by models but never searched for. These unscanned areas represent a small but distinct gap in the current coverage. The study concludes that the "search budget" for the resonance program is well-defined and manageable. The current published record is consistent with the number of random fluctuations expected, and the path forward involves either filling in the few remaining theoretical gaps or embracing the fully combinatorial scan, which remains statistically feasible and offers a powerful way to find the unexpected without needing to guess where to look.

Technical Summary: The Search Budget of the BSM Resonance Program

Problem Statement
Conventional Beyond the Standard Model (BSM) searches fix the model, final state, and mass window prior to data inspection, rendering the "look-elsewhere effect" (LEE) a minor correction. However, modern data-directed scans, machine-learning anomaly searches, and general searches sweep large portions of mass space simultaneously, causing the trials factor to dominate sensitivity. While individual analyses report their own trials factors, the total number of independent looks across the entire ATLAS BSM resonance program has never been enumerated. This lack of a global count obscures the true discovery threshold required for a 5 $\sigma$ global significance in agnostic or combinatorial searches.

Methodology
The authors construct a "search budget" by counting the number of effective independent resolution elements ( $N_{\text{trials}}$ ) across three distinct bases. The counting relies on the asymptotic Gross–Vitells relation, where the local significance ( $Z_{\text{local}}$ ) required for a 5 $\sigma$ global discovery is derived from the total trials:
$Z_{\text{local}} = \sqrt{25 + 2 \ln N_{\text{trials}}}$

The methodology defines a "look" as one resolution element ( $n_s$ ) within a mass window $[M_{\text{lo}}, M_{\text{hi}}]$ , calculated as $n_s = \frac{1}{r} \ln(M_{\text{hi}}/M_{\text{lo}})$ , where $r$ is the fractional mass resolution. The resolution is propagated from ATLAS object-performance measurements (e.g., calorimeter energy, muon sagitta, jet resolution).

The study evaluates three bases:

Model Space: Every spectrum where a public BSM model predicts a peak, regardless of whether a search exists. This is mapped from the FeynRules model database and literature to 63 distinct spectra.
Published ATLAS Program: A census of 290 resonance papers (2010–2026) grouped into 86 catalogued searches. This counts the actual "looks" already performed by the collaboration.
Hypothetical Combinatorial Scan: A scan of every invariant mass constructible from up to four reconstructed objects (electrons, muons, taus, photons, jets, etc.) across all exclusive event categories. This includes "selection lenses" (e.g., high $H_T$ , displaced activity) applied one at a time.

Crucially, the study imposes a "fittability" requirement: a spectrum is only counted if it contains at least 100 events and 25 resolution elements, based on a power-law background model ( $n(m) \propto m^{-6}$ to $m^{-8}$ ) scaled to Run 2 and Run 3 luminosity.

Key Contributions and Results

Global Discovery Thresholds:
- Published Program: The 104 charged (search, axis) pairs in the published record sum to $N_{\text{trials}} = 7.9 \times 10^3$ . This requires a local significance of $Z_{\text{local}} = 6.55$ for a 5 $\sigma$ global discovery.
- Full Combinatorial Scan: Scanning every fittable mass from up to four objects yields $N_{\text{trials}} = 2.0 \times 10^5$ (rising to $3.6 \times 10^5$ with event-level selection lenses). This increases the required local significance to $Z_{\text{local}} = 7.11$ .
- Cost of Breadth: The factor of 46 increase in looks (from the published program to the full combinatorial scan) costs only 0.56 $\sigma$ in local significance. The authors note that the statistics requirement (fittability) eliminates 80% of potential combinations before the trials factor becomes the limiting factor.
Validation of the Published Record:
The observed excess population in ATLAS data aligns with the calculated trials factor. The census predicts $\sim 11$ local $3\sigma$ fluctuations and 0.002 spurious $5\sigma$ fluctuations, matching the reported open excesses (largest $\sim 3.5\sigma$ local, none above $2.8\sigma$ global) and the absence of unexplained $5\sigma$ discoveries. This confirms that the published program scans almost exactly the axes motivated by known models.
Machine Learning and Imperfect Estimators:
For network-driven scans where the trials factor cannot be enumerated (e.g., BumpNet), the authors propose a two-stage unblinding procedure. By pre-registering windows exceeding a threshold ( $Z_{\text{cut}}$ ) in a fraction of the data and unblinding the remainder only there, spurious signals (which do not repeat) are filtered out.
- This safeguard converts unmeasured inflation into a countable list.
- At the measured defect rate of BumpNet, the two-stage procedure is more sensitive than a single-stage scan with an "honest" but inflated trials factor, despite a penalty of $\sim 0.5\sigma$ in discovery reach.
- The authors recommend selecting candidates via an absolute significance threshold rather than rank-based methods (like argmax or Benjamini-Hochberg) when the estimator is imperfect, as rank-based methods are highly susceptible to coherent defects (e.g., background mismodeling).

Significance
The paper establishes a quantitative "search budget" for the BSM resonance program, demonstrating that the transition from model-specific searches to fully agnostic, combinatorial scans is statistically manageable, requiring only a modest increase in local significance (from 6.55 to 7.11). It argues that the primary challenge in future wide scans is not the trials factor itself, but the reliability of the significance estimator. If the estimator is imperfect, the cost of a false discovery is high; therefore, the paper advocates for two-stage unblinding and absolute thresholding as necessary safeguards against spurious signals in the era of data-directed and machine-learning searches. The work provides a reproducible framework (via the associated repository) for calculating these budgets, ensuring that discovery claims in high-multiplicity scans are grounded in a rigorous accounting of independent looks.

More like this