⚛️ high-energy experiments

Look everywhere effects in anomaly detection

This paper investigates the "look everywhere" effect in machine learning-based anomaly detection, demonstrating that while training and testing on the same data leads to miscalibrated $p$ -values, strategies like k-folding offer an effective balance between statistical calibration and sensitivity to new physics.

Original authors: Marie Hein, Benjamin Nachman, David Shih

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marie Hein, Benjamin Nachman, David Shih

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to find a single, tiny clue hidden inside a massive library filled with millions of books. In the world of particle physics, this "library" is the data collected by giant machines like the Large Hadron Collider (LHC), and the "clue" is a sign of new, unknown physics.

For a long time, detectives had a specific list of clues they were looking for. But now, they are using Artificial Intelligence (AI) to scan the entire library at once, hoping to spot any weird pattern that doesn't fit the normal rules. This is called Anomaly Detection.

This paper, written by Marie Hein, Benjamin Nachman, and David Shih, tackles a tricky problem with this "scan everything" approach. They call it the "Look Everywhere Effect."

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Look Everywhere" Trap

Imagine you are looking for a needle in a haystack.

The Old Way: You decide to look only in the top 10% of the haystack. If you find a needle there, it's a big deal.
The New AI Way: You tell the AI, "Look at the entire haystack, every single straw, and tell me which spot looks the most suspicious."

The problem is that if you look at everywhere at once, you are bound to find a spot that looks suspicious just by pure luck (a random clump of straw). If you pick that "lucky" spot and say, "Aha! I found a needle!" without adjusting your math, you are likely wrong. You've been fooled by a statistical fluke.

In the paper, the authors show that when AI models train and test on the same data, they get "overconfident." They memorize the random noise (the lucky clumps) instead of learning the real structure. This leads to miscalibrated p-values.

Translation: The AI thinks it has found a 99.9% certain discovery, but it's actually just a random coincidence.

2. The Solutions and Their Trade-offs

The authors tested three different ways to handle this "Look Everywhere" problem. Think of these as different ways to organize your detective team.

Option A: The "Same Team" Approach (Training and Testing on the same data)

How it works: You give the AI all the data to study and then ask it to solve the case using that same data.
The Result: The AI is very sensitive (it finds the needle quickly), but it is lying to you. It thinks it found a needle when it just found a random clump. The math is broken.
Analogy: It's like a student taking a test using the answer key they just memorized. They get a perfect score, but they haven't actually learned the material.

Option B: The "Split Team" Approach (Training on half, Testing on the other half)

How it works: You split the data in two. The AI studies the first half, then you test it on the second half, which it has never seen.
The Result: The math is perfect. If the AI finds a needle, it's real. No lying.
The Catch: Because the AI only studied half the data, it's less sensitive. It might miss a real needle because it didn't have enough practice.
Analogy: The student studies half the textbook and takes the test on the other half. They get a fair grade, but they might miss a question they could have answered if they had studied the whole book.

Option C: The "K-Folding" Approach (The Middle Ground)

How it works: You split the data into five (or more) groups. The AI studies four groups and tests on the fifth, then rotates so every group gets a turn being the test group. Finally, you combine the results.
The Result: This strikes a sweet spot. It uses almost all the data (so it stays sensitive) but rotates the testing so the AI doesn't just memorize the noise.
The Catch: It's not perfectly calibrated like Option B, but it's much better than Option A. The authors found this to be the best balance for most situations.

3. The "Overfitting" Connection

The paper also connects this to a concept called Overfitting (or "Overtraining").

The Analogy: Imagine a student who studies for a test by memorizing the specific random typos in their practice book. When they take the real test, they get confused because the real book doesn't have those typos.
The authors found that the "Look Everywhere Effect" is essentially the same thing as overfitting. The AI is memorizing the random noise of the training data instead of the real physics.
The Fix: They used a technique called Early Stopping (telling the AI to stop learning before it gets too obsessed with the noise). This helped fix the math errors, especially for Neural Networks, but didn't work as well for a different type of AI called BDTs (Boosted Decision Trees).

4. The Final Verdict

The authors conclude that there is no perfect, free lunch.

If you want perfect math (calibration), you have to sacrifice some sensitivity (you might miss the signal).
If you want maximum sensitivity, you have to accept some math errors and fix them later.
The Winner: The K-Folding method (Option C) is the best compromise. It keeps the AI sensitive enough to find new physics while keeping the math errors small enough to be manageable.

In summary: When using AI to search for new physics by looking "everywhere," you have to be careful not to let the AI trick itself with random noise. The paper shows that splitting the data smartly (K-Folding) is the best way to keep the search both sensitive and honest.

Technical Summary: Look Everywhere Effects in Anomaly Detection

Problem Statement
Machine learning (ML)-based anomaly detection (AD) methods offer a model-agnostic approach to searching for new physics in high-dimensional spaces, complementing traditional searches with reduced theory bias. However, these methods face a statistical challenge analogous to the "look elsewhere effect" (LEE) found in classical searches. In traditional searches, the LEE arises from testing a pre-determined set of signal regions, requiring a "trials factor" to correct p-values. In ML-based AD, the detector effectively searches "everywhere" simultaneously across the phase space without predefined discrete regions. This creates a "look everywhere effect" where the statistical power is diluted, and p-values become miscalibrated if the detector overfits to statistical fluctuations in the training data. The paper investigates the trade-off between achieving perfectly calibrated p-values and maintaining high sensitivity to anomalies, a tension that arises from how data is partitioned for training and evaluation.

Methodology
The authors employ a systematic statistical study using two primary datasets:

Toy Model: A 2D Gaussian distribution with known background and signal templates, allowing for precise analytical control over statistical effects.
Realistic Benchmark: The LHC Olympics 2020 R&D dataset, featuring a QCD dijet background and a $W'$ resonance signal, processed with standard jet reconstruction and feature sets.

The study evaluates three distinct data usage strategies for weakly supervised classifiers (specifically Neural Networks and Boosted Decision Trees):

Full Data (Train/Test on same set): Maximizes statistics but introduces a significant trials factor, leading to overfitting on fluctuations.
Independent Split (50/50): Uses half the data for training and half for testing. This eliminates the trials factor and ensures calibration but reduces the effective training statistics, potentially lowering sensitivity.
k-Fold Cross-Validation ( $k=5$ ): Trains on $k-1$ folds and evaluates on the remaining fold, repeating for all folds. This aims to utilize full statistics for evaluation while mitigating overfitting, though the authors investigate whether the resulting p-values remain independent.

The analysis compares the empirical cumulative distribution of p-values against calculated p-values to assess calibration. It also evaluates signal sensitivity using the Significance Improvement Characteristic (SIC, $\epsilon_S/\sqrt{\epsilon_B}$ ) and examines the relationship between the look everywhere effect and overtraining (overfitting).

Key Contributions and Results

Identification of the Look Everywhere Effect: The paper demonstrates that training and testing on the same dataset results in severely miscalibrated p-values. The anomaly detector effectively selects the region with the largest statistical fluctuation, reporting a local p-value as a global one.
Overfitting Equivalence: The authors establish a direct link between the look everywhere effect and overtraining. Both phenomena stem from the model relying on statistical fluctuations of the training set rather than the underlying data distribution.
Impact of Regularization:
- Neural Networks (NN): Applying early stopping significantly reduces miscalibration when evaluating on the training set and can render k-fold cross-validation results perfectly calibrated.
- Boosted Decision Trees (BDT): Early stopping is less effective for BDTs. Due to the nature of BDTs (grouping events and averaging values based on the training set), they remain sensitive to training fluctuations even with early stopping, leading to persistent miscalibration in k-fold scenarios.
The Sensitivity-Calibration Trade-off:
- Independent Test Sets: Provide perfectly calibrated p-values but suffer from reduced sensitivity because the classifier is trained on fewer events.
- Full Data Training: Offers the highest sensitivity (largest training set) but produces uncalibrated p-values that require correction.
- k-Fold Cross-Validation: Strikes an effective balance. While not perfectly calibrated in all regimes (particularly for large excesses where the same fluctuation appears in multiple folds), the miscalibration is limited. The authors find that k-folding allows for high sensitivity while keeping p-value deviations manageable, often requiring only minor corrections.
Calibration Necessity: The study shows that to achieve optimal sensitivity, one may need to accept slight miscalibration and apply post-hoc calibration fits (derived from background-only simulations) rather than strictly enforcing independence which sacrifices statistical power.

Significance
The paper argues that the "look everywhere effect" is a fundamental statistical property of modern anomaly detection that must be explicitly managed. It highlights that there is no single "best" strategy; rather, the choice of data partitioning (full, split, or k-fold) and regularization (early stopping) dictates a specific trade-off between p-value accuracy and signal sensitivity.

The authors emphasize that while calibration is essential, it must be tested in a setup as close to the final analysis as possible, noting that small changes in architecture (e.g., switching from NN to BDT) can drastically alter statistical properties. The findings suggest that k-fold cross-validation is a robust practical approach for experimental searches, offering a compromise that maintains high sensitivity while keeping statistical dilution within acceptable bounds. The work provides a framework for understanding how ML-based searches can be rigorously calibrated without sacrificing the discovery potential inherent in using the full dataset.