Sensitivity to New Physics Phenomena in Anomaly Detection: A Study of Untunable Hyperparameters

This paper systematically evaluates the sensitivity of four semi-supervised anomaly detection methods to untunable hyperparameters in the search for Beyond the Standard Model physics and proposes a robust, non-parametric permutation test for statistical assessment.

The Gist

Imagine you are a detective trying to find a single, tiny, invisible thief in a massive crowd of 10 million innocent people. You don't know what the thief looks like, you don't know what they are wearing, and you don't even know if they are actually there. You only know what the "normal" people look like.

This is exactly the challenge particle physicists face at the Large Hadron Collider (LHC). They smash protons together to create a storm of particles. Most of the time, these particles behave exactly as predicted by the "Standard Model" (the rulebook of physics). But sometimes, a new, unknown particle might appear—a "New Physics" signal. The goal is to spot this stranger without knowing what they look like in advance.

This paper is a study on how to build the best "spot-the-difference" tools (called Anomaly Detection algorithms) to find these strangers, specifically focusing on a tricky problem: How much does the tool's internal "knob" setting matter if you can't tune it?

Here is the breakdown of their findings using simple analogies:

1. The Tools: Four Different Ways to Spot the Thief

The researchers tested four different computer algorithms, each with a different way of thinking about "normal":

• Auto-Encoders (AE) & Deep-SVDD: Think of these as high-tech memory artists. They are trained to memorize the faces of the 10 million innocent people. When a new person walks in, the artist tries to draw them from memory. If the drawing looks nothing like the real person (a high "reconstruction error"), the artist screams, "Anomaly!"
• Isolation Forest (iForest): Imagine a game of "Cut the Cake." You keep slicing the crowd randomly. Normal people are in the thick of the crowd, so it takes many slices to isolate them. A thief standing alone on the edge gets isolated with just one or two slices. The algorithm counts how many cuts it took to isolate a person. Fewer cuts = more suspicious.
• Histogram-based Outlier Score (HBOS): This is like a census taker. They count how many people fall into specific categories (e.g., "wearing a hat," "holding a bag"). If a person falls into a category that is almost empty, they are flagged as an anomaly.

2. The Problem: The "Untunable" Knobs

Every one of these tools has a setting that is hard to adjust because you don't have a "test answer key" (since you don't know what the new physics looks like yet).

• For the Memory Artists, it's the size of their "sketchbook" (how much detail they can remember).
• For the Cake Cutter, it's the number of slices they are allowed to make.
• For the Census Taker, it's how many categories they create.

The researchers asked: "If we change these settings, does our ability to find the thief change drastically?"

3. The Findings: Surprising Stability

The study found something very reassuring: The tools are surprisingly robust.

• The "Goldilocks" Myth: You might think there is a perfect setting (not too big, not too small) for the sketchbook or the number of slices. The researchers found that for most signals, it doesn't matter much. Whether the sketchbook is small or huge, the artist still spots the thief about the same amount of time.
• Shallow vs. Deep: The simpler tools (iForest and HBOS) and the complex deep-learning tools (AE and Deep-SVDD) performed similarly. The complex tools didn't magically become much better just because they were "deeper."
• The "Best Feature" Rule: The study showed that these smart algorithms are basically just as good as the single best physical measurement you could take (like "how heavy is this particle?"). They manage to find the thief without needing to be told which measurement is the best one.

4. The Twist: How You Measure "Success" Matters

This is the most critical part of the paper. The researchers tried two different ways to judge if the tools were working:

• Method A (The Standard Score): They used a standard score called ROC AUC. This is like a teacher grading a test where they know the right answers.
  • Result: The tools looked great, and the settings didn't matter much.
• Method B (The Real-World Test): They used a Permutation Test with a new statistic called Cramér's (Cr). This is like a judge looking at two piles of evidence (one pile of known innocent people, one pile of mixed data) and asking, "Are these two piles statistically different?"
  • Result: This is where things got interesting. The Deep Learning tools (the Memory Artists) suddenly looked much better than the simple tools.
  • Why? The simple tools give scores that are "capped" (they can't go very high). The deep tools give scores that can go infinitely high if the anomaly is weird enough. The new statistical test (Cr) is very good at catching these extreme, long-tail outliers, while the old standard score missed them.

5. The Conclusion: Don't Bet on One Horse

The paper concludes with a few key takeaways for physicists:

1. Don't stress too much about the "knobs": Since the performance doesn't change wildly with different settings, you don't need to spend years trying to find the perfect setting for your anomaly detector.
2. Use the right ruler: If you want to find new physics, don't just use the standard "test score" (ROC AUC). Use the new statistical test (Cramér's) because it is better at spotting the weird, extreme outliers that deep learning tools find.
3. Combine your tools: Different tools spot different things. The "Memory Artist" (AE) and the "Deep Center Finder" (Deep-SVDD) sometimes spot different types of anomalies. Using them together is better than using just one.

In short: The paper tells us that these anomaly detection tools are sturdy and reliable. They don't need perfect tuning to work, but they do need the right statistical "ruler" to measure their success, and using a combination of different tools gives you the best chance of catching the invisible thief.

Technical Summary

Technical Summary: Sensitivity to New Physics Phenomena in Anomaly Detection

Problem Statement
The search for physics beyond the Standard Model (BSM) at collider experiments increasingly relies on model-independent strategies to avoid missing unexpected signals. While Anomaly Detection (AD) techniques have been extensively studied for identifying deviations from Standard Model (SM) distributions, the sensitivity of these methods to "untunable" hyperparameters has not been systematically compared. In semi-supervised settings, where models are trained solely on SM background data without access to signal labels, hyperparameters such as latent space dimensions or the number of bins cannot be optimized via standard validation metrics. Consequently, there is a lack of understanding regarding how these fixed parameters influence the ability of AD models to detect new physics. Furthermore, statistical interpretability remains a challenge, as anomaly scores often lack well-defined significance measures for signal-agnostic searches.

Methodology
This study investigates four semi-supervised AD methods trained exclusively on simulated SM background events (proton-proton collisions at $\sqrt{s}=13$ TeV, featuring two leptons, one bottom jet, and large $H_T$). The methods evaluated include:

1. Auto-Encoders (AE): Deep neural networks trained to minimize reconstruction error.
2. Deep Support Vector Data Description (Deep-SVDD): Deep networks mapping data to a hypersphere to minimize distance from a center.
3. Histogram-based Outlier Score (HBOS): A shallow method estimating probability density via feature histograms.
4. Isolation Forest (iForest): A tree-based method isolating anomalies via random partitions.

The models were tested against six diverse BSM benchmark signals (Heavy Vector-like Quarks, Flavour Changing Neutral Currents, Randall-Sundrum radion, Two-Higgs-Doublet Model, and Left-Right Symmetric Model).

The analysis proceeds in two stages:

1. Hyperparameter Sensitivity: The authors assess the sensitivity of each method to specific untunable hyperparameters (e.g., latent space dimension for AE/Deep-SVDD, number of estimators for iForest, number of bins for HBOS) using the Receiver Operating Characteristic Area Under the Curve (ROC AUC) as a discrimination metric.
2. Statistical Significance: To address the lack of signal labels in real searches, the paper proposes a non-parametric permutation test using signal-agnostic statistics. Two test statistics are introduced:
   • $M_\Delta$: The maximum difference between empirical cumulative distribution functions (eCDFs), inspired by the Kolmogorov-Smirnov test.
   • Cramér's statistic ($Cr$): The integral of the squared difference between eCDFs, noted for its sensitivity to distribution tails.
     The permutation test evaluates the null hypothesis ($H_0$) that the analysis sample (data) and control sample (SM simulation) originate from the same distribution.

Key Contributions

• Systematic Hyperparameter Analysis: The paper provides a comparative study of how untunable hyperparameters affect the performance of four distinct AD architectures across multiple BSM scenarios.
• Decoupling Reconstruction from Sensitivity: The study demonstrates that for Auto-Encoders, improved background reconstruction quality (measured by $R^2$) does not necessarily correlate with improved signal discrimination. Sensitivity depends on the relative difference in reconstruction error between signal and background rather than the absolute quality of background reconstruction.
• Signal-Agnostic Statistical Framework: The authors introduce a robust statistical testing framework using permutation tests and the $Cr$ statistic. This allows for the assessment of new physics evidence without prior knowledge of the signal hypothesis, addressing the limitations of ROC AUC in signal-agnostic contexts (e.g., insensitivity to symmetric distributions).

Results

• Hyperparameter Stability: Across most BSM signals and AD methods, the choice of untunable hyperparameters resulted in negligible variation in ROC AUC. The semi-supervised methods generally performed as well as the single most discriminating feature for each signal, regardless of the specific hyperparameter configuration.
• Metric Divergence: While shallow methods (HBOS, iForest) often outperformed Deep-SVDD in terms of ROC AUC, the permutation test using the $Cr$ statistic revealed that deep learning methods (AE and Deep-SVDD) achieved lower p-values (higher sensitivity) for many signals. This discrepancy is attributed to the long-tailed nature of deep learning anomaly scores, which the $Cr$ statistic captures effectively, whereas the bounded scores of shallow methods and the $M_\Delta$ statistic do not.
• Test Statistic Efficacy: The $M_\Delta$ statistic failed to produce evidence for new phenomena (median p-values $> 0.05$) across all signals and methods. In contrast, the $Cr$ statistic successfully identified deviations, particularly for deep learning models, highlighting the critical importance of selecting an appropriate test statistic for the discriminant domain.
• Complementarity: The results indicate sensitivity complementarity between AE and Deep-SVDD, suggesting that different AD methods capture different notions of anomalies.

Significance and Claims
The paper claims that the choice of untunable hyperparameters in semi-supervised AD models significantly impacts search sensitivity, though this impact is not always monotonic or predictable via standard metrics like ROC AUC. The authors argue that relying on a single model or metric is insufficient; instead, strategies aggregating results from models with varying hyperparameters should be explored.

Crucially, the work establishes a pathway for purely semi-supervised searches by introducing a statistical test capable of rejecting the "SM-only" hypothesis without signal-specific assumptions. The authors modestly conclude that while their permutation test and $Cr$ statistic offer a robust method for quantifying deviations, the "no free lunch" theorem applies: no single AD model or hyperparameter configuration outperforms all others for every task, necessitating diverse methodological approaches in future searches.