Shapes are not enough: CONSERVAttack and its use for finding vulnerabilities and uncertainties in machine learning applications

This paper introduces the CONSERVAttack, a novel adversarial method designed to expose hidden vulnerabilities and uncertainties in machine learning models used in High Energy Physics by exploiting deviations between simulation and data that evade standard physical validation checks, while also proposing strategies to mitigate these risks.

The Gist

Imagine you are a master chef trying to perfect a new recipe for a soup that predicts the weather. You have two bowls of ingredients:

1. The Real Soup: Actual data from the real world (real weather patterns).
2. The Simulated Soup: A computer-generated recipe that should taste exactly like the real thing.

In the world of High Energy Physics (the study of tiny particles), scientists use "Deep Learning" (super-smart computer brains) to taste these soups and tell the difference between a "Signal" (a rare, exciting discovery) and "Background" (boring, common noise).

For decades, scientists have checked their work by tasting the soup for obvious flaws: "Is the salt level right? Is the temperature consistent?" These are like checking the marginal distributions (the average taste of each ingredient) and linear correlations (does the salt usually go with the pepper?).

The Problem:
The paper argues that these standard taste tests aren't enough. A clever chef could tweak the soup in a very subtle, complex way that changes the overall flavor profile just enough to trick the computer brain, but keeps the salt and pepper levels looking perfectly normal. The computer thinks, "This tastes like a Signal!" but it's actually a fake.

This is where the CONSERVAttack comes in.

The CONSERVAttack: The "Ghost Chef"

The authors created a new type of "Ghost Chef" (an adversarial attack). This chef's goal is to sneakily alter the Simulated Soup so that:

1. It tricks the computer: The computer brain misidentifies the soup (e.g., calling a "Background" soup a "Signal").
2. It passes the taste test: The salt, pepper, and temperature levels remain statistically identical to the original. The standard checks say, "Everything is fine!"

The Analogy: Imagine a spy trying to sneak into a high-security building.

• Standard Checks: The guard checks your ID badge and your height.
• The Attack: The spy wears a perfect mask (hiding their face) and stands on a box (hiding their height). To the guard, everything looks normal. But the spy is still a threat.
• The Result: The CONSERVAttack shows that even if your "ID" and "height" are perfect, the computer brain can still be fooled by subtle, invisible changes to the "shape" of the data.

Why Does This Matter?

In particle physics, if a computer is fooled, scientists might think they found a new particle when they didn't, or miss a real discovery. This creates a "hidden uncertainty." The paper suggests we need to measure how easily our computers can be tricked by these "Ghost Chefs" to know how much we can really trust our results.

The Solutions: How to Defend the Kitchen

The paper doesn't just point out the problem; it offers two ways to fix the kitchen:

1. Adversarial Training (The "Spicy Soup" Method)
Instead of just teaching the computer brain with normal soup, the chefs start adding "Ghost Chef" soups to the training menu. They say, "Here is a soup that looks normal but is actually a trick. Learn to spot it!"

• Result: The computer brain becomes tougher and less likely to be fooled in the future.
• Bonus: Surprisingly, this also makes the computer better at tasting real soup, even if it hasn't seen the tricks before. It's like training a dog with difficult obstacles; it becomes better at navigating the whole park.

2. The Adversarial Detector (The "Sniffer Dog")
Instead of trying to make the main computer brain un-foolable, they train a second, specialized dog (a detector network).

• How it works: This dog doesn't care if the soup is Signal or Background. Its only job is to sniff out: "Is this soup a trick?"
• Result: Before the main computer makes a decision, the Sniffer Dog checks the soup. If it smells a "Ghost Chef," it flags it. This catches the tricks that the main brain missed.

The "Donut" Example

To make this clear, the authors used a simple toy example called the "Donut."

• Signal: A circle of dots in the center.
• Background: A donut shape of dots surrounding the center.
• The Attack: The Ghost Chef pushes some donut dots into the center circle. To the naked eye (and standard checks), the donut still looks like a donut. But the computer now thinks those pushed dots are part of the center circle.
• The Detector: The Sniffer Dog learns to see that these pushed dots have a weird "history" or shape that doesn't quite fit, even if they look like they belong in the center.

The Big Takeaway

The paper concludes with a new workflow for scientists:

1. Test: Try to trick your computer model with the CONSERVAttack.
2. Defend: Use a Sniffer Dog (Adversarial Detector) to catch the tricks.
3. Decide:
   • If the Sniffer Dog catches almost all the tricks, and the remaining "fooling" rate is tiny, you can be confident your results are solid.
   • If the Sniffer Dog fails to catch many tricks, or if the "fooling" rate is huge, you have to admit, "Hey, our model is vulnerable!" You then have to add a "safety margin" (uncertainty) to your scientific results to account for this risk.

In short: Just because a computer says "It's safe!" doesn't mean it is. We need to actively try to break our own models to find the hidden cracks, and then build stronger defenses to ensure our discoveries in the universe are real.

Technical Summary

1. Problem Statement

In High Energy Physics (HEP), deep learning models are increasingly used for tasks like event classification, regression, and simulation. Standard validation procedures in HEP rely heavily on comparing marginal distributions (1D histograms) and linear pairwise correlations between simulated training data and real experimental data in "control regions."

The authors argue that these standard checks are insufficient because:

• They do not probe the full complexity of high-dimensional, non-linear decision boundaries used by neural networks.
• They fail to account for hypothetical sources of mismodeling that are unknown, overlooked, or physically unmotivated.
• A model could be robust to changes in marginals and linear correlations but still be highly sensitive to specific, subtle perturbations in the high-dimensional feature space.

This gap creates a potential "blind spot" where adversarial perturbations could exist that fool the classifier while remaining statistically indistinguishable from real data under standard HEP validation checks.

2. Methodology: CONSERVAttack

The paper introduces CONSERVAttack, a novel adversarial attack designed specifically to exploit the remaining space of deviations between simulation and data after standard physical checks.

Core Objective:
Construct adversarial perturbations that induce misclassification (fooling the model) while strictly preserving:

1. Marginal feature distributions.
2. Inter-feature correlations (specifically linear correlations, though later extended to non-linear).

Technical Implementation:

• Optimization Strategy: Unlike standard attacks (e.g., PGD) that constrain perturbations per-event (e.g., $L_\infty$ norm), CONSERVAttack optimizes at the dataset level. It iteratively searches for candidate perturbations that minimize a custom loss function:
  $$L := \alpha \cdot \text{JSD} + \beta \cdot \Delta_{FN}$$
  Where:
  • JSD (Jensen-Shannon Distance): Measures the divergence between the marginal distributions of clean and perturbed data.
  • $\Delta_{FN}$ (Relative Frobenius Norm): Measures the difference between the correlation matrices of clean and perturbed data.
• Gradient Usage: The attack uses the sign of the gradient of the model's loss to generate candidate perturbations but discards the magnitude, focusing on dataset-level statistical consistency.
• Constraints: The algorithm enforces user-defined thresholds for maximum allowed JSD and $\Delta_{FN}$, ensuring the perturbations remain "invisible" to standard statistical sanity checks.

3. Key Contributions

1. New Adversarial Attack (CONSERVAttack): A method to generate adversarial examples that are statistically consistent with standard HEP validation metrics but successfully fool deep learning models.
2. Quantification of Systematic Uncertainty: The attack provides a new metric (Fooling Ratio under statistical constraints) to estimate an upper bound on a model's systematic vulnerability to unknown mismodeling.
3. Data Augmentation: Demonstrated that adversarial examples generated by CONSERVAttack can be used as a data augmentation strategy to improve model performance in low-data regimes.
4. Defense Strategies:
   • Adversarial Training: Retraining models with generated adversarial examples.
   • Adversarial Detector: Training a secondary binary classifier to distinguish between "clean" and "adversarial" events, effectively filtering out vulnerable inputs.
5. Extension to Non-Linear Dependencies: The method was extended to preserve Distance Correlation (capturing non-linear dependencies), proving that the vulnerability exists even when stricter statistical constraints are applied.

4. Experimental Results

The authors evaluated the approach on two HEP tasks:

1. Higgs Boson Classification: Distinguishing Higgs events from background.
2. Jet Tagging: Distinguishing Top-quark jets ($TT$) from W-boson jets ($WW$).

Key Findings:

• High Fooling Ratio: The attack achieved a Fooling Ratio (FR) of ~0.89 on the Higgs dataset and ~0.675 on the Jet Tagging task. This means nearly 90% of inputs were misclassified.
• Statistical Invisibility: Despite the high FR, the perturbations resulted in negligible changes to statistical properties:
  • Average JSD remained below 0.02.
  • Average $\Delta_{FN}$ remained below 0.2.
  • Visual inspection of distributions and correlation matrices showed no significant deviation from clean data.
• Defense Efficacy:
  • Adversarial Training: Reduced the fooling ratio to ~0.15–0.20.
  • Adversarial Detector: Reduced the fooling ratio further to ~0.05–0.08, demonstrating superior robustness.
• Real-Data Generalization: The Adversarial Detector, trained solely on simulated data, generalized well to real CMS collision data (2012 Single Mu dataset), maintaining high efficiency in identifying clean events.
• Systematic Misclassification: Analysis of the detector revealed that a statistically significant subset of "clean" simulated events consistently exhibited adversarial-like behavior, suggesting inherent structural similarities between certain clean events and adversarial perturbations.

5. Significance and Implications

• Redefining Uncertainty: The paper argues that robustness to adversarial effects must be considered a component of systematic uncertainty in HEP. If a model can be fooled without violating statistical constraints, the current uncertainty estimates may be underestimated.
• Workflow Proposal: The authors propose a practical workflow for HEP analyses:
  1. Train a baseline model.
  2. Generate CONSERVAttack examples.
  3. Train an Adversarial Detector.
  4. Calculate the "Corrected Fooling Ratio" (after filtering by the detector).
  5. Decision Rule: If the corrected fooling ratio is within the bounds of known physical systematic uncertainties, no additional uncertainty is needed. If it exceeds these bounds, it indicates unaccounted-for mismodeling or requires assigning additional uncertainty.
• Beyond HEP: While focused on HEP, the methodology applies to any scientific domain where deep learning is used on simulated data and where rigorous statistical validation is required (e.g., climate modeling, medical imaging).
• Limitations: The method relies on large statistics to maintain distributional constraints and is less effective on datasets dominated by categorical variables.

In conclusion, CONSERVAttack reveals that "shapes" (marginals and linear correlations) are not enough to guarantee model reliability. It provides a rigorous framework to quantify and mitigate hidden vulnerabilities in scientific machine learning applications.