Solving Key Challenges in Collider Physics with Foundation Models

This paper demonstrates how a new Foundation Model for hadronic jets addresses three critical challenges in collider physics—reducing computational costs for reconstruction, enabling comprehensive uncertainty quantification, and facilitating model-agnostic new physics searches—thereby transitioning jet-based Foundation Models from proof-of-concept studies to practical tools for researchers.

The Gist

Imagine you are trying to solve a massive, cosmic mystery. You have a giant microscope (the Large Hadron Collider) that smashes particles together to see what's inside. But the data coming out is like a chaotic storm of trillions of tiny pieces. Physicists need to sort this storm to find hidden patterns, like a needle in a haystack, or to understand the rules of the universe.

For years, scientists have used "Deep Learning" (super-smart computer programs) to help sort this data. But there's a problem: these programs are hungry. They need to eat massive amounts of data to learn, and creating that data is incredibly slow and expensive. It's like trying to teach a child to recognize a cat by showing them a million photos, but every time you take a photo, it costs a million dollars and takes a week to develop.

This paper introduces a new solution called OmniLearn, which acts like a "Foundation Model" for particle physics. Think of it as a super-genius student who has already read every book in the library before they even walk into your classroom.

Here is how OmniLearn solves three big problems using simple analogies:

1. The "Fast-Forward" Training (Saving Computing Power)

The Problem: Usually, to train a computer to recognize a specific type of particle (like a "Top Quark"), you need to simulate millions of collisions on a supercomputer. This takes forever and uses huge amounts of energy.
The OmniLearn Solution: Imagine you want to learn to drive a Ferrari. Instead of starting with a toy car and practicing for 10,000 hours, you hire a driving instructor who has already driven every car in the world. You just give them a few hours of practice in the Ferrari, and they are instantly ready to race.
The Result: OmniLearn was trained on a "fast simulation" (a rough sketch of reality). When the scientists gave it just 10% of the real, expensive data needed for a new task, it performed just as well as models trained on 100% of the data. This saves massive amounts of time and electricity.

2. The "Instant Translator" (Fixing Measurement Errors)

The Problem: When particles hit the detector, the machine gets "blurry" (like a camera with a dirty lens). Physicists need to "unblur" the image to see what really happened. Doing this mathematically is like trying to solve a giant puzzle where you have to try thousands of different pieces to see which one fits. It's slow and prone to errors.
The OmniLearn Solution: Think of OmniLearn as a translator who already knows the language of the "blurry" detector and the "clear" reality. Because it has seen so many examples before, it doesn't need to guess; it just knows the answer.
The Result: OmniLearn can fix these blurry measurements twice as fast as previous methods. It also gives scientists a much better idea of how confident they can be in the results, which is crucial for making sure they aren't seeing things that aren't there.

3. The "Super-Sleuth" (Finding New Physics)

The Problem: Sometimes, scientists are looking for something totally new (New Physics) that has never been seen before. They use "Anomaly Detection" to find weird things that don't fit the normal pattern. But if the "weird" signal is very faint (like a whisper in a hurricane), old computers can't hear it because they haven't seen enough examples of the "hurricane" to know what a whisper sounds like.
The OmniLearn Solution: OmniLearn is like a detective who has memorized the sound of every normal wind, rain, and storm in history. Because it knows the "background noise" so perfectly, it can instantly spot a whisper that is too quiet for anyone else to hear.
The Result: Using OmniLearn, scientists were able to find "whispers" (rare signals) that were twice as faint as what previous methods could detect. This means they can find new particles or forces that were previously invisible.

The Big Picture

The authors are saying: "Stop starting from scratch every time."

In the past, every time a physicist wanted to solve a new problem, they had to build a new computer brain from the ground up, feeding it data until it learned. With OmniLearn, they can start with a pre-trained "brain" that already understands the basics of particle physics. They just need to give it a little bit of specific data to "fine-tune" it for the job at hand.

Why does this matter?
It means scientists can do more science with less money and less time. They can build better tools, find rarer particles, and maybe one day, discover the secrets of the universe that have been hiding in plain sight. It's the difference between building a house brick-by-brick from scratch versus using a pre-fabricated, high-tech frame that you just customize.

Technical Summary

1. Problem Statement

Deep learning has revolutionized high-energy physics (HEP) by enabling holistic analysis of high-dimensional data without relying on low-dimensional summary statistics. However, deploying these methods at scale faces three critical bottlenecks:

• Data Scarcity & Simulation Costs: State-of-the-art models require massive datasets (tens of millions of events). Generating these via full detector simulation is computationally prohibitive, while fast simulations often lack the necessary accuracy for precision tasks.
• Computational Overhead in Uncertainty Quantification: Methods requiring likelihood ratio estimation (e.g., for unfolding or parameter estimation) often necessitate retraining models thousands of times to estimate uncertainties (bootstrapping/ensembling), which is computationally infeasible for full phase-space inference.
• Limited Sensitivity in Anomaly Detection: Current anomaly detection methods trained directly on data are limited by dataset size. High-dimensional methods struggle to detect rare signals unless the signal injection is already strong, limiting their ability to discover truly new physics.

The authors propose that Foundation Models (FMs)—neural networks trained on large, diverse datasets capable of being fine-tuned for various downstream tasks with minimal data—can overcome these limitations.

2. Methodology: OmniLearn

The core of the study is OmniLearn, a new foundation model for hadronic jets based on supervised representation learning.

• Architecture: OmniLearn utilizes a Point-Edge Transformer (PET) backbone.
  • It combines attention mechanisms and dynamic convolutional operations to capture both global and local descriptions of particles within a jet.
  • The architecture is modular, consisting of a shared "PET body" (the foundation) and task-specific "heads" (for classification or particle generation).
  • During downstream adaptation, irrelevant heads can be discarded, keeping the model compact (<2M parameters), allowing it to run efficiently on a single GPU.
• Training Data: The model was pre-trained on the JetClass dataset, containing 100 million jets across 10 different jet classes.
• Adaptation Strategy: The authors demonstrate that OmniLearn can be fine-tuned on small subsets of high-fidelity (full simulation) data to achieve state-of-the-art performance, effectively transferring knowledge from the large pre-training set to specific, data-scarce tasks.

3. Key Contributions & Results

The paper validates OmniLearn across three distinct challenges in collider physics:

A. Reducing Simulation Costs (Jet Tagging)

• Task: Top-quark tagging using the ATLAS dataset (full detector simulation with pileup).
• Approach: Fine-tuning OmniLearn on only 10% (4M events) of the full training dataset (40M events) compared to training from scratch.
• Results:
  • OmniLearn fine-tuned on 10% of the data achieved an AUC of 0.961, matching the performance of the best previous models (ParticleNet) trained on the full dataset.
  • When trained on the full 40M events, OmniLearn achieved an AUC of 0.965, outperforming all benchmarks.
• Significance: Demonstrates that foundation models can drastically reduce the computational cost of generating training data for new taggers without sacrificing accuracy.

B. Accelerating Unfolding (Uncertainty Quantification)

• Task: Unfolding detector distortions to recover true particle distributions (OmniFold algorithm).
• Challenge: OmniFold requires training $2^n$ networks (where $n$ is the number of iterations) to estimate uncertainties, often requiring tens of thousands of training runs.
• Approach: Using OmniLearn as a pre-trained initialization for the classification step in OmniFold.
• Results:
  • Speed: OmniLearn converged twice as fast as training from scratch, reducing the total training time by nearly a factor of 2.
  • Precision: OmniLearn achieved lower validation loss and better physics metrics (triangular distance) than both classical bin-based unfolding (IBU) and the standard OmniFold with DeepSets.
• Significance: Makes full uncertainty quantification for high-dimensional, unbinned data computationally feasible for the first time.

C. Enhancing Anomaly Detection

• Task: Resonant anomaly detection (finding a new boson $A \to B \to C$) in the LHC Olympics dataset.
• Approach: Using OmniLearn to generate a background reference and then distinguishing synthetic data from real data in the signal region (a CATHODE-like approach but using low-level inputs).
• Results:
  • OmniLearn detected signals with an initial significance of $S/\sqrt{B} \sim 2$ (approx. 600 injected events).
  • Previous methods (e.g., CATHODE) required signal injections with $S/\sqrt{B} \sim 4$ (approx. 1400 events) to be discoverable.
  • Training from scratch performed worse than previous studies due to the small signal-region dataset size (~100k events), highlighting the necessity of the foundation model pre-training.
• Significance: Proves that foundation models can push the sensitivity of model-agnostic anomaly detection to levels where rare, previously undetectable signals can be found.

4. Significance and Conclusion

This paper marks a paradigm shift in applying machine learning to particle physics:

1. From Scratch to Foundation: It moves the field away from training models from scratch for every new analysis. Instead, a single pre-trained foundation model can be adapted to diverse tasks (tagging, unfolding, generation, anomaly detection) with minimal data.
2. Resource Efficiency: It solves the "data hunger" problem of modern deep learning, allowing experiments with limited computing resources to achieve state-of-the-art results.
3. Scientific Potential: By enabling full uncertainty quantification and increasing anomaly detection sensitivity, OmniLearn unlocks the potential to analyze high-dimensional data more holistically, potentially leading to the discovery of new physics that was previously obscured by methodological limitations.

The authors conclude that OmniLearn is ready to join the toolkit of practitioners and envision a future where a library of foundation models enables all tasks in particle physics and beyond. All code and data are publicly available.