Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions

This paper demonstrates that the OmniLearned particle physics foundation model, pre-trained on high-energy collision data, can be effectively transferred to low-energy fixed-target neutrino experiments to outperform models trained from scratch on tasks like energy regression and pion classification, thereby validating the potential for detector-agnostic inference across vastly different energy scales and physics processes.

The Gist

Imagine you are trying to teach a robot to recognize different types of cars.

The Old Way: You would start with a blank robot. You'd show it pictures of a Ford, a Toyota, and a Honda, one by one, and say, "This is a Ford," "This is a Toyota." It would take a long time, and it would need thousands of examples just to learn the basics.

The New Way (This Paper): Imagine you first taught that same robot to be an expert on all vehicles. You showed it millions of pictures of trucks, motorcycles, buses, and race cars from every angle. It learned the fundamental rules of how wheels work, how engines sound, and how aerodynamics shape a vehicle.

Now, you want to teach it to identify a specific, rare type of boat. Instead of starting from scratch, you just say, "Hey, remember those rules about wheels and engines? Well, boats have hulls and propellers, but the logic of how they move through water is similar." Because the robot already understands the "physics of movement," it learns to spot the boat incredibly fast, with very few examples.

That is exactly what this paper is about.

The Cast of Characters

1. The "Super-Student" (OmniLearned): This is a massive AI model that was already trained on data from the world's biggest particle colliders (like the Large Hadron Collider). It has seen trillions of high-energy particle collisions. It's like a master chef who has cooked every dish in a 5-star restaurant.
2. The "New Kitchen" (MINERvA): This is a different experiment. Instead of smashing particles at near-light speed, it shoots a beam of neutrinos (ghostly particles) at a block of material to see how they bounce off. It's a much smaller, quieter, and very different "kitchen" than the collider.
3. The Challenge: The scientists wanted to see if the "Master Chef" (OmniLearned) could walk into the "New Kitchen" (MINERvA) and start cooking immediately, or if they had to teach it everything from the beginning.

The Big Hurdle: A Massive Gap

Usually, transferring knowledge is hard.

• The Collider: Imagine a massive explosion where hundreds of particles fly out in every direction (like a firework display). The energy is huge (Trillions of electron-volts).
• The Neutrino Experiment: Imagine a gentle tap where only a few particles bounce off (like a cue ball hitting a few other balls on a pool table). The energy is much lower.

It's like asking a Formula 1 race car driver to immediately drive a tractor in a muddy field. The machines are totally different, the terrain is different, and the physics of driving are different.

What They Did

The researchers took the pre-trained "Master Chef" (OmniLearned) and tried to use it on the neutrino data. They gave it two specific jobs:

1. The "Energy Meter" (Regression): Guessing how much energy was released in the crash.
2. The "Party Guest Counter" (Classification): Figuring out exactly what kind of particles came out of the crash (e.g., "Did we get a pion? Did we get a neutral pion?").

The Results: A Miracle of Transfer

The results were surprising and exciting.

• Speed: The pre-trained model learned the new task much faster than a model trained from scratch. It reached the same level of skill in half the time.
• Accuracy: Even with the same amount of computing power, the pre-trained model was more accurate.
• The "Inductive Bias": This is the fancy term for the "intuition" the model learned. Even though the collider and the neutrino experiment are different, the model learned fundamental rules about how particles move and interact in space. It learned that "particles tend to cluster in certain shapes" and "energy flows in predictable ways." These rules apply whether you are looking at a high-energy explosion or a low-energy bounce.

Why This Matters

Think of it like universal language learning.
If you learn English, Spanish, and French, you understand the concept of "grammar" and "sentence structure." If you then try to learn a completely new language, you don't start from zero; you just learn the new vocabulary because you already understand the structure of language.

This paper suggests that Particle Physics has a "Universal Grammar."

By training one giant AI model on diverse data, scientists can create a "Foundation Model" that can be adapted to any new experiment.

• For new experiments: They won't need to spend years training a new AI from scratch. They can just "fine-tune" the existing giant model.
• For the future: This could lead to a world where one single AI model can help analyze data from the Large Hadron Collider, the Deep Underground Neutrino Experiment, and even future telescopes, all with minimal retraining.

The Bottom Line

The scientists proved that an AI trained on the most violent, high-energy crashes in the universe can instantly become an expert at analyzing gentle, low-energy neutrino interactions. It's a huge step toward a future where we don't need to reinvent the wheel for every new physics experiment; we just need to teach the wheel how to roll on a new road.

Technical Summary

1. Problem Statement

The paper addresses the challenge of applying machine learning (ML) to few-GeV neutrino-nucleus scattering experiments (specifically the MINERvA experiment at Fermilab). These experiments face significant uncertainties in reconstructing neutrino energy and identifying interaction types due to complex nuclear effects and low particle multiplicities ($O(1-10)$ objects per event).

Traditionally, ML models for these tasks are trained from scratch on specific datasets, requiring large amounts of labeled simulation data and significant computational resources. The authors investigate whether Foundation Models—large-scale models pre-trained on diverse, high-energy collider data—can be effectively transferred to this "medium-distance" domain gap. The specific challenge involves bridging the gap between:

• Source Domain: High-energy ($TeV$) collider jets (e.g., $pp$ and $ep$ collisions) with high multiplicity ($O(10^2)$ particles), hermetic $4\pi$ detectors, and QCD fragmentation dominance.
• Target Domain: Low-energy ($few-GeV$) fixed-target neutrino interactions with low multiplicity, asymmetric detector geometry, and complex nuclear physics (quasi-elastic, resonance, deep-inelastic scattering, and nuclear effects).

2. Methodology

A. Dataset and Preprocessing

• Data Source: Simulated MINERvA Medium Energy Forward Horn Current (FHC) events from standard Monte Carlo playlists (1A and 1B), covering various nuclear targets (Helium, Water, Carbon, Iron, Lead).
• Event Representation: Events are converted into variable-length sets of tokens (reconstructed objects).
  • Tokens: Include muons, photons, prongs (directed energy deposits), and blobs (undirected calorimetric deposits).
  • Features: Each token includes kinematic variables ($\eta, \phi, \log p_T, \log E$) defined relative to the neutrino beam axis, a particle type ID, and 5 additional continuous features.
  • Global Features: 15 event-level features summarizing total energy, recoil, and specific reconstruction flags.
• Tasks:
  1. Regression: Predicting the Available Hadronic Energy ($E_{available}$), defined as the sum of kinetic energies of protons and full energies of photons, pions, electrons, and kaons.
  2. Binary Classification: Three distinct tasks identifying specific charged-current (CC) final states:
     • $CC1\pi^\pm$: Exactly one charged pion.
     • $CCN\pi^\pm$: $N \ge 1$ charged pions.
     • $CC1\pi^0$: Exactly one neutral pion (no charged pions).

B. Models Evaluated

The authors compare several architectures to isolate the effect of pre-training:

1. MLP (Baseline): A lightweight Multi-Layer Perceptron (284k params) using only global event features (proxy for classical reconstruction).
2. Transformer (Scratch): Vision Transformer (ViT)-style point-cloud transformers trained from scratch at xsmall (800k params) and small (3M params) scales.
3. OmniLearned (Foundation Model): The Particle Encoder Transformer v2 (PET2) pre-trained on diverse collider jet data.
   • OmniLearned-small (3M params): Fine-tuned from a pre-trained checkpoint.
   • OmniLearned-small-rw (3M params): A randomly initialized variant of the same architecture to control for architectural differences.
   • OmniLearned-medium (53M params): Pre-trained backbone frozen; only task-specific heads are trained due to compute constraints.

C. Training Strategy

• Optimizer: Adam with a learning rate of $10^{-4}$ and batch size of 2048.
• Loss Functions:
  • Regression: Smooth L1 Loss on $\log(1 + E_{available})$.
  • Classification: Cross-entropy with class weights to handle imbalance.
• Evaluation: Models are evaluated on validation sets every 1000 steps. Performance is measured against floating-point operations (FLOPs) and training steps to assess efficiency.

3. Key Contributions

1. Demonstration of "Medium" Transfer: The paper provides the first evidence that particle-level foundation models trained on high-energy collider jets can successfully transfer to low-energy neutrino interactions, despite vast differences in energy scale, detector technology, and underlying physics (QCD fragmentation vs. nuclear scattering).
2. Inductive Bias Discovery: The authors argue that pre-training instills geometric and kinematic inductive biases (specifically, attending to the relative positions and energies of sparse point clouds) that are universal across particle physics domains.
3. Efficiency Benchmarking: The study establishes that pre-trained models achieve superior performance with fewer training steps and lower computational budgets compared to models trained from scratch or classical baselines.
4. Open Science: The work leverages the MINERvA Open Data initiative, making the code and data publicly available to facilitate detector-agnostic inference research.

4. Results

A. Computational Efficiency

• FLOP Efficiency: Pre-trained OmniLearned models (specifically OmniLearned-small) achieve lower validation loss at the same cumulative FLOP budget compared to similarly sized models trained from scratch (Transformer-small) and the randomly initialized architecture (OmniLearned-small-rw).
• Step Efficiency: Pre-trained models converge faster, reaching the performance level of Transformer-small in approximately 45% fewer steps for classification and 50% fewer steps for regression.

B. Classification Performance

• Overall: Pre-trained models consistently outperform baselines across all three classification tasks ($CC1\pi^\pm$, $CCN\pi^\pm$, $CC1\pi^0$).
• Kinematic Dependence: The performance gain is most pronounced in low-energy and low-multiplicity regimes (e.g., low pion energy for $CC1\pi^\pm$ and low invariant mass $W$ for $CCN\pi^\pm$). This suggests the pre-trained representations are particularly effective at extracting information from sparse, difficult-to-reconstruct final states.
• Limitations: For the multi-pion task ($CCN\pi^\pm$) at high invariant mass ($W$), all models (including pre-trained ones) underperform compared to a simple cut-based baseline, indicating a need for further investigation in high-multiplicity regimes.

C. Regression Performance

• Energy Resolution: Pre-trained models provide a narrower Interquartile Range (IQR) in the residual distribution of available hadronic energy across all momentum transfer ($q_3$) bins, indicating more precise energy reconstruction.
• Bias: While resolution improves, a slight systematic bias (MPV slightly below unity) persists at higher $q_3$, likely solvable with different regression loss functions.
• Distribution Shape: Pre-trained models produce more symmetric residual distributions, particularly in the lowest $q_3$ bin where nuclear effects are strongest.

5. Significance and Outlook

• Detector-Agnostic Inference: The results point toward a paradigm where a single foundation model can be adapted to new experiments (e.g., DUNE, Hyper-Kamiokande) with minimal retraining, reducing the need for extensive simulation and labeled data during detector commissioning.
• Accelerated Physics Discovery: By lowering the cost of training specialized classifiers, foundation models enable rapid hypothesis testing and optimization of new experimental designs.
• Future Directions: The authors suggest that future foundation models could include neutrino events directly in their pre-training phase to further close the domain gap. They also highlight the potential for applying these techniques to other neutrino detectors like MicroBooNE.

In conclusion, this paper validates the hypothesis that particle-level foundation models learn universal physical priors that transcend specific experimental setups, offering a powerful tool for the next generation of precision neutrino physics.