Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training

This paper proposes a self-supervised sparse ViT framework that leverages masked autoencoding and relational objectives to learn reusable representations from heterogeneous neutrino detector data, significantly improving reconstruction performance and data efficiency for energy-frontier experiments like FASERCal while demonstrating strong transferability across diverse detector technologies.

The Gist

Imagine you are trying to understand a massive, chaotic traffic jam in a futuristic city, but you only have a handful of police reports (labeled data) to figure out what happened. The cars are moving so fast and are so tightly packed that they look like a single, solid blob of metal. Traditional methods of looking at the traffic report fail because the details are too crowded.

This is exactly the problem physicists face with neutrinos (ghostly subatomic particles) hitting detectors at the energy frontier. These particles smash into matter with such force that they create "traffic jams" of energy so dense and overlapping that standard computer programs can't make sense of them.

Here is a simple breakdown of what this paper proposes, using some everyday analogies:

1. The Problem: The "Black Box" Traffic Jam

In the past, physicists could look at a neutrino collision and say, "Ah, that's a muon, and that's an electron." But at the new, high-energy levels (like those at the Large Hadron Collider), the collisions are so messy that the signals overlap. It's like trying to identify individual instruments in a symphony where everyone is playing the same note at the same time, loudly, and the microphones are broken.

Usually, to teach a computer to solve this, you need thousands of examples where a human has already labeled every single particle. But getting those labels is expensive and slow.

2. The Solution: The "Self-Taught Intern" (Foundation Model)

The authors propose a new way to train AI, similar to how humans learn. Instead of just memorizing answers to specific questions (like "What is this particle?"), they let the AI study the raw data first, without any labels.

Think of it like this:

• Old Way (Training from Scratch): You hand a student a textbook with the answers hidden and say, "Here are 10,000 questions. Memorize the answers." This takes forever and requires a huge library.
• New Way (Self-Supervised Pre-training): You give the student a massive library of books but cover up 75% of the text. You say, "Read the visible parts and guess what the missing words are." The student learns the structure of the language, the grammar, and the context just by trying to fill in the blanks.

In this paper, the AI is a Vision Transformer (a type of AI good at seeing patterns). It looks at the detector data and tries to "reconstruct" the missing parts of the collision. By doing this, it learns the "grammar" of particle physics: how energy flows, how particles scatter, and how they overlap.

3. The Secret Sauce: "Relational" Clues

The researchers didn't just stop at guessing missing text. They added a second layer of learning called Relational Objectives.

Imagine you are looking at a crime scene photo.

• Task 1 (Reconstruction): "Fill in the missing parts of the photo."
• Task 2 (Relational): "Look at this specific spot. Is it a shadow (a ghost signal)? Is it the main suspect (primary particle) or a bystander (secondary particle)? Is it a weapon (hadronic) or a tool (electromagnetic)?"

By forcing the AI to answer these specific questions about the relationships between particles while it's learning to fill in the blanks, it becomes much smarter at understanding the messy, crowded parts of the collision.

4. The Results: Superpowers with Fewer Clues

Once this "intern" has studied the library (pre-training), they are ready for the real job. The researchers tested them on specific tasks:

• Identifying the particle type: Is it an electron, a muon, or a tau?
• Finding the crash site: Where exactly did the collision happen?
• Measuring the speed: How much energy was involved?

The findings were impressive:

• Less Data Needed: The pre-trained AI could do the job of a "fresh graduate" (trained from scratch) using only 1,000 labeled examples instead of 10,000. It's like a student who reads a whole library being able to pass a test with just a few practice questions.
• Better at the Hard Stuff: The AI was especially good at the most confusing, crowded collisions where other methods fail.
• Traveling Skills: The best part? This AI learned general "physics intuition." When they took the same brain and showed it data from a completely different type of detector (like a liquid argon tank instead of a plastic scintillator), it still performed better than models trained from scratch on that new data. It's like a chef who learned to cook in a French kitchen and immediately excelled in a Japanese kitchen because they understood the principles of cooking, not just the recipes.

5. Why This Matters

This paper suggests a new path forward for particle physics. Instead of building a new, specialized AI for every single experiment (which is slow and expensive), we can build a Foundation Model.

Think of it as a "Universal Physics Translator." We train it once on massive amounts of simulated data, and then we can fine-tune it quickly for any new experiment, even if we don't have a lot of labeled data. This makes it possible to study the most extreme, high-energy events in the universe that were previously too messy to analyze.

In short: They taught an AI to read the "language" of particle collisions by playing a game of "fill-in-the-blanks" on millions of simulated crashes. Now, that AI can solve real-world physics puzzles faster, with less data, and even understands different types of detectors.

Technical Summary

1. Problem Statement

Accelerator-based neutrino physics is entering an energy-frontier regime (TeV scale), characterized by interactions that produce exceptionally dense, collimated, and overlapping detector signatures.

• The Challenge: Conventional reconstruction algorithms fail in this regime due to the complexity of event topologies. Even supervised machine learning models trained from scratch struggle, particularly when labeled data is scarce and the analysis must span diverse downstream objectives (classification, regression, vertex reconstruction).
• The Specific Context: The paper focuses on the FASERCal concept at the LHC, a proposed upgrade featuring a highly granular 3D calorimeter (3DCal) with over 460,000 readout voxels, followed by electromagnetic/hadronic calorimeters and a muon spectrometer.
• Data Characteristics: The data is heterogeneous (sparse 3D volumetric hits, dense global summaries, variable-length track data) and sparse (only a fraction of voxels are active per event). The goal is to learn reusable representations that can handle these complex inputs without requiring massive labeled datasets for every new task.

2. Methodology

The authors propose a Sparse Vision Transformer (ViT) framework designed for heterogeneous detector data, utilizing a two-stage training strategy.

A. Architecture: Sparse Multimodal Encoder

• Input Processing: Uses Sparse 3D Convolutions (SpConv) to convert 3DCal and AHCAL voxel grids into patch tokens, processing only occupied regions to scale with occupancy rather than total volume.
• Hierarchical Attention:
  • Module-level Self-Attention: 3DCal tokens are grouped by detector module (10 longitudinal modules) and processed with learned position embeddings to capture local shower patterns before global mixing.
  • Perceiver-IO Fusion: A bottleneck architecture fuses the calorimetric tokens with compact representations of auxiliary streams (ECAL energy matrix and Muon Spectrometer hit planes). This allows the model to integrate heterogeneous inputs (sparse volumes, dense matrices, variable-length tracks) into a fixed-size latent representation.

B. Training Strategy

The framework employs a Self-Supervised Pre-training phase followed by Joint Fine-tuning.

1. Stage 1: Self-Supervised Pre-training

   • Masked Autoencoder (MAE): 75% of occupied calorimeter patches are masked. A lightweight decoder reconstructs voxel occupancy and charge. This forces the encoder to learn non-local spatial correlations and cross-detector context.
   • Relational Objectives (MAE+Rel): In a second phase, the model is augmented with relational voxel-level tasks on the kept (unmasked) patches:
     • Ghost Identification: Distinguishing reconstructed deposits with no matched true particle.
     • Interaction Hierarchy: Labeling voxels as background, primary, or secondary activity.
     • Particle Identification (PID): Classifying deposits as electromagnetic, muonic, or hadronic.
   • Note: Semantic targets (hierarchy/PID) use soft labels (fractional contributions) rather than hard one-hot assignments to handle overlapping particle showers.

2. Stage 2: Joint Fine-tuning

   • The pre-trained encoder is retained, and the decoder is discarded.
   • The encoder is fine-tuned jointly on multiple downstream tasks:
     • Classification: Neutrino flavor ($\nu_e, \nu_\mu, \nu_\tau$) and charm-quark identification.
     • Regression: Visible energy ($E_{vis}$), missing transverse momentum ($p_T^{miss}$), lepton/jet momenta, and primary vertex reconstruction.
   • Multi-task learning is used with learned homoscedastic uncertainty weights to balance losses.

3. Key Contributions

1. Sparse Encoder for Heterogeneous Data: Introduction of a novel architecture combining sparse convolutions, module-aware self-attention, and Perceiver-IO fusion to handle the specific mix of 3D volumetric and auxiliary detector data.
2. Multimodal Pre-training Strategy: Formulation of a composite pre-training objective that augments standard masked reconstruction with relational voxel-level targets. This addresses the specific physics challenges of dense showers (ghosts, hierarchy, overlapping particles).
3. Demonstration of Foundation-Style Capabilities: Proof that a single pre-trained encoder can be effectively transferred across different detector technologies (scintillators to Liquid Argon TPCs) and energy scales, and significantly reduces the need for labeled data.

4. Key Results

A. Performance on FASERCal (Source Domain)

• Classification: Pre-training (MAE) improved performance over training from scratch (Scratch). Adding relational objectives (MAE+Rel) yielded the largest gains, particularly for topologically complex and rare channels (e.g., $\nu_\tau$ and charm decays).
  • Example: For $\nu_\tau \to had$, the Figure of Merit (FOM) increased from 1.58 (Scratch) to 4.58 (MAE+Rel).
• Regression: Pre-trained models showed reduced bias and variance in kinematic reconstruction (energy, momentum) and significantly improved vertex resolution (mean error reduced from ~240 mm to ~100 mm with only $10^3$ labeled events).
• Interpretability:
  • Saliency Maps: The model focused on interaction regions and main shower structures rather than diffuse noise.
  • Latent Space: UMAP projections showed that MAE+Rel created a more structured latent space with clearer flavor clustering and energy ordering compared to Scratch.
  • Ablation: Removing the 3DCal backbone caused catastrophic failure, confirming it as the primary information source, while auxiliary branches provided channel-specific complementary information.

B. Data Efficiency

• Label Reduction: The pre-trained model achieved performance comparable to a "Scratch" model trained on 10x more data using only $10^3$ labeled events.
  • At $10^3$ events, MAE+Rel matched Scratch's flavor classification performance at $10^4$ events.
  • This is critical for energy-frontier physics where generating labeled truth data (simulation) is computationally expensive.

C. Transfer Learning (Generalization)

The pre-trained encoder was successfully transferred to two public benchmarks with different technologies:

1. Fine-grained Plastic Scintillator (Ref. [38]): Despite differences in energy scale (GeV vs. TeV) and magnetic environment, the model outperformed published baselines (GBDT, RNN) for protons, muons, and electrons.
2. PILArNet (Liquid Argon TPC): Transferring from a forward hadronic calorimeter to a LArTPC (a significant domain shift).
   • Single-particle: Accuracy improved from 0.8798 (Scratch) to 0.9154.
   • Multi-particle: Accuracy improved from 0.9333 to 0.9662, surpassing the strongest published ensemble baseline.

5. Significance

• Paradigm Shift: This work moves neutrino detector analysis from task-specific supervised models to foundation-style models capable of learning reusable representations from raw detector data.
• Feasibility: It demonstrates that for energy-frontier events (where conventional reconstruction is impossible), self-supervised learning is not just an enhancement but a mandatory prerequisite for viable physics extraction.
• Scalability: The approach proves that high-fidelity simulation can be leveraged to pre-train models that generalize across detector technologies and energy regimes, drastically reducing the dependency on expensive labeled datasets.
• Physics Impact: By improving the reconstruction of rare and complex channels (like $\nu_\tau$ and charm), this method directly enhances the physics reach of future collider neutrino experiments like FASER.

In conclusion, the paper establishes that combining masked autoencoding with physics-aware relational objectives in a sparse transformer architecture creates a robust, data-efficient, and transferable foundation for analyzing the most complex neutrino interactions.