Reducing Simulation Dependence in Neutrino Telescopes with Masked Point Transformers

This paper introduces the first self-supervised training pipeline for neutrino telescopes using masked point transformers to leverage real data, thereby significantly reducing reliance on simulations and mitigating associated systematic uncertainties.

The Gist

The Big Problem: The "Perfect World" vs. The "Real World"

Imagine you are teaching a student to identify different types of birds. You have a textbook full of perfect, crystal-clear photos of birds (this is Simulation). You also have a messy, real-world video feed from a forest where the birds are often hidden by leaves, the lighting is bad, and there are random leaves blowing in the wind (this is Real Data).

Traditionally, scientists train their computer models (the students) using only the perfect textbook photos. The problem is that when the model goes out to the real forest, it gets confused. It doesn't know how to handle the messy leaves or the weird lighting because it never saw them in the textbook. In the world of neutrino telescopes (giant detectors buried in ice or deep underwater), these "messy leaves" are things like random electronic noise or unexpected environmental effects that the computer simulations didn't predict.

The New Solution: "Self-Supervised Learning"

The authors of this paper propose a new way to train these models. Instead of just studying the perfect textbook, they let the model practice on the messy, real-world forest video without a teacher telling it what bird is what.

They call this Self-Supervised Learning (SSL).

The Analogy: The "Missing Puzzle" Game
Imagine you have a huge puzzle of a forest scene, but someone has covered 75% of the pieces with black tape (this is Masking).

1. The Task: The computer model has to look at the visible pieces and guess what the hidden pieces look like.
2. The Learning: To do this, the model has to learn the structure of the forest. It learns that "trees usually have leaves," "birds fly in certain patterns," and "wind moves leaves in a specific way." It learns these rules by looking at the messy real data itself, not by reading a textbook.
3. The Result: Once the model has mastered the "forest structure" by playing this guessing game, you can then show it a few labeled pictures from the textbook to teach it specific bird names. Because it already understands the messy environment, it handles the real world much better than a model that only studied the textbook.

The Tool: "Neptune"

To make this work, the authors built a specific type of computer brain called neptune (a "Neutrino Event Transformer").

• How it works: Neutrino telescopes detect "hits" (flashes of light) from sensors. These hits are scattered in 3D space and time, like a cloud of points.
• The Innovation: Nepture treats these scattered points like a "point cloud" (similar to how a 3D scanner sees a room). It uses a "Transformer" (a type of AI famous for understanding language) to understand the relationships between these scattered light flashes, even when some of them are missing or noisy.

The Experiment: Testing the "Noise"

The researchers tested two scenarios to see if their new method worked better than the old one:

Scenario 1: The "Total Surprise" (Un-modeled Noise)

• The Setup: They trained the old model on a "clean" simulation (no noise). They tested it on "real" data that had a lot of random noise (like static on a radio).
• The Result: The old model crashed. It couldn't figure out the direction of the neutrinos or distinguish between different types of events. It was like a student who only studied in a quiet library failing a test in a loud construction zone.
• The Winner: The new SSL model (which practiced on the noisy data first) remained calm and accurate. It knew what "noise" looked like because it had seen it during its "missing puzzle" training.

Scenario 2: The "Slight Mismatch" (Varying Noise Rates)

• The Setup: Both the training data and the test data had noise, but the amount was slightly different (e.g., 500 Hz in training vs. 600 Hz in testing).
• The Result: In this case, the old model was actually okay. It could handle small differences. However, the new SSL model performed just as well, proving it is a safe, robust choice for both small and big problems.

The Bottom Line

The paper claims that by using this "guess the missing piece" technique on real, unlabeled data, scientists can build models that are much less dependent on perfect simulations.

• Old Way: Train on perfect simulations $\rightarrow$ Fail when real life is messy.
• New Way: Learn the structure of messy real life first $\rightarrow$ Succeed even when simulations are imperfect.

This approach doesn't just fix small errors; it acts as a safety net against "unknown unknowns"—things in the real detector that the scientists didn't even know to simulate in the first place.

Technical Summary

Technical Summary: Reducing Simulation Dependence in Neutrino Telescopes with Masked Point Transformers

Problem Statement
Machine learning (ML) models in neutrino physics, particularly for large-scale telescopes like IceCube, KM3NeT, and Baikal-GVD, have traditionally relied on labeled Monte Carlo (simulation) data. While these models enable fast event reconstruction and classification, they face a persistent challenge: discrepancies between simulations and real data arising from complex environmental conditions, detector-specific systematics, and unmodeled physical effects. These discrepancies can introduce biases in reconstruction or lead to incorrect coverage assessments, ultimately impacting analysis conclusions. Although self-supervised learning (SSL) has emerged as a powerful paradigm for reducing dependence on labeled datasets in computer vision and natural language processing, its application to neutrino telescopes has been limited, primarily explored for domain adaptation rather than as a primary training strategy to mitigate simulation mis-modeling.

Methodology
The authors propose a novel training pipeline that shifts the majority of model training onto unlabeled real data, thereby bypassing simulation discrepancies. The core of this approach involves:

1. Model Architecture (neptune): The study utilizes a custom transformer architecture termed "neptune" (an Efficient Point Transformer for Ultrarelativistic Neutrino Events). This model is grounded in point cloud methodologies and consists of three components:

   • Event Tokenizer: Converts irregular raw sensor hits (4D spatio-temporal coordinates) into token sequences. It employs a PointNet-inspired strategy using per-point MLPs. To handle variable event sizes, it utilizes Farthest Point Sampling (FPS) if the hit count exceeds a maximum ($T_{max}=512$) and 4D k-Nearest Neighbors (KNN) to aggregate spatial and temporal context.
   • Transformer Encoder: Processes the token sequences, enriched with spatial positions and first-hit times.
   • Downstream Task Head: Aggregates encoder outputs via mean pooling for specific tasks.

2. Self-Supervised Pre-training: The model is pre-trained on unlabeled "real" data using a masked autoencoder approach. The tokenizer masks spatio-temporal coordinates (either temporal-only or spatio-temporal), and the transformer is trained to reconstruct these masked inputs using smooth L1 loss. High masking ratios (0.75 to 1.0) are employed to force the model to learn the inherent structure of neutrino data without explicit labels.

3. Fine-tuning: Following pre-training, a prediction head is attached, and the model is fine-tuned on a smaller set of labeled simulation data. To prevent catastrophic forgetting of the target domain during this shift, the authors employ a "block expansion" technique, inserting identity-initialized transformer blocks atop the frozen pre-trained layers.

Experimental Setup
The study evaluates the approach using two benchmark tasks:

• Directional Reconstruction: Reconstructing the direction of muon neutrinos ($\nu_\mu$ CC).
• Cascade Classification: Separating double cascades (from $\nu_\tau$ CC) from single cascade backgrounds.

Datasets were generated using the Prometheus simulation framework with an IceCube-like configuration. To test robustness, the authors introduced controlled discrepancies by injecting uncorrelated noise hits into the "data" set at specific rates (e.g., 100 Hz or 600 Hz) while keeping the simulation set clean or mismatched. Two scenarios were tested:

1. Un-modeled Noise: Simulation contains zero noise, while "data" contains noise.
2. Varying Noise Rates: Both sets contain noise, but with a modest mismatch (600 Hz in data vs. 500 Hz in simulation).

Key Results
The experiments compare the proposed SSL approach against a baseline supervised model trained directly on labeled simulation:

• Un-modeled Noise Scenario: When the simulation lacks noise entirely but the real data contains it, the supervised model suffers significant performance degradation.
  • Directional Reconstruction: The supervised model's median angular error on "data" worsened to 20.5°, whereas the SSL model maintained a robust 5.0° (compared to ~2° on simulation for both).
  • Cascade Classification: The supervised model's PR-AUC dropped to 0.226 on "data" (from 0.364 on simulation), while the SSL model generalized better with a score of 0.287.
• Varying Noise Rates: When both datasets contained noise with a modest mismatch (600 Hz vs. 500 Hz), both supervised and SSL models performed comparably. This indicates that supervised models are resilient to moderate, known systematic errors, but fail when effects are entirely unmodeled.

Significance and Claims
The paper claims to present the first self-supervised training pipeline for neutrino telescopes that leverages point cloud transformers and masked autoencoders. The primary significance lies in demonstrating that SSL provides a "valuable safeguard" against unmodeled discrepancies between simulations and real detector data.

The authors argue that while traditional supervised methods are adequate for handling small, known systematic errors, they are brittle against subtle, unmodeled phenomena. In contrast, the SSL approach, by learning representations from the internal structure of unlabeled real data, maintains stable performance even when the simulation does not perfectly capture the detector's behavior. This represents a fundamental departure from previous ML applications in the field, paving the way for improved event reconstruction and classification in the presence of unknown systematics. The authors note that future work will focus on deploying this approach on real experimental data, specifically assessing robustness in large-scale detectors like IceCube.