jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation

jBOT is a self-distillation pre-training method for LHC jet data that learns semantic representations by combining local particle-level and global jet-level distillation, enabling emergent class clustering that improves performance in downstream anomaly detection and classification tasks.

The Gist

The "Master Chef" of Particle Physics: An Explanation of jBOT

Imagine you are a world-class food critic, but you’ve never actually seen a menu or been told what a "dish" is. All you have is a massive, unlabeled mountain of thousands of different plates of food delivered to your table every day.

Some plates have pasta, some have sushi, some have tacos, and some are just random scraps of bread. You don't know the names of these foods, but you start to notice patterns: "This group of plates always has something small and crunchy," or "This group always seems to be soft and saucy."

Eventually, you become so good at spotting these patterns that if someone handed you a brand-new, mysterious dish, you could say, "I don't know what this is called, but it definitely belongs in the 'crunchy' family."

This is exactly what the researchers did with "jBOT."

---

The Problem: The Messy "Jets" of the LHC

In the world of particle physics (specifically at the Large Hadron Collider), scientists smash particles together to see what comes out. When these collisions happen, they create "jets"—sprays of tiny particles that fly out like a burst of confetti.

Identifying what caused that "confetti burst" is incredibly hard. Was it a common, boring particle (like a quark)? Or was it something rare and exciting (like a Top quark or a hint of "New Physics")?

Usually, to teach a computer to recognize these, scientists have to manually label millions of examples: "This is a quark, this is a gluon, this is a W boson." This is slow, expensive, and exhausting.

The Solution: jBOT (The Self-Teaching Student)

Instead of giving the computer a "cheat sheet" (labels), the researchers created jBOT. They used a method called Self-Distillation.

Think of jBOT as a student studying for an exam by playing a game of "Predict the Missing Piece."

1. The Masking Game: The researchers take a "jet" of particles and hide some of them (like covering part of a jigsaw puzzle with a cloth).
2. The Teacher and the Student: They use two versions of the same AI. The "Teacher" sees the whole, complete puzzle. The "Student" only sees the puzzle with pieces missing. The Student’s entire job is to guess what the hidden pieces look like by looking at the pieces that are there.
3. The Result: By playing this game millions of times, the AI accidentally learns the "grammar" of physics. It learns how particles naturally hang out together. It discovers the "flavor" of different jets without ever being told their names.

Why is this a big deal? (The Two Superpowers)

The paper proves that jBOT gained two incredible superpowers:

1. The "Fast Learner" (Classification)

Because jBOT already understands the "patterns" of particles, it doesn't need much help to learn specific names. If you finally give it a few labeled examples (e.g., "By the way, this pattern is called a Top quark"), it learns much faster and more accurately than an AI that started from scratch. It’s like teaching a professional chef how to make a specific recipe versus teaching a toddler.

2. The "Security Guard" (Anomaly Detection)

This is the most exciting part for discovering new science. Because jBOT becomes an expert on "normal" physics (the boring, everyday particles), it becomes incredibly sensitive to anything "weird."

If a collision happens that produces a jet that doesn't fit any known pattern, jBOT flags it immediately. It’s like a security guard at a club who knows exactly how every regular customer walks; the moment someone walks in with a strange, suspicious gait, the guard knows something is "off." This is how we might find "New Physics"—by looking for the things that don't belong.

Summary

jBOT is a way to train AI to understand the fundamental building blocks of the universe simply by letting it observe them, hide parts of them, and try to reconstruct them. It turns a mountain of unlabeled data into a powerful tool for both identifying known particles and hunting for the unknown.

Technical Summary

Technical Summary: jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation

1. Problem Statement

In High-Energy Physics (HEP), particularly at the CERN Large Hadron Collider (LHC), a critical task is jet tagging: identifying the originating particle of a jet (e.g., quarks, gluons, W/Z bosons, or top quarks) by analyzing its substructure. This is challenging due to the high complexity of jet constituents and background noise.

Traditionally, machine learning in HEP has relied on supervised learning, which requires large amounts of accurately labeled simulated data. However, this approach faces two major hurdles:

1. Label Scarcity: Labeled data can be expensive or difficult to obtain for rare processes.
2. Domain Gap: Models trained on simulated labels may not generalize perfectly to real experimental data.

The authors aim to bridge this gap by applying the paradigm of Self-Supervised Learning (SSL)—pre-training on massive amounts of unlabeled data to learn generic features before fine-tuning on specific tasks.

2. Methodology: The jBOT Framework

The authors introduce jBOT, a pre-training method adapted from the computer vision framework iBOT. It utilizes a self-distillation mechanism with a teacher-student architecture.

• Architecture: The model uses a Vision Transformer (ViT)-style encoder. Particles within a jet are tokenized, and a special [CLS] token is prepended to aggregate global jet-level information.
• Data Augmentation: To create "positive pairs" for the model to learn from, the authors apply three physics-inspired augmentations:
  1. Uniform rotation around the jet axis.
  2. Gaussian smearing of particle positions.
  3. Collinear splitting of particles (conserving momentum).
• Self-Distillation Objectives:
  • Particle-level (Same-view) Distillation: The student network sees a "masked" version of a jet (where some particles are replaced by a [MASK] token) and is trained to predict the representations of those particles as seen by the teacher network (which sees the full jet). Masking is momentum-aware, ensuring a target percentage of the jet's transverse momentum ($p_T$) is masked rather than just a fixed number of particles.
  • Jet-level (Cross-view) Distillation: The student's global [CLS] token from one augmented view is trained to match the teacher's [CLS] token from a different augmented view of the same jet.
• Regularization: A KoLeo loss is used to ensure the learned embeddings are diverse and well-spread in the feature space, preventing "information collapse."

3. Key Contributions

• Introduction of jBOT: A novel SSL framework specifically tailored for the irregular, particle-based structure of jet data.
• Emergent Semantics: The discovery that self-distillation on unlabeled jets causes different physics classes (q, g, W, Z, t) to naturally cluster in the embedding space without any explicit labels.
• Dual-Purpose Utility: Demonstrating that the learned representations are effective for both supervised classification (via fine-tuning) and unsupervised anomaly detection (via distance metrics).

4. Experimental Results

The model was tested on the JetNet dataset using two model sizes: Small (jBOT-S) and Base (jBOT-B).

• Classification Performance:
  • Label Efficiency: When fine-tuned on only 10% of labeled data, jBOT significantly outperformed supervised models trained from scratch.
  • Full Data: Even with 100% of the data, jBOT-B achieved higher accuracy (76.43%) than the supervised baseline (76.04%).
  • Probing: Even with a "frozen" encoder (no fine-tuning), simple k-Nearest Neighbor (k-NN) classifiers achieved ~71% accuracy, proving the embeddings are highly discriminative.
• Anomaly Detection:
  • The authors trained the model only on background jets (QCD: quarks and gluons).
  • When tested on "anomalous" signals (W, Z, and top jets), the model successfully identified them using distance-based metrics (k-NN, Cosine Similarity, GMM).
  • The performance was comparable to or better than state-of-the-art reconstruction-based Autoencoders (e.g., CNNAE, GNNAE), particularly in separating W and Z signals.

5. Significance

This work represents a significant step toward the development of Foundation Models for Physics. By proving that self-distillation can capture the underlying semantic structure of particle collisions, the authors provide a roadmap for:

1. Reducing reliance on labels: Leveraging the vast amounts of unlabeled data produced by the LHC.
2. Robustness: Creating models that learn fundamental physics features rather than just memorizing simulation labels.
3. Discovery: Providing a powerful tool for anomaly detection, which is essential for finding "New Physics" that does not conform to known Standard Model patterns.