⚛️ phenomenology

The fundamental limit of jet tagging: Beyond top jets

This paper extends a transformer-based generative modeling approach, previously used to establish the fundamental performance limits for boosted top-quark jet tagging, to W, Z, and Higgs-to-gluon jets, revealing that the gap between current machine-learning taggers and the theoretical optimal limit is strongly jet-dependent and significantly smaller for these more challenging tasks.

Original authors: Sarah Koller, Humberto Reyes-González

Published 2026-08-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sarah Koller, Humberto Reyes-González

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the universe as a giant, high-speed particle collider, smashing tiny bits of matter together at speeds close to light. When these collisions happen, they don't just vanish; they explode into sprays of new particles called "jets." Think of a jet like a firework that has just gone off, scattering sparks everywhere. Physicists are obsessed with figuring out what kind of firework started the explosion. Was it a heavy, rare particle like a top quark, or just a common, boring particle from the background noise? This detective work is called "jet tagging."

For years, scientists have used super-smart computer programs, powered by machine learning, to sort these jets. These programs are getting incredibly good, but a big question has been hanging over the field: How good can they actually get? Is there a theoretical ceiling, a "perfect score" that no computer could ever beat, no matter how smart it gets? To find out, researchers need to know the "perfect" answer to compare their computers against. But since the real universe is messy and the math is incredibly complex, no one knew what that perfect answer looked like until now. This paper is about building a map to that perfect score and seeing how close our current detectives are to solving the case.

The Detective's Dilemma: Chasing the Perfect Score

In the world of particle physics, the goal is to spot a "signal" (a rare, interesting particle) hiding inside a sea of "background" (common, boring particles). The paper explains that the ultimate detective tool, the "perfect classifier," is a mathematical formula called a likelihood ratio. In plain English, this formula asks: "How much more likely is this jet to be a rare signal than a common background?" If you could calculate this perfectly, you would have the best possible detector in the universe.

The problem is that we don't know the exact recipe for how these jets are made. It's like trying to guess the ingredients of a secret sauce without ever seeing the chef cook. Without the recipe, we can't calculate the perfect score. To fix this, the authors used a clever trick: they built a "generative model." Think of this as a super-advanced AI chef that learned to cook by tasting millions of simulated jets. Once the AI learned the recipe, it could cook up its own "synthetic" jets and, crucially, tell us the exact probability of every single one. This gave the scientists a known "perfect" benchmark to test their real detectors against.

The Experiment: From Top Quarks to the Higgs

The researchers took this new testing ground and applied it to four different types of particles: the top quark, the W boson, the Z boson, and the Higgs boson. They used a specific type of AI called an "autoregressive transformer" (a fancy name for a model that predicts the next piece of a puzzle based on the pieces before it) to learn the patterns of these particles. They trained these models on a massive dataset containing about 10 million events for each type of jet.

Once the models were trained, they generated 10 million new, fake jets for each category. Because the AI knew the "recipe," it could calculate the perfect likelihood ratio for every single fake jet. This created the "Gold Standard" curve—the best possible performance anyone could hope for. Then, they took a standard machine-learning classifier (the "Baseline Transformer Classifier" or BTC) and tried to beat the Gold Standard using the same fake data.

The Big Surprise: The Gap Shrinks

Here is where the story gets interesting. In a previous study, the researchers looked at top quark jets and found a huge gap between what the AI could do and the perfect theoretical limit. It was like having a detective who was good, but the perfect detective was still miles ahead.

However, when they looked at the W, Z, and Higgs jets, the story changed completely. The gap between the current AI and the perfect limit shrank dramatically. For some of these particles, the current AI was almost as good as the theoretical best possible detector.

Why is this happening? The authors suggest it's because of how "messy" the particles are.

Top Quarks are like complex, three-pronged fireworks with a distinct heavy core (a bottom quark). They have a lot of unique features and structure. Because they are so complex, there is a lot of hidden information that current AI might be missing, leaving a big gap between "good" and "perfect."
W, Z, and Higgs jets are more like simple, uniform sprays. They look more like the boring background noise and have less distinct structure. Because they are simpler and more chaotic, there is less "hidden information" to find. The perfect detector isn't much better than the current one because there simply isn't much more to learn.

What This Means

The paper doesn't claim to have solved the problem of jet tagging forever, nor does it say the current AI is perfect. Instead, it suggests that the "ceiling" of performance depends heavily on what you are trying to find. For complex particles like the top quark, there is still plenty of room for improvement; our detectors are far from the limit. But for simpler particles like the W, Z, and Higgs, we might be hitting the wall of what is physically possible to distinguish.

The authors are careful to note that these results come from simulations, not real-world data yet. They are currently working on making sure their "AI chef" is truly perfect and checking if these limits hold up as they add more data. But for now, the study gives us a clear map: for some particles, we are almost at the finish line; for others, the race is just beginning.

Technical Summary: The Fundamental Limit of Jet Tagging: Beyond Top Jets

Problem Statement
Jet tagging, the identification of the origin of high-energy hadronic jets (e.g., distinguishing boosted heavy particles from QCD background), is a central challenge in LHC physics. While machine learning (ML) architectures have achieved significant performance gains, a fundamental question remains: how close are current classifiers to the theoretical optimum allowed by the information contained within the jet? According to the Neyman–Pearson lemma, the optimal discriminator between two classes is the likelihood ratio $p(x|A)/p(x|B)$ . However, because the underlying probability density functions (PDFs) of jet data are not known analytically, this optimal limit has historically been inaccessible for direct comparison with modern taggers.

Methodology
This work extends a framework previously established for boosted top-quark jets [1] to a broader set of benchmark processes: boosted $W \to q\bar{q}'$ , $Z \to q\bar{q}$ , and $H \to gg$ jets, using QCD jets as the background. The methodology relies on the following steps:

Generative Modeling: Autoregressive transformers are trained on the JetClass dataset (containing $\sim$ 10 million training and 2 million testing events per class) to perform explicit density estimation. The models operate at the constituent level, utilizing $(p_T, \eta, \phi)$ information discretized into a $(40, 30, 30)$ binning scheme, resulting in a vocabulary of approximately 39,000 tokens.
Synthetic Data Generation: A separate transformer is trained for each signal class (Top, $W$ , $Z$ , $H$ ) and the QCD background. These models provide an explicit approximation of the underlying PDFs.
Optimal Classifier Construction: Using the trained generative models, 10 million synthetic events are generated per class. Because the models yield exact likelihood values for these events, the optimal discriminator is constructed directly via the log-likelihood ratio (LLR):
$\text{LLR} = \log \frac{p_S(x)}{p_{\text{QCD}}(x)}$
This LLR defines the theoretical optimal Receiver Operating Characteristic (ROC) curve.
Baseline Comparison: The same synthetic datasets are used to train a Baseline Transformer Classifier (BTC). The BTC shares the same backbone architecture as the generative models but includes a classification head, allowing for a direct comparison between the learned classifier and the estimated theoretical optimum under controlled conditions.

Key Results
The study compares the performance of the BTC against the optimal LLR classifier across different jet classes:

Top Jets: A sizeable gap between the BTC performance and the optimal limit is observed, consistent with previous findings [1].
$W$ , $Z$ , and $H \to gg$ Jets: In contrast to top jets, the performance gap for these processes is substantially smaller. In several cases, the gap is nearly closed, indicating that current classifiers are approaching the theoretical limit for these specific tasks.
Training Dynamics: Training and validation loss curves for the generative models (shown for Top and $H \to gg$ ) indicate no evidence of conventional overfitting.

Interpretation and Significance
The paper posits that the variation in the performance gap is strongly dependent on the specific jet class and its underlying structure:

Complexity vs. Information: Top jets possess rich, distinctive features, such as a characteristic three-prong topology and the presence of a $b$ -quark. These complex features provide more "room" for classifiers to improve beyond current baselines, resulting in a larger gap to the optimum.
Simplicity and Stochasticity: $W$ , $Z$ , and $H \to gg$ jets exhibit less distinctive substructure and are morphologically closer to QCD jets, sharing similar stochastic radiation patterns. While this reduces the overall discrimination power of the task, it also limits the amount of exploitable information, making the optimal limit easier for ML models to approximate.

Scope and Limitations
The authors emphasize that the primary goal of this work is not to assign a definitive, absolute value to the optimal limit, but to compare different jet classes within a common framework. They acknowledge that the inferred limit depends on the choice and quality of the generative model and that additional nuisances may affect interpretation. Consequently, the paper frames these results as a comparative analysis highlighting a "strongly reduced gap" for $W$ , $Z$ , and $H \to gg$ tagging tasks relative to top tagging.

Future Directions
Ongoing work focuses on:

Understanding the evolution of the estimated limit with increased training statistics and improved generative model optimization.
Identifying robust convergence criteria for learned probability densities.
Systematically validating these densities using statistical hypothesis testing.
Investigating connections to scaling laws in jet generation and classification.
Exploring whether the larger gap observed in top tagging can be quantitatively related to the intrinsic stochasticity of different jet classes.

The Detective's Dilemma: Chasing the Perfect Score

The Experiment: From Top Quarks to the Higgs

The Big Surprise: The Gap Shrinks

What This Means

Technical Summary: The Fundamental Limit of Jet Tagging: Beyond Top Jets

More like this