⚛️ high-energy experiments

Predict before you train: Scaling Laws for particle physics foundation models

This paper demonstrates that joint model-and-data scaling laws fitted on small transformer models can accurately predict the performance of much larger particle physics foundation models before training, enabling the translation of compute budgets into expected physics outcomes and validating the approach against state-of-the-art benchmarks.

Original authors: Jan-Lucas Uslu, Benjamin Nachman, Christopher Re

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jan-Lucas Uslu, Benjamin Nachman, Christopher Re

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the perfect cake, but you don't know how much flour, sugar, or heat you need. In the world of particle physics, scientists are constantly building "digital brains" (machine learning models) to analyze the chaotic debris from particle collisions, hoping to find hidden patterns that reveal new laws of nature. These digital brains are like giant ovens: the bigger the oven (the model) and the more ingredients you feed it (the data), the better the cake usually tastes. However, baking these cakes is incredibly expensive; it requires massive amounts of electricity and supercomputers. For years, scientists had to guess how big their oven should be before they started baking. They would spend millions of dollars on a huge oven, only to find out later that a medium-sized one would have done the job just as well, or that they needed a much bigger one to get the result they wanted. The big question was: Can we predict the taste of the cake before we even turn on the oven?

This paper, titled "Predict before you train," tackles that exact problem. The researchers wanted to see if they could figure out a "recipe rule" (called a scaling law) that tells them exactly how much computing power they need to get a specific result, without having to actually build the giant model first. They focused on a specific type of digital brain called a "transformer," which is usually used for reading text, but here they taught it to read particles flying out of a collider. By training a series of small, cheap models first, they tried to draw a line on a graph that predicts what would happen if they built a model one hundred times bigger. They also wanted to know if a better "pre-training" (learning general patterns) would actually help the model get better at specific physics tasks later, like spotting a top quark or telling a quark from a gluon.

The "Crystal Ball" for Particle Physics

The team, led by researchers from Stanford and SLAC, decided to test a bold idea: could they predict the performance of a massive machine learning model just by looking at the results of tiny, cheap ones? To do this, they used a dataset called "OmniLearned," which contains about one billion simulated particle jets (sprays of particles). They built a family of models called ParticleViT. Think of these models as a set of digital detectives. The smallest one is a rookie detective, and the largest is a super-sleuth with a massive memory.

The researchers started by training the rookie detectives on a small amount of computing power (less than $10^{19}$ FLOPs, which is a unit of calculation). They watched how well these small detectives learned. Then, they used a mathematical formula (a "scaling law") to draw a line on a graph, predicting what would happen if they trained a detective with a memory one hundred times larger.

The Prediction Came True
The result was like having a crystal ball. When they actually trained the giant models (using up to $3.4 \times 10^{20}$ FLOPs), the results landed almost exactly where the prediction said they would. The actual performance was within one percent of the forecast. This is a huge deal because, until now, no one had successfully predicted the performance of a model they hadn't trained yet in particle physics. It means scientists can now do a "dry run" with small, cheap models to figure out exactly how much computing power they need to buy before they commit to the expensive, giant training runs.

The "Information Density" Surprise

One of the most interesting findings was about how much "brain power" (model size) is actually needed. In the world of language models (like the ones that write essays or chat with you), the rule of thumb is that you need a huge model to handle a lot of data. But for particle physics, the researchers found something different.

They discovered that a single particle in a jet carries much less "information" than a single word in a sentence. A word can mean a million different things depending on context, but a particle is just a particle with a few specific numbers attached to it. Because the data is less "dense" with information, the optimal model size for this billion-particle dataset is actually much smaller than what language experts would guess. The paper suggests that for this specific job, you don't need a giant brain; you just need to feed the brain a lot of examples. The "sweet spot" for the model size was found to be far smaller than the standard rules for language models, suggesting that for particle physics, it's better to spend your budget on more data rather than a bigger model.

Does a Better "General Knowledge" Help?

The team also checked if doing well at the general pre-training task (learning about all the particles) actually helped the model get better at specific physics jobs later. They tested two common tasks:

Top Tagging: Identifying if a jet of particles came from a heavy "top quark."
Quark/Gluon Tagging: Distinguishing between jets made by quarks and those made by gluons.

They found a clear, straight line: the models that had lower "loss" (meaning they learned the general patterns better during pre-training) also performed better on these specific tasks after being fine-tuned. In other words, if your digital detective is a genius at general particle physics, it will be a genius at spotting top quarks, too. This confirms that the "compute budget" can be directly translated into an expected physics performance. If you know how much computing power you have, you can now predict how good your model will be at finding new physics.

The Final Showdown

Finally, the researchers compared their new, generic "ParticleViT" models against OmniLearned, a famous, highly specialized model that was built with specific physics rules baked into its design (like knowing the laws of relativity). The results were surprising. The generic ParticleViT models, which had no special physics rules inside them, performed just as well as the specialized OmniLearned model on most metrics.

The only place where the specialized model had a slight edge was in the very hardest, most extreme cases of "top tagging" (the "high-purity tail"), where it was about 5% better at rejecting background noise. But for almost everything else, the simple, generic model that learned from scratch was just as good. This suggests that for many physics tasks, you don't necessarily need to build complex, physics-specific architectures; a standard, well-scaled transformer trained on enough data can do the job just as effectively.

What This Means for the Future

The paper concludes by releasing five of these models (from small to extra-large) and the full code, so other scientists can use them. The main takeaway is that the era of "guessing and hoping" is over for this field. Scientists can now use a small, cheap experiment to predict the outcome of a massive, expensive one. They can calculate exactly how much computing power they need to reach a specific goal, saving time and resources. While the paper doesn't claim to have solved all of physics, it has provided a reliable map for navigating the expensive landscape of training machine learning models in particle physics, proving that sometimes, the best way to see the future is to start small.

Technical Summary: Predict Before You Train: Scaling Laws for Particle Physics Foundation Models

Problem Statement
In particle physics, the largest machine learning models are also the most computationally expensive to train. A critical gap exists in the field: while scaling laws have been fitted retrospectively for jet data, they have not been demonstrated to predict the performance of models they were not fitted on. Consequently, the return on investment for scaling a specific architecture cannot be estimated before compute resources are committed. Furthermore, no study has successfully connected a pretraining loss scaling law to downstream physics metrics (such as background rejection), leaving practitioners unable to translate a compute budget into expected physics performance in advance.

Methodology
The authors introduce ParticleViT, a generic transformer architecture applied to jet constituents with minimal physics-specific inductive bias (omitting Lorentz equivariance or pairwise interaction features). The study employs a "fit-then-forecast" protocol:

Data and Objective: The models are pretrained on the OmniLearned corpus, a collection of approximately $1.06 \times 10^9$ jets. The pretraining objective is a 210-class classification task (jet processes, flavors, and dataset categories).
Scaling Sweep: A geometric ladder of ParticleViT models is trained on an IsoFLOP grid, varying model size ( $N$ ) and dataset size ( $D$ ) while holding training compute constant. The sweep covers ten compute budgets ranging up to $3 \times 10^{18}$ FLOPs.
Scaling Law Fitting: The authors fit a joint model-and-data scaling law (Chinchilla-style) to the pretraining loss of models trained below $10^{19}$ FLOPs. The functional form is:
$L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + L_\infty$
Held-Out Extrapolation: Using only the small-model data, the authors predict the loss of five larger models trained with up to $113\times$ more compute (up to $3.4 \times 10^{20}$ FLOPs).
Downstream Validation: The pretrained models are fine-tuned on two standard benchmarks: Top Quark Tagging and Quark/Gluon discrimination. The authors analyze the correlation between pretraining loss and downstream fine-tuning loss/background rejection.
Benchmarking: The largest "frontier" ParticleViT model is compared against OmniLearned, a state-of-the-art physics-aware foundation model trained on the same corpus.

Key Contributions

Validated Forecasting: The study presents the first validated forecast of a scaling law in particle physics. By fitting the law on models trained below $10^{19}$ FLOPs, the authors predicted the loss of models trained with over $100\times$ more compute. The measured losses landed within 1% of the predictions (mean relative error 0.26%).
Compute-to-Performance Chain: The authors establish a direct link between compute budget and physics performance. They demonstrate that lower pretraining loss systematically yields lower downstream fine-tuning loss and higher background rejection across both tagging tasks. This allows a physics performance target to be translated into an estimated compute budget before training a large model.
Compute-Optimal Allocation: The fitted scaling law reveals that the compute-optimal model size for the billion-jet corpus is significantly smaller than the rule of thumb derived from language models (which suggests $\sim 1$ parameter per 20 tokens). The fitted exponent for data ( $\beta \approx 0.06$ ) is much smaller than that for model size ( $\alpha \approx 0.23$ ), implying that for particle physics, compute is best spent overwhelmingly on increasing the number of training examples rather than increasing model capacity.
Open Release: Five pretrained ParticleViT models (S through XL), the complete training recipe, and fine-tuning protocols are released to the community.

Results

Scaling Accuracy: The extrapolated loss predictions for the five held-out models (ranging from 16M to 397M parameters) were highly accurate, with a maximum relative error of 0.66%.
Downstream Performance: Lower pretraining loss consistently correlated with better downstream performance. The largest ParticleViT model (397M parameters) achieved performance consistent with the published numbers for the physics-aware OmniLearned model (423M parameters) on accuracy, AUC, and quark/gluon rejection.
Physics-Aware Comparison: The generic transformer (ParticleViT) matched the physics-aware baseline (OmniLearned) on bulk metrics. The physics-aware model retained a measurable edge only in the high-purity tail of top tagging (background rejection at $\epsilon_S=0.5$ and $0.3$), where rare radiation patterns are most critical.
Information Density: The authors interpret the small data exponent and the preference for smaller models as a reflection of the lower information content of a "particle token" compared to a "language token."

Significance and Claims
The paper claims to close the loop between compute budget and expected physics performance. By demonstrating that a scaling law fitted on small models can accurately forecast the loss of much larger models, and that this loss predicts downstream physics metrics, the authors argue that practitioners can now plan large training runs with confidence.

The authors explicitly state that this methodology is not specific to jets; any data-rich scientific domain capable of running a sweep of small models could apply this "fit-then-forecast" protocol. They emphasize modesty regarding their claims: the scaling law predicts performance within the specific model family and tasks studied, and the connection to downstream metrics is an observed transfer, not a guarantee for arbitrary new physics questions. The study does not claim that the physics-aware model is obsolete, but rather that a generic transformer, when scaled according to the derived law, can reach performance consistent with specialized physics-aware baselines on standard benchmarks, with the specialized model offering only a marginal advantage in extreme high-purity regimes.