⚛️ high-energy experiments

Scaling Collider Event Generation with Residual-Quantized Tokens

This paper introduces a scalable, autoregressive transformer-based framework for generating full collider events using residual-quantized tokens, demonstrating that token-level loss effectively predicts physical fidelity and enabling fast, ML-based surrogates to overcome bottlenecks in High-Luminosity LHC simulations.

Original authors: Dan Godi, Dmitrii Kobylianskii, Eilam Gross

Published 2026-10-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dan Godi, Dmitrii Kobylianskii, Eilam Gross

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the heart of Europe, where the Large Hadron Collider smashes protons together at nearly the speed of light, scientists are facing a problem of sheer volume. When the machine reaches its full power in the coming years, it will generate so much data that the computers needed to simulate what happens inside the detector will run out of time and memory. To understand these collisions, researchers traditionally rely on massive, step-by-step computer programs that mimic the behavior of every single particle as it flies through the detector. These programs are incredibly accurate but also incredibly slow, requiring vast amounts of computing power just to create the synthetic data needed to test new theories. The scientific community needs a faster way to generate these simulations, one that can keep up with the machine without sacrificing the physical reality of the results.

A team of researchers at the Weizmann Institute of Science in Israel has proposed a solution by borrowing a technique from the world of language models. Just as modern artificial intelligence can write coherent stories by predicting the next word in a sentence, these scientists have trained a similar system to predict the next "piece" of a particle collision. Instead of dealing with the messy, continuous numbers that describe a particle's speed and position, they first translate every particle into a short, discrete sequence of symbols, much like turning a sentence into a string of letters. They then use a powerful computer model to learn the patterns of these symbol sequences, allowing it to generate new, realistic collision events by simply predicting the next symbol in the chain. This approach transforms the complex physics of a particle detector into a language problem, where the computer learns the grammar of subatomic interactions.

The researchers tested this method on data from the CMS detector, a massive instrument that records the debris of proton collisions. They started by taking real collision events and breaking them down into their fundamental components: the particles that emerge from the collision and the signals those particles leave behind in the detector. To make this data manageable for their model, they used a specialized tool to compress the continuous physical properties of each particle—such as its momentum and direction—into a fixed set of digital codes. This process is like translating a high-resolution photograph into a series of pixel values that a computer can easily store and manipulate. By training their model on millions of these translated events, the system learned to generate new sequences of codes that, when converted back into physical numbers, looked and behaved exactly like real detector data.

What makes this work particularly significant is how the researchers verified that the generated data was not just statistically similar, but physically faithful. They did not simply check if the average speed of the particles matched; they examined the entire structure of the events, from individual particles to the complex sprays of debris known as jets. They found that the model could successfully recreate the subtle, random fluctuations that occur when particles interact with the detector material. Crucially, the system was able to generate events for physical processes it had never seen before, such as specific types of rare particle decays, by conditioning its predictions on the initial collision parameters. This suggests the model learned the underlying rules of the detector's response rather than just memorizing the training data, a vital requirement for a tool that must handle the unknown physics of future discoveries.

The team also investigated how the size of the model and the amount of data it saw affected its performance. They trained versions of the system ranging from ten million to over one billion parameters, testing them on datasets of varying sizes. They discovered a clear and predictable relationship: as the models grew larger and saw more data, the error in their predictions dropped in a consistent, mathematical way. Perhaps most importantly, they found that a simple measure of how well the model predicted the next symbol in the sequence was a reliable indicator of how accurate the final physical results would be. This means that scientists can use the model's internal error rate to gauge the quality of the generated physics without needing to run expensive, full-scale simulations to check the output.

By demonstrating that these discrete, language-based models can faithfully reproduce the complex output of a particle detector, the researchers have provided a new pathway for scaling up collider simulations. Their work shows that it is possible to bypass the computational bottlenecks that threaten to slow down future discoveries. The method relies on a straightforward principle: if you can translate the language of physics into a format that a computer can read as a sequence of symbols, you can use the most powerful tools of artificial intelligence to speak that language back to you. The results suggest that in the era of the High-Luminosity Large Hadron Collider, the future of simulation may not be about building bigger computers to crunch numbers, but about teaching machines to understand the grammar of the universe.

Technical Summary: Scaling Collider Event Generation with Residual-Quantized Tokens

Problem Statement

At the High-Luminosity Large Hadron Collider (HL-LHC), the computational requirements for full detector simulation and reconstruction of synthetic datasets are projected to exceed available resources, creating a significant bottleneck for physics analysis. Traditional Fast Simulation (FastSim) methods rely on approximations that sacrifice physical fidelity for speed. While Machine Learning (ML) has been increasingly adopted in High Energy Physics (HEP), and Large Language Models (LLMs) have advanced discrete generative modeling, there remains a need for scalable, high-fidelity surrogates that can generate full-event data while preserving compatibility with existing downstream analysis pipelines. Previous ML approaches have largely focused on jet-level or calorimeter-level modeling, or utilized flow-matching techniques for conditional full-event simulation, but have not fully leveraged the scaling laws and discrete modeling paradigms established in natural language processing.

Methodology

The authors propose a two-stage framework that extends discrete modeling to the full-event level, treating collider events as sequences of tokens derived from residual-quantized representations.

1. Data Representation and Tokenization (RQ-VAE)

The approach utilizes a Residual-Quantized Variational Autoencoder (RQ-VAE) to map continuous particle-level data into a discrete latent space.

Input: Events consist of generator-level "Truth" (TR) particles and reconstructed "Particle Flow" (PF) objects. Each particle is described by transverse momentum ( $p_T$ ), pseudorapidity ( $\eta$ ), azimuthal angle ( $\phi$ ), class, and production vertex coordinates.
Quantization: Instead of a single vector quantization (which suffers from codebook collapse and capacity limits), the model employs residual quantization. A latent vector $z$ is encoded as an ordered sequence of $M$ codes. At each depth $m$ , the model quantizes the residual vector $r_{m-1}$ against a codebook $C_m$ , subtracts the selected code, and passes the new residual to the next depth.
Architecture: The tokenizer consists of a self-attention transformer encoder, the residual-quantization module, and a transformer decoder. It is trained end-to-end to minimize reconstruction error.
Output: An event with $N$ particles is flattened into a sequence of $N \times M$ integer tokens, converting the generation problem into a sequence modeling task.

2. Autoregressive Generative Model

A decoder-only transformer with causal self-attention models the distribution over these token sequences.

Conditioning: The model is conditioned on the TR-level event. The input sequence concatenates the tokenized TR event (prefixed with a special token) and the tokenized PF event. The TR block acts as a prefix, allowing the model to learn the detector response as a physical map independent of the specific production process.
Multiplicity Prediction: Since events have variable particle counts, a separate feed-forward network predicts the target number of PF particles ( $N_{PF}$ ) from the pooled TR prefix. This count is fed into the model via a "countdown" embedding.
Training Objective: The model is trained using a composite Cross-Entropy Loss comprising next-token prediction and cardinality prediction objectives.
Generation: During inference, the TR input is tokenized and prefilled. The model samples the target multiplicity, then autoregressively samples $N_{PF} \times M$ tokens. These tokens are passed through the frozen RQ-VAE decoder to recover continuous particle-level quantities.

Key Contributions

Discrete Full-Event Modeling: The work extends the discrete modeling paradigm to full-event simulation, demonstrating that residual-quantized representations can effectively compress and reconstruct point-cloud data (particle showers) for collider physics.
Conditional Generation: The framework enables conditional generation from detector-stable particles (TR), allowing the surrogate to model the stochastic detector response rather than just the final output distribution. This facilitates generalization to physical processes (e.g., rare or theoretical signals) not seen during training.
Scaling Laws and Repetition Analysis: The authors conduct a comprehensive scaling study across model sizes (10M to 1B parameters) and dataset sizes (up to 100% of available data), including the effects of repeated data exposure. They fit various scaling laws (Chinchilla, Skaling, and repetition-aware variants) to the token cross-entropy loss.
Loss-Fidelity Correlation: A primary contribution is the demonstration that token-level loss (next-token prediction cross-entropy) systematically predicts downstream physical fidelity. The authors show that improvements in cross-entropy translate predictably into higher fidelity across particle, jet, and global event observables.

Results

Reconstruction Fidelity: The RQ-VAE tokenizer achieves high fidelity, with codebook utilization near 100% at deeper residual levels and minimal class confusion. The reconstructed features closely match the original distributions.
Generation Performance: The autoregressive model successfully generates PF events that match the statistical properties of full simulation (Geant4) across in-distribution ( $t\bar{t}$ $t \overset{ˉ}{t}$ , QCD) and out-of-distribution ( $H \to 4\ell$ $H \to 4 ℓ$ , high- $p_T$ $p_{T}$ QCD) datasets.
- Particle Level: Residual distributions of $p_T$ , $\eta$ , $\phi$ , and vertex coordinates for generated events closely track the true detector response.
- Event Level: Global variables such as missing transverse energy ( $E_T^{miss}$ ) and scalar sum of $p_T$ ( $H_T$ ) are well-reproduced.
- Jet Level: Kinematic properties and substructure observables ( $C_2$ , $D_2$ ) of clustered jets show high agreement with truth-matched jets.
Scaling Behavior:
- The next-token prediction (NTP) loss follows familiar scaling laws. The study identifies that the optimal model size for a given compute budget is data-constrained in the current regime.
- Data repetition effects are captured by a custom "Prescriptive" scaling law, which includes a penalty term for repeated exposure.
- Predictive Power: A power-law relationship ( $SWD = a(L - L_0)^b + SWD_f$ ) is established between the NTP loss and the Sliced Wasserstein Distance (SWD) of physical observables. This confirms that token-level loss is a strong proxy for physical performance, with high $R^2$ values (0.96–0.97) for event, jet, and particle residuals.

Significance and Claims

The paper claims to provide an empirical framework for scalable collider full-event generation based on residual-quantized representations. Its significance lies in:

Validating Discrete Architectures: Demonstrating that autoregressive transformers trained on tokenized data can serve as high-fidelity surrogates for complex, continuous physical simulations.
Guiding Resource Allocation: Establishing that token-level scaling laws can reliably guide decisions regarding model capacity, dataset size, and compute allocation for ML surrogates in HEP.
Efficiency: Showing that a single surrogate model, conditioned on truth-level inputs, can serve multiple signals and backgrounds without per-analysis re-tuning, potentially alleviating the computational bottlenecks anticipated at the HL-LHC.

The authors maintain a modest stance, noting that while the token-level loss is a strong predictor, the ultimate validation remains the fidelity of the generated physical observables, which they have empirically verified across multiple scales and datasets.