Accelerating multijet-merged event generation with neural network matrix element surrogates

This paper proposes a method to accelerate multijet event generation for LHC analyses by employing neural-network surrogates within a two-stage rejection-sampling algorithm, achieving over a tenfold reduction in generation time for Z+jets processes at the HL-LHC compared to the baseline Sherpa generator.

The Gist

Imagine you are running a massive, high-stakes lottery to predict what happens when two particles smash together at the speed of light inside the Large Hadron Collider (LHC).

To understand the results of these collisions, physicists need to simulate billions of "what-if" scenarios. They use a computer program (a Monte Carlo generator) to calculate the odds of every possible outcome. However, there's a catch: calculating the exact odds for complex collisions (like a Z boson smashing into 6 jets of particles) is incredibly slow and expensive, like trying to solve a million-piece puzzle for every single ticket you buy.

Most of the time, the computer calculates a scenario, realizes the odds are tiny, and throws it away. This is called "rejection sampling." It's like a bouncer at a club who checks every ID, but 99% of people get rejected. The bouncer spends all his time checking IDs that don't matter, and the club (the simulation) moves very slowly.

The Problem: The "Bottleneck" at the High-Luminosity LHC

The LHC is getting upgraded to the "High-Luminosity" (HL-LHC) version, which means it will produce data at a rate that is currently impossible to simulate. If we keep using the old "bouncer" method, we will run out of computing power before we can analyze the data. We need a way to speed up the process without losing accuracy.

The Solution: The "AI Bouncer" (Neural Network Surrogates)

The authors of this paper propose a clever two-step solution using Artificial Intelligence (AI). Think of it as hiring a super-fast, slightly imperfect AI bouncer to do the initial screening.

Step 1: The AI Guess (The Fast Filter)
Instead of the slow, perfect computer calculating the exact odds for every single event, the AI (a Neural Network) makes a quick, educated guess.

• The Analogy: Imagine the AI is a bouncer who can glance at a crowd and instantly say, "Hey, 90% of these people probably don't belong here, let's skip them."
• The AI is fast but not perfect. It might let a few "bad" people in or reject a few "good" ones. But because it's so fast, it clears the line much quicker.

Step 2: The Double-Check (The Correction)
If the AI says, "Okay, this person might belong," the slow, perfect computer steps in to do the final, exact calculation.

• The Analogy: The AI lets a few people through the first door. Then, a second, very strict bouncer (the real computer) checks their ID one last time. If the AI was wrong and let a bad person in, the second bouncer kicks them out. If the AI was right, they get in.

The Magic Trick:
Because the AI is so fast, it filters out the vast majority of useless calculations before the slow computer ever has to touch them. The few times the slow computer does have to work, it's only on the events that actually matter.

Why This Paper is Special

Previous attempts at this "AI bouncer" method were like testing it on a small, empty street. This paper takes the method to the "busy city center" of real-world physics. They had to solve several tricky problems to make it work for the LHC:

1. The "Color" Confusion: Particles have a property called "color charge" (unrelated to actual color). Calculating this is hard. The authors found that for the AI to work best, they had to simplify how they handled these colors, essentially grouping similar particles together so the AI didn't get confused.
2. The "Rare Event" Bias: Sometimes, physicists are looking for very rare, weird collisions (like a particle flying off at a weird angle). The standard simulation ignores these because they are rare. The authors taught the AI to over-produce these rare events so they can be studied, then mathematically corrected the results later.
3. The "Overweight" Problem: Sometimes the AI is too generous and lets in people who shouldn't be there. The authors developed a system to handle these "overweight" events so they don't ruin the final statistics.

The Results: A Speed Boost of 10x

When they tested this on simulating Z + Jets (a common but complex collision) at the future HL-LHC:

• Old Method: Took a huge amount of time and computing power.
• New Method: Took 10 times less time.

To put it in perspective: If the old method took 100 years of computer time to generate the data needed for the next 10 years of experiments, the new method would only take 10 years.

The Bottom Line

This paper is a breakthrough because it proves that we can use AI not just to guess physics, but to accelerate the most difficult parts of physics simulations. By using a fast AI to filter out the noise and a slow, perfect computer only for the signal, we can simulate the future of particle physics today. It's like upgrading from a hand-cranked calculator to a supercomputer, ensuring that when the High-Luminosity LHC turns on, we won't be left behind by our own data.

Technical Summary

Here is a detailed technical summary of the paper "Accelerating multijet-merged event generation with neural network matrix element surrogates" (MCNET-25-12).

1. Problem Statement

The simulation of high-multiplicity final states (e.g., $Z$+jets with many jets) at the Large Hadron Collider (LHC) and the upcoming High-Luminosity LHC (HL-LHC) faces a severe computational bottleneck.

• Computational Cost: The evaluation of hard-scattering matrix elements (ME) for processes with high parton multiplicities is extremely expensive.
• Unweighting Inefficiency: To generate unweighted events (essential for detector simulation), Monte Carlo generators use rejection sampling. For high-multiplicity processes, the acceptance efficiency of this step is often extremely low (e.g., $<0.02\%$), meaning the vast majority of CPU time is wasted on rejected events.
• Complexity of Realistic Setups: Previous attempts to use neural networks (NN) as surrogates for MEs were limited to idealized, single-partonic-channel scenarios. Realistic LHC simulations require multijet merging (combining different jet multiplicities), handling color degrees of freedom, biased phase-space sampling (to enhance rare kinematics), and process mapping. The interplay of these factors with surrogate unweighting had not been fully addressed.

2. Methodology

The authors extend a previously proposed two-stage rejection-sampling algorithm to realistic, multijet-merged event generation using the Sherpa event generator.

A. The Two-Stage Surrogate Algorithm

The method replaces the direct evaluation of the squared matrix element ($w_{ME}$) with a fast neural network surrogate ($s_{ME}$) in a two-step acceptance process:

1. First Step (Surrogate): A phase-space point is accepted with probability $p_1 = \min(1, s/w_{max})$. If accepted, the full ME is not yet calculated.
2. Second Step (Correction): For events passing step 1, the true ME ($w$) is calculated. The ratio $\tilde{x} = w/s$ is computed. The event is accepted with probability $p_2 = \min(1, \tilde{x}/\tilde{x}_{max})$.
   • This ensures the final sample is unbiased and statistically equivalent to standard generation.
   • Partial Unweighting: The authors utilize "reduced maximums" ($s_{max}, x_{max}$) rather than the strict global maximum. This allows some events to carry "overweights" ($>1$), significantly increasing acceptance efficiency.

B. Integration with Advanced Monte Carlo Features

The paper generalizes the algorithm to handle complex Sherpa features:

• Multijet Merging: The algorithm is applied to a merged sample of $Z+N$-jet processes ($N=0$ to $6$), where different multiplicities are combined using a merging scale ($Q_{cut}$) and Sudakov veto factors.
• Color Treatment: The authors compare color-summed (exact sum over color configurations) vs. color-sampled (probabilistic sampling) matrix elements. They find that the NN surrogate architecture (based on Catani-Seymour dipole factorization) works best with color-summed MEs, as color sampling introduces noise that degrades the surrogate's accuracy.
• Phase-Space Biasing: To study rare high-$p_T$ tails, a biasing function $h(\Phi)$ is applied. The surrogate is trained on the un-biased ME but the unweighting logic accounts for the bias to ensure unbiased final samples.
• Process Mapping: To reduce the number of required NNs, distinct partonic channels with identical kinematic structures (differing only by PDFs or flavor symmetries) are mapped to generic "process groups." Surrogates are trained only on these reduced groups.

C. Neural Network Optimization

• Architecture: Factorization-aware networks combining trained layers for non-universal parts with analytically calculated dipole functions.
• Loss Function Innovation: Instead of standard Mean Squared Error (MSE), the authors introduce a Gain Factor Loss Function. The network is first pre-trained with MSE, then fine-tuned to directly minimize the inverse of the effective CPU gain factor. This explicitly optimizes the network for the specific goal of computational speedup rather than just weight regression accuracy.
• Hyperparameters: Optimized via scans (e.g., 4 hidden layers, 128 nodes, Swish activation).

3. Key Contributions

1. Generalization to Realistic Setups: Successfully adapted surrogate unweighting from single channels to fully merged, multijet LHC simulations including Sudakov vetoes and phase-space biasing.
2. Optimized Training Strategy: Developed a two-stage training protocol (MSE $\to$ Gain Factor loss) that significantly improves the effective gain factor, particularly for low over-weight tolerances.
3. Comprehensive Benchmarking: Provided a rigorous comparison between three approaches for $Z$+jets production:
   • Baseline Color-Summed (Comix).
   • Baseline Color-Sampled (Sherpa default).
   • Surrogate Unweighting (applied to dominant high-multiplicity channels).
4. Realistic Timing Metrics: Defined "Effective Time per Event" and "Total Merged Process Time" (in HS23 Mega-years) that account for statistical dilution from overweights, phase-space generation, and parton showering overhead.

4. Results

The study focuses on inclusive $Z$+jets production at $\sqrt{s}=13$ TeV, merging tree-level matrix elements up to 6 final-state partons.

• Performance Gains:
  • For $Z+5$ and $Z+6$ jets (the most computationally expensive channels), the surrogate approach reduces the total event generation time by a factor of ~11 compared to the best baseline method (which is color-sampled for these high multiplicities).
  • Compared to the color-summed baseline, the speedup is even higher for $Z+5$ jets (factor of ~3.3), though color-summing is generally faster for $N < 5$.
• Resource Savings:
  • Generating a sample corresponding to $2 \times 3000 \text{ fb}^{-1}$(HL-LHC luminosity) for$Z+\leq 6$ jets takes 131 MHS23y with the best baseline method.
  • With the surrogate approach, this is reduced to 12 MHS23y, saving approximately 100 MHS23y of compute time.
• Efficiency:
  • The surrogate approach shifts the computational bottleneck. In the baseline, ME evaluation dominates (up to 96% of time). In the surrogate approach, ME evaluation drops to 15%, and Phase Space generation becomes the dominant cost (67%).
• Validation:
  • Physics observables (dilepton invariant mass, $p_T$, jet multiplicity, jet $p_T$ spectra) generated with surrogates are statistically indistinguishable from the baseline, confirming the method is unbiased.

5. Significance

• Feasibility for HL-LHC: The results demonstrate that generating full-statistics, high-multiplicity samples (up to 6 jets) for the HL-LHC is computationally feasible using current resources, provided surrogate unweighting is employed.
• Paradigm Shift: The work highlights that simply speeding up the Matrix Element is no longer sufficient; the entire event generation chain (including phase-space sampling) must be optimized. The method identifies that future gains will come from improving phase-space samplers (e.g., via neural importance sampling) and GPU acceleration.
• Future Outlook: The authors plan to extend this to Next-to-Leading Order (NLO) calculations and top-quark pair production, noting that the algorithm is generalizable to non-positive definite weights (NLO) by tracking weight signs.

In summary, this paper provides a robust, validated framework for accelerating high-multiplicity LHC event generation, offering an order-of-magnitude reduction in computational costs essential for future high-luminosity physics analyses.