End-to-end optimisation of HEP triggers

This paper proposes and demonstrates a constrained end-to-end differentiable framework for High-Energy Physics trigger systems that jointly optimizes all processing stages against a unified physics objective, achieving a 2-4x improvement in true-positive rates for Higgs boson pair production while maintaining interpretability and calibration constraints.

The Gist

Imagine you are running a massive, high-speed sorting factory. Every second, a conveyor belt dumps 40 million raw, messy packages onto the line. Your job is to find the few "golden tickets" (rare, valuable physics events) hidden among millions of junk packages (background noise).

The problem? You only have a tiny amount of time to make a decision, and you can only keep a few hundred packages a second. If you keep the junk, the factory chokes. If you throw away the gold, you miss the discovery.

This is the daily reality of High-Energy Physics (HEP) experiments like those at CERN. The paper you provided proposes a revolutionary way to fix the "sorting machine" (called a Trigger System) by changing how we design it.

Here is the explanation in simple terms, using analogies.

1. The Old Way: The "Specialist Assembly Line"

Traditionally, these sorting factories are built like a relay race with specialized runners.

• Runner 1 (Quantization): Takes the raw package and shrinks it to fit in a small box. They are trained to shrink things perfectly without losing detail.
• Runner 2 (Denoising): Takes the shrunk box and tries to remove the dirt and scratches. They are trained to make the picture as clean as possible.
• Runner 3 (Clustering): Groups the items together. They are trained to group things accurately.
• Runner 4 (Calibration): Weighs the final group. They are trained to be precise on the scale.

The Flaw: Each runner is a world-class expert at their specific job. But they don't talk to each other. Runner 1 might shrink the box so much that Runner 2 can't see the dirt anymore. Runner 2 might clean the picture so aggressively that they accidentally erase a tiny "golden ticket."

Because everyone optimizes for their own local goal (perfect shrinking, perfect cleaning), the final result isn't the best possible outcome for finding the gold. It's like a choir where every singer is perfect at their own note, but they aren't singing in harmony, so the song sounds terrible.

2. The New Way: The "End-to-End Orchestra"

The authors propose a new design: End-to-End Optimization.

Instead of training each runner separately, they treat the entire factory as one giant, connected brain. They train the whole system at once with one single goal: "Find the Golden Tickets."

• The Magic: The system learns that sometimes, it's okay to be slightly messy in the "cleaning" stage if it helps the "weighing" stage find the gold better.
• The Trade-off: It might decide to keep a little bit of "dirt" (noise) because that dirt actually helps identify a specific type of gold. Or, it might shrink the box in a weird way that looks bad to a human, but is perfect for the final decision.

The system learns to make global trade-offs that no single specialist could figure out on their own.

3. The "Hardware" Constraints

You might ask, "Can we actually build this? Real factories have rules: limited space, limited electricity, and strict time limits."

Yes! The paper shows that they can bake these rules directly into the training.

• Bandwidth: They taught the system how to compress data (like a Zip file) while learning to find the gold, ensuring the data fits through the narrow pipes.
• Speed: They taught the system to be fast enough to run on the specific computer chips (FPGAs) used in the lab.

It's like training a race car driver who knows they have a small engine and a bumpy road. They don't just drive fast; they learn the exact line that balances speed with the car's limitations to win the race.

4. The Results: A Massive Win

When they tested this new "Orchestra" approach against the old "Assembly Line" approach using a simulation of the Higgs Boson (a rare particle often called the "God Particle"):

• The Old Way: Caught 1 out of every 100 Higgs events.
• The New Way: Caught 2 to 4 times more Higgs events, while still throwing away the same amount of junk.

Why does this matter?
In particle physics, finding rare events is like looking for a needle in a haystack. If you can double or quadruple your success rate without building a bigger machine, you effectively extend the life of the experiment. The authors calculate this could save the Large Hadron Collider (LHC) up to 40 years of data-taking time. That's a massive saving in money and scientific potential.

Summary

• The Problem: Old trigger systems are like a team of specialists who don't talk to each other, leading to a sub-optimal final result.
• The Solution: Train the whole system together as one unit, optimizing for the final goal (finding rare particles) rather than intermediate steps.
• The Analogy: Moving from a relay race of specialists to a synchronized orchestra where every musician adjusts their playing to make the whole song perfect.
• The Outcome: A smarter, faster, and more efficient filter that finds significantly more "gold" in the data, potentially revolutionizing how we discover new physics.

This paper proves that by letting the whole system "think" together, we can solve problems that were previously thought to be impossible under strict hardware limits.

Technical Summary

Here is a detailed technical summary of the paper "End-to-end optimisation of HEP triggers" by Clarke Hall et al.

1. Problem Statement

High-Energy Physics (HEP) experiments, such as those at the Large Hadron Collider (LHC), face extreme data rates (up to hundreds of terabytes per second) that necessitate real-time filtering systems known as triggers. These systems must reduce event rates from 40 MHz to ~1 kHz for permanent storage while preserving sensitivity to rare physics processes (e.g., Higgs boson pair production).

The Core Challenge:
Traditionally, trigger systems are constructed as sequential, modular chains of algorithms (e.g., quantization $\to$ denoising $\to$ clustering $\to$ calibration $\to$ selection). Each module is optimized independently using local objectives (e.g., minimizing Mean Squared Error for denoising).

• Limitation: This "sequential optimization" approach fails to guarantee global optimality. A locally optimal step (e.g., perfect pixel-level denoising) may not be optimal for the final physics goal (e.g., distinguishing signal from background).
• Gap: There is a lack of frameworks that jointly optimize the entire trigger chain, including data encoding and hardware constraints, under a single unified physics objective.

2. Methodology

The authors propose a constrained end-to-end (E2E) optimization framework where the entire trigger chain is treated as a single differentiable system.

A. Mathematical Formulation

• Objective: The trigger is modeled as a discriminant $y(x; \theta)$ parameterized by $\theta$. The goal is to minimize the expected risk $R$ (loss function) over the event distribution $p(x, c)$.
• Joint Loss Function: The framework combines classification and reconstruction tasks:
  $$L = (1 - \alpha)C[f_\phi(y)] + \alpha D(t, y)$$
  Where $C$ is a classification term (Binary Cross-Entropy), $D$ is a reconstruction term (distance to truth), and $\alpha$ balances the two. A learnable monotone bijection $f_\phi$ decouples the scales of classification scores and physical values.
• Multi-Trigger Optimization: For conditionally independent triggers (e.g., different jets), the total loss is the sum of individual losses, allowing simultaneous optimization.

B. Hardware Constraints Integration

The framework explicitly incorporates hardware limitations into the training process via Quantization-Aware Training (QAT) and learnable data encoding:

1. Learnable Quantization: The discretization of detector data (quantization bin widths) is treated as a trainable parameter. The system learns the optimal encoding rule to maximize physics performance under bandwidth constraints.
2. Algorithmic Constraints: Model compression (e.g., bit-precision reduction) and latency budgets are enforced during training.
3. Monotonicity: Calibration layers are constrained to be monotonic functions of transverse momentum ($p_T$) to ensure stability in threshold-based selections, a critical requirement for LHC triggers.

C. Experimental Setup

• Case Study: A realistic hardware jet trigger inspired by the ATLAS High-Luminosity LHC (HL-LHC) design.
• Pipeline: Detector Images $\to$ Learnable Quantization $\to$ CNN-based Denoising $\to$ Cone Clustering $\to$ Neural Network Calibration $\to$ Trigger Decision.
• Constraints: Simulated 2 Tbps bandwidth limit and 10 $\mu$s latency budget.
• Datasets: Simulated proton-proton collisions at $\sqrt{s}=14$ TeV, including rare signals (Higgs pair production, Top quark pairs) and background (QCD di-jets) with heavy pile-up ($\langle\mu\rangle \approx 200$).

3. Key Contributions

1. Unified Differentiable Framework: The first implementation of a full HEP trigger chain (from raw detector readout to final decision) as a single differentiable graph optimized via gradient descent against a global physics objective.
2. Learnable Data Encoding: Demonstrated that the quantization rule (how analog detector signals are digitized) can be optimized end-to-end. The system learns to prioritize resolution in kinematic regions most relevant for signal discrimination rather than uniform resolution.
3. Task-Aware Trade-offs: Showed that the E2E system learns to sacrifice local metrics (e.g., perfect pixel-level reconstruction) to improve global metrics (e.g., signal efficiency). For instance, the E2E denoiser allows more noise pixels if they help preserve signal features, whereas the sequential model aggressively removes all noise.
4. Joint Classification & Calibration: Successfully integrated object calibration (regression) and event classification (binary decision) into a single loss, resolving scale differences via learnable bijections.

4. Results

The E2E framework was compared against a standard Sequential Optimization baseline (where each stage is trained independently using MSE or standard classification losses).

• Performance Gains:

  • Signal Efficiency: At a fixed False Positive Rate (FPR) of $10^{-3}$, the E2E approach achieved a **2x to 4x improvement** in True Positive Rate (TPR) for Higgs boson pair production ($HH \to b\bar{b}b\bar{b}$) compared to the sequential approach.
  • AUC: Significant improvements in Area Under the ROC Curve (AUC) across all signal channels (e.g., $0.90$vs$0.88$for$p_{T,1}$ in ggF HH).
  • Generalization: Improvements were observed even for triggers (like $H_T$ and $H^{miss}_T$) that were not explicitly included in the optimization loss, due to better underlying noise suppression and calibration.

• Intermediate Object Behavior:

  • Quantization: The E2E quantizer developed non-uniform bin widths, compressing low-energy regions and expanding high-energy regions critical for discrimination, unlike the uniform MSE-optimal sequential quantizer.
  • Calibration: The E2E calibration network learned an optimal, $|\eta|$-dependent $p_T$ threshold for seeding clusters, effectively suppressing low-$p_T$ background clusters that the sequential model retained.

• Physics Impact: The authors estimate that the efficiency gain is equivalent to extending the HL-LHC data-taking period by up to 40 years for Higgs pair production studies.

5. Significance

• Paradigm Shift: This work challenges the decades-old modular design philosophy of HEP triggers. It proves that "locally optimal" components do not necessarily yield a "globally optimal" system.
• Hardware-Software Co-Design: By optimizing data encoding and model compression jointly with physics objectives, the framework enables a new level of co-design between detector readout electronics and reconstruction algorithms.
• Broad Applicability: While demonstrated on a jet trigger, the framework is applicable to other trigger objects (electrons, photons), High-Level Triggers (HLT), and other real-time scientific data acquisition systems (e.g., neutrino experiments, astroparticle observatories).
• Interpretability: Crucially, the E2E system maintains interpretable intermediate physics objects (clusters, calibrated momenta) and respects physical constraints (monotonicity), addressing a common criticism of "black box" deep learning in physics.

In conclusion, the paper establishes that end-to-end optimization is a practical and highly effective paradigm for next-generation real-time event selection, offering substantial gains in discovery potential for rare physics processes.