Deploying a Hybrid PVFinder Algorithm for Primary Vertex Reconstruction in LHCb's GPU-Resident HLT1

This paper presents the development and integration of a hybrid deep neural network for primary vertex reconstruction into LHCb's GPU-resident HLT1 framework, detailing the challenges of real-time constraints and data layout translation while outlining a roadmap for significant performance improvements through mixed-precision computing and model compression.

The Gist

Imagine the Large Hadron Collider (LHC) as the world's most chaotic, high-speed particle accelerator. Inside, protons smash together 30 million times every second. Each smash creates a tiny explosion of particles, and physicists need to figure out exactly where the explosion started (the "Primary Vertex") to understand what happened.

In the past, this was done with rigid rules. But now, the LHCb experiment is using Artificial Intelligence (AI) to do this job because it's much better at spotting patterns. However, there's a catch: the computer system has to make these decisions in a fraction of a second, or the data is lost forever.

Here is the story of how the team built a bridge between a powerful AI and a super-fast, rigid computer system.

1. The Problem: The "Speed Trap"

Think of the LHCb's computer system (called Allen) as a high-speed assembly line in a factory.

• The Rule: Every item (collision event) must be processed in under 400 microseconds (that's 0.0004 seconds).
• The Constraint: The factory has a fixed amount of storage space (memory) and only one worker (a single processing stream) who cannot stop to talk to anyone else. If the worker stops to grab a new tool or organize the workspace, the whole line backs up.

The team wanted to bring in a new, super-smart AI worker (PVFinder) to find the collision points. But this AI was trained to work in a "flexible" environment where it could grab tools, rearrange furniture, and talk to other workers. If they just dropped this AI into the rigid factory, it would break the rules, causing the line to stop.

2. The Solution: The "Translator"

The team built a special Translation Layer. Imagine this as a universal adapter plug or a bilingual interpreter.

• The Data: The factory speaks "Structure-of-Arrays" (SoA), which is like organizing data by columns (all X-coordinates together, all Y-coordinates together). The AI speaks "Tensors," which is like organizing data by rows (all data for one item together).
• The Adapter: Instead of physically moving the data from one format to another (which takes time), the adapter simply re-labels the boxes. It tells the AI, "Hey, look at this pile of boxes; it's actually your format now."
• Zero-Copy: This means no data is actually moved. It's like telling a librarian, "The books are still on the shelf, but I'm going to read them as if they were on my desk." This saves precious time.

3. How the AI Works (The Three-Stage Pipeline)

The AI, PVFinder, acts like a detective solving a crime scene in three steps:

1. The Detective (Fully Connected Network): It looks at the raw clues (9 features from each particle track) and creates a rough sketch of where the explosion might have happened. It does this using standard math (Native CUDA).
2. The Pattern Spotter (The CNN/UNet): This is the heavy lifter. It takes the rough sketch and uses a "neural net" (like a highly trained eye) to clean up the picture. It separates overlapping explosions and sharpens the image to find the exact center. This is the slowest part.
3. The Decision Maker (Peak Finding): It looks at the final map and says, "Okay, the highest peak here is the real explosion point."

4. The Current Reality: A Traffic Jam

When they first installed this AI into the factory, they found a problem.

• The Result: The AI was so smart, but also so heavy, that it slowed the entire factory down by 75%.
• The Analogy: Imagine a Formula 1 car engine (the AI) dropped into a bicycle frame (the factory). The engine is too big and hot; it's overheating the system.
• Why? The AI was eating up all the memory bandwidth (like a water hose spraying everywhere) and the computer's "thinking units" (processors) were sitting idle while waiting for data.

5. The Roadmap: Tuning the Engine

The team isn't giving up. They have a plan to make the AI 24 times faster by 2030. Here is their "tuning kit":

• Switch to "Lightweight" Math (FP16): Currently, the AI does calculations with high precision (like measuring a grain of sand with a ruler that has 10 decimal places). They plan to switch to "half-precision" (5 decimal places).
  • Analogy: It's like switching from a heavy steel hammer to a lightweight titanium one. You lose a tiny bit of precision (which doesn't matter for this job), but you can swing it twice as fast.
• Shrink the Brain (Model Compression): The AI currently has 64 "channels" (neural pathways). They plan to cut it down to 32.
  • Analogy: It's like taking a 100-page instruction manual and condensing it to 50 pages. The core instructions remain, but it's much faster to read.
• Fix the Workflow (Memory Optimization): They will rearrange how the AI grabs data so it doesn't have to walk back and forth across the factory floor.
  • Analogy: Instead of the worker running to the warehouse for every single screw, they will keep a box of screws right next to the workbench.

The Bottom Line

This paper is a proof-of-concept. They successfully proved that you can put a heavy, modern AI into a rigid, ultra-fast physics trigger system without breaking it.

Right now, the AI is too slow for the daily rush hour (Run 3). But with their new "tuning kit," they believe they can make it fast enough to handle the future (Run 4 and beyond) by 2030. They have built the bridge; now they just need to pave it with asphalt so the cars can drive across at full speed.

Technical Summary

1. Problem Statement

The LHCb experiment at the Large Hadron Collider (LHC) is undergoing a major upgrade for Run 3, increasing the collision rate to 30 MHz with an average of 5.6 primary vertices (PVs) per event. To handle this data volume, LHCb has transitioned to a fully software-based High-Level Trigger (HLT1) running on GPUs.

The core challenge is integrating advanced Machine Learning (ML) algorithms, specifically for primary vertex reconstruction, into the Allen framework (LHCb's GPU-resident trigger system) without violating its strict real-time constraints:

• Latency: Sub-400 µs processing time per event.
• Memory: Fixed memory pools with no dynamic allocation allowed during runtime.
• Execution: Single-stream execution to ensure deterministic behavior and avoid serialization.
• Conflict: Standard ML inference libraries (like cuDNN) rely on dynamic memory, multi-stream execution, and specific tensor layouts (NCHW), which conflict with Allen's Structure-of-Arrays (SoA) layout and deterministic execution model.

2. Methodology

The authors developed an inference engine for PVFinder, a hybrid deep neural network, and created a custom translation layer to bridge the gap between Allen's environment and standard ML libraries.

A. PVFinder Architecture

The algorithm reconstructs primary vertices from track parameters (9 features per track) using a three-stage pipeline:

1. Feature Extraction (FC): Six fully connected layers implemented in native CUDA. This processes track contributions into an 800-bin histogram (8 channels × 100 bins).
2. Spatial Pattern Recognition (CNN): A 5-layer UNet-style Convolutional Neural Network (currently 64 channels). It uses an encoder to capture local/contextual patterns and a decoder to refine probability distributions for vertex separation.
3. Peak Finding: A heuristic stage using learned thresholds to identify local maxima in the probability histogram, determining final vertex positions.

B. The Translation Layer

To integrate cuDNN into Allen without breaking its constraints, a three-phase translation layer was implemented:

• Prepare Phase: Creates non-owning tensor views that reinterpret Allen's SoA buffers as cuDNN-compatible tensors. Since the FC stage output is already in the required [B, C, L] format, minimal data reorganization is needed. Shape validation occurs at initialization to minimize per-event overhead.
• Execute Phase: Invokes cuDNN operations on Allen's single stream. A fixed 16 MB workspace arena is pre-allocated at initialization to satisfy cuDNN's memory needs without runtime allocation. This phase strictly avoids default CUDA stream operations to prevent implicit synchronization.
• Extract Phase: Maps cuDNN outputs back to Allen's SoA buffers. Data remains in the native layout until consumed by the peak-finding algorithm, minimizing data movement and cache pollution.

3. Key Contributions

• First Proof-of-Concept: Demonstrates the feasibility of deploying hybrid deep learning models within LHCb's strict, deterministic GPU trigger framework.
• Zero-Copy Translation Layer: A novel adapter that bridges Allen's SoA data layout with cuDNN's tensor format while maintaining zero-copy semantics and deterministic behavior.
• Performance Characterization: Detailed profiling of the integration, identifying specific bottlenecks (memory bandwidth, cache interference, and low SM occupancy) that standard standalone benchmarks miss.
• Optimization Roadmap: A concrete plan to achieve order-of-magnitude performance improvements to meet 2030 operational targets.

4. Results

Physics Performance

• Efficiency: >97% for events with 3–8 vertices.
• False Positives: 0.03 per event.
• Spurious Vertices: 0.6% per true PV.
• Precision Impact: Switching to FP16 (mixed precision) results in <0.5% efficiency degradation, which is acceptable.

Throughput Performance (Current State)

• Baseline HLT1: 100% relative throughput.
• FC Stage Only: ~95% throughput (5% overhead).
• Full Hybrid (FC + CNN): ~25% throughput (75% reduction).
• Bottleneck Analysis: The CNN stage dominates the overhead. Profiling revealed three main issues:
  1. Memory Bandwidth Saturation: Combined load of tracking and inference saturates the GPU.
  2. Cache Interference: PVFinder's working set conflicts with other algorithms' data structures.
  3. Low SM Occupancy: Current kernels achieve <50% occupancy due to suboptimal launch configurations and register pressure.

Optimization Projections (Target for 2030)

The authors propose a phased optimization strategy targeting a ~24x speedup to return throughput to ≥95%:

1. Mixed-Precision (FP16): Utilizing Tensor Cores for a 2x–4x speedup with negligible physics loss.
2. Model Compression: Reducing the UNet from 64 to 32 channels. Since convolution cost scales quadratically with channels, this yields a 4x speedup while maintaining >96% efficiency.
3. Memory & Kernel Tuning: Fusing memory layouts and optimizing thread block dimensions to increase SM occupancy from ~45% to 75–80%, providing a 1.5x speedup.

Combined, these optimizations are projected to reduce the CNN overhead from 75% down to ~3–5%, meeting the <5% regression target.

5. Significance

This work is critical for the future of high-energy physics triggers. It proves that state-of-the-art deep learning algorithms can be deployed in real-time, deterministic GPU environments, provided that:

• Co-design of data formats, memory management, and execution models is prioritized over "plug-and-play" library usage.
• System-level integration is performed early to identify bottlenecks (like cache interference) that isolated development misses.

The translation layer and optimization patterns established here serve as a template for integrating other ML applications (track fitting, particle ID, heavy flavor tagging) into LHCb's trigger system, enabling the experiment to fully leverage the increased luminosity of Run 3 and beyond.