A real-time demonstrator of track reconstruction with FPGAs at LHCb

This paper presents a real-time FPGA-based demonstrator for 30 MHz track reconstruction in the LHCb VELO detector, detailing its architecture, data distribution, and synchronization with alignment constants while processing live data during LHCb's Run 3 operations.

The Gist

Imagine the Large Hadron Collider (LHC) as a massive, high-speed train station where particles are the passengers. Every second, 30 million "bunches" of these particles crash into each other, creating a chaotic explosion of data. The LHCb experiment is like a giant camera trying to take a picture of every single crash to figure out what happened.

The problem? There is too much data. If you tried to save every single photo, your hard drive would fill up instantly, and the computer would freeze. Usually, a "bouncer" (a computer program) stands at the door and throws away most of the photos, keeping only the interesting ones. But as the train station gets busier (more collisions), the bouncer needs to work faster and smarter.

This paper describes a new, super-fast "bouncer" built using special computer chips called FPGAs. Here is how it works, explained simply:

1. The "Artificial Retina" (The Smart Eye)

The team built a system they call the "Artificial Retina." Think of it like a giant, high-tech security grid.

• The Grid: Imagine a checkerboard where every square is a tiny, independent worker.
• The Job: Each worker is assigned a specific "pattern" of a particle path (a track).
• The Process: When a particle hits a sensor, it sends a signal (a "hit"). The system doesn't just look for one pattern; it checks if that hit fits many different patterns at the exact same time.
• The Result: If a hit fits a pattern well, that worker gets "excited" (like a lightbulb turning on). If enough workers for a specific pattern get excited, the system says, "Aha! We found a track!"

2. The Traffic System (The Distribution Network)

The hardest part is getting the data from the sensors to the right workers.

• The Problem: One particle hit might fit several different patterns, meaning it needs to be copied and sent to multiple workers. This creates a traffic jam.
• The Solution: The team built a custom "highway system" made of optical cables (light-speed data). They designed a smart sorting machine (a switch) that organizes the traffic.
• The Optimization: Instead of randomly sending data, they arranged the workers so that similar patterns are grouped together. This is like organizing a library so that books on the same topic are on the same shelf, making it much faster to find what you need. This prevented the system from getting clogged up.

3. The Test Drive (The Demonstrator)

The team built a prototype (a "demonstrator") to test this idea.

• The Setup: They used 8 powerful computer boards connected by fiber-optic cables, all fitting inside a single server rack.
• The Target: They focused on a specific part of the detector called the VELO (Vertex Locator), which is like the "front door" of the experiment where the collisions happen first. They covered about 1/4 of this area.
• The Simulation: First, they fed the system fake data that mimics the real LHC collisions. The system ran for 10 days straight without crashing, processing data at a rate of 19 million events per second. That is incredibly fast! (The goal is 30 million, but they are very close).

4. The Real-World Test (Live Data)

The real test was using the system on live data while the LHC was actually running physics experiments.

• The Challenge: Real data is messy and changes constantly. The system also needed to use the most up-to-date "alignment constants" (think of these as the latest map coordinates) to know exactly where the sensors were.
• The Result: They built a special bridge to feed live data from the experiment's monitoring system into their prototype. The system ran smoothly during real physics runs in July and September.
• The Outcome: The tracks the prototype found looked just like the tracks found by the standard, slower software. It proved the system works in the real world without breaking anything.

The Bottom Line

This paper shows that a new type of hardware (FPGAs) arranged in a "Retina" pattern can act as a super-fast filter for particle physics data. It successfully processed real-time data from the LHC, handling millions of collisions per second without getting overwhelmed.

The team concludes that this technology is ready for the next big upgrade of the LHC (Run 4). By moving this heavy lifting onto these fast chips, they can save the main computers' power for other tasks, allowing the experiment to handle even more collisions in the future.

Technical Summary

Technical Summary: A Real-Time Demonstrator of Track Reconstruction with FPGAs at LHCb

Problem Statement
High Energy Physics (HEP) experiments face increasing data-flow and complexity challenges while Moore's Law shows signs of slowing. The LHCb experiment, specifically for its Run 3 and the future "Upgrade II" (which will increase luminosity by a factor of 5 to 10), requires advanced heterogeneous computing architectures to handle real-time data processing at the full LHC collision rate (30 MHz). Traditional CPU/GPU-based High Level Triggers (HLT) may struggle with the sheer volume of data and the need for zero-latency processing without time-multiplexing or excessive buffering. The challenge is to develop a scalable, parallel processing solution capable of reconstructing tracks in real-time directly from the detector readout.

Methodology
The authors present a demonstrator based on the "Artificial Retina" (Retina) architecture, a highly parallelized custom tracking processor implemented on Field-Programmable Gate Arrays (FPGAs).

• Algorithmic Approach: The Retina architecture discretizes the track parameter space into a matrix of cells ($M \times N$). Each cell represents a reference track and contains a computational engine that calculates an "excitation level" based on the compatibility of incoming detector hits with that reference track. This mimics a Hough transform. Hits contribute to the accumulator with a weight determined by their distance from the track's "receptor" (intersection with the detector layer), using a Gaussian distribution. Local maxima in the excitation map are identified by Track Processing Units (TPUs) to estimate candidate track parameters.
• Hardware Implementation: The demonstrator utilizes eight PCIe-hosted Stratix 10 FPGA boards (Bittware 520N) housed in a single server, interconnected via a fast optical network. The system is integrated into the LHCb Coprocessor TestBed facility, allowing it to process live data opportunistically during physics runs without interfering with the standard Data Acquisition (DAQ) system.
• Distribution Network: A core component is the custom distribution network built from modular splitters (2s), mergers (2m), and dispatchers (2d, 4d, 8d). This network routes hits from the DAQ readout to the relevant FPGA cells. To prevent network congestion caused by hit multiplication (where one hit may contribute to multiple tracks), the authors implemented an optimization strategy. This involves ordering TPUs to pair those with the most common hits, thereby delaying hit duplication to the layers nearest the TPUs. Additionally, the Post-Switch was modified to an 8x16 dispatcher configuration to limit line occupancy.
• Data Processing: The system was tested in two modes:
  1. Simulated Data: Events generated at nominal Run 3 luminosity ($2 \times 10^{33} \text{ cm}^{-2}\text{s}^{-1}$) were loaded into internal RAMs.
  2. Live Data: A custom data chain was developed to feed real LHCb collision data from the monitoring farm to the demonstrator. This chain included an "AlignmentChecker" to fetch real-time alignment constants from the Condition Database and a "DecodeTransform" to convert local sensor coordinates to the global LHCb system.

Key Contributions

• Prototype Demonstrator: Construction of a sizeable demonstrator covering 16 of the 38 VELO (Vertex LOcator) modules (specifically the right-hand side downstream quadrant), representing a significant portion of the actual detector.
• Optimized Distribution Network: Development of a specific ordering algorithm for the switch network that minimizes hit multiplication factors in intermediate layers, significantly improving throughput.
• Real-Time Integration: Successful integration of the FPGA-based system with the live LHCb DAQ environment, demonstrating the ability to process real collision data with updated alignment constants in real-time.
• Synchronization: Implementation of a mechanism to synchronize track reconstruction with the most up-to-date detector alignment constants, a critical requirement for accurate physics analysis.

Results

• Throughput: The system achieved an event rate of 19.0 MHz when processing simulated data at nominal Run 3 luminosity. This represents a massive improvement over a non-optimized switch configuration, which yielded only 170 kHz under the same conditions.
• Reliability: The system ran uninterruptedly for 10 days on simulated data and operated smoothly for approximately 60 days during live data taking (mid-July and September) without crashing or producing anomalous behavior.
• Accuracy: The reconstructed tracks from the demonstrator showed qualitative agreement with standard LHCb software reconstruction (HLT2) in terms of the two-dimensional distribution of track parameters ($u-v$ space). Bit-level consistency was confirmed by periodically injecting simulated data packets into the live data stream, which matched the output of a C++ bitwise emulator.
• Limitations: The achieved rate of 19.0 MHz is currently a factor of 1.58 below the target LHC average beam crossing rate of 30 MHz. The authors note that the final selection logic (acceptance cuts, clone killing) was not yet implemented in this demonstrator.

Significance and Claims
The paper claims that the Retina technology has reached a level of maturity suitable for real-world applications in future HEP experiments. The successful demonstration of processing live LHCb data at high rates, combined with the identification of specific firmware and architectural optimizations (such as further clock frequency increases and logic chain improvements), suggests that the target rate of 30 MHz is within reach using the current hardware generation.

Consequently, the LHCb collaboration has submitted a proposal to the LHC scientific committee to implement a first physics application of the Retina architecture in Run 4. This application aims to offload parts of the tracking reconstruction to FPGAs operating transparently at the readout level, thereby saving HLT1 computing power. The work validates the feasibility of using FPGA-based "Artificial Retina" architectures to handle the extreme data rates anticipated in future high-luminosity runs.