A converged architecture for processing 32 Tbps of physics data in real-time at the LHCb experiment

Imagine a giant, high-speed camera that doesn't take pictures of sunsets or puppies, but of the universe's tiniest building blocks smashing together. This is the world of particle physics, where scientists use massive machines called colliders to fire particles at each other billions of times every second. When these particles crash, they create a chaotic explosion of new particles, like shattering a vase to see what's inside. The goal is to catch a glimpse of something rare and new hidden in that debris. But here's the catch: the camera is so fast and the collisions so frequent that it produces a mountain of data—so much that it would fill millions of hard drives every second. If scientists tried to save every single piece of data, their computers would melt, and their storage would run out instantly.

To solve this, physicists use a "filter" or a "trigger." Think of it like a bouncer at a very exclusive club. The bouncer has to decide in a split second which collisions are interesting enough to keep and which are just boring background noise to throw away. For decades, these bouncers were a mix of specialized hardware and computer software. But as the machines got faster, the old bouncers couldn't keep up. They were too slow and too rigid, often throwing away interesting events just because they didn't look like what the hardware expected. The big question became: How do we build a filter that is fast enough to handle a flood of data, smart enough to look at the whole picture, and flexible enough to change its mind on the fly?

This paper tells the story of how the LHCb experiment at the Large Hadron Collider solved this problem with a massive upgrade. The team built a new system that acts like a super-efficient, all-in-one factory. Instead of using a slow, multi-step process with different teams handling different parts of the job, they created a "converged architecture." This means they combined the data collection, the sorting, and the filtering into one streamlined process running on powerful graphics cards (GPUs) usually found in gaming computers. They managed to process a staggering 32 terabits of data every second in real-time. That's the highest data rate ever handled by software in any physics experiment. By removing the old, rigid hardware filters and replacing them with a smart, flexible software system, they didn't just keep up with the speed; they actually improved the quality of the science they can do, ensuring that no rare, exciting event slips through the cracks.

The Problem: A Flood of Data

The Large Hadron Collider (LHC) smashes particles together billions of times a second. Every time they collide, it's called an "event." In the past, the LHCb experiment used a two-step filter system. First, a "low-level trigger" made of specialized, custom-built electronics would quickly scan the data and throw away most of the boring collisions. Only the ones that looked promising would be passed to a "high-level trigger" (software) for a deeper look.

However, this old system had a flaw. Because the hardware filter only looked at a few specific sensors, it sometimes missed interesting events that happened elsewhere in the detector. Also, it was hard to change the rules once the hardware was built. The LHCb team decided to get rid of the first, hardware-based filter entirely. This meant their system had to handle 40 times more data than before. They needed to process a raw data stream of 32 terabits per second (Tbps) and filter it down to the most interesting events in real-time. If they failed, the data would be lost forever.

The Solution: A Converged Factory

The team built a new system using 164 servers (computers) that act as a single, massive machine. Here is how they made it work:

1. The "Zero-Copy" Highway
Usually, when data moves from a network card to a computer's memory, and then to a processor, it has to be copied multiple times. This is like moving a box from a truck to a warehouse, then to a shelf, then to a display case. It takes time and energy. The LHCb team used a "zero-copy" technique. Imagine a conveyor belt that runs straight from the truck, through the warehouse, and directly into the display case without ever stopping to be moved by hand. This allowed them to move data incredibly fast without slowing down the computer.

2. The GPU Powerhouse
Instead of using standard computer processors (CPUs) for the heavy lifting, they used Graphics Processing Units (GPUs). These are the chips usually found in video game computers, designed to handle thousands of tasks at once. The team realized that reconstructing the paths of particles is a lot like rendering a complex 3D video game scene. By putting the entire filtering process on GPUs, they could process thousands of collisions simultaneously.

3. Smarter Scheduling
One challenge was that not all collisions are the same size or complexity. Some are quick to analyze; others take longer. If the system tried to process them one by one, it would get stuck waiting. The team developed a clever "scheduler" that groups events together into batches. It's like a chef preparing a meal: instead of cooking one dish at a time, they prep all the vegetables, then chop all the meat, then cook everything together. This keeps the GPUs busy and efficient. They also used a "multi-event scheduler" that looks at the whole batch and decides the best order to run the steps, ensuring the most expensive calculations are only done on the events that actually need them.

4. The Network
To connect all 164 servers, they built a custom network using InfiniBand, a technology often used in supercomputers. They designed the network so that every server could talk to every other server efficiently, avoiding traffic jams (a problem known as "incast"). This allowed them to assemble the pieces of each collision from different parts of the detector and send them to the GPUs for filtering without any bottlenecks.

The Results: Breaking Records

The team tested their system and found it worked exactly as planned.

• Speed: They successfully processed the full 32 Tbps of incoming data in real-time.
• Filtering: The system applied a filter that reduced the data by a factor of 30:1. This means for every 30 collisions, only 1 was kept for further study.
• Scalability: When they tested the system with more servers, it scaled up perfectly. The system could handle the full design rate of the LHC, which is 30 million collisions per second.
• Efficiency: The system used 164 servers equipped with 328 GPUs (two per server). The total computing power was massive, but the design was efficient enough that they didn't need thousands of machines like in the past.

The paper confirms that this system is capable of processing the highest data rate in software ever achieved in the field of physics. It proved that you can remove the old, rigid hardware filters and replace them with a flexible, software-based system that is faster and smarter.

Why It Matters

This isn't just about handling more data; it's about seeing more of the universe. By removing the hardware filter, the LHCb experiment can now look at the entire detector for interesting events, not just the parts the old hardware was programmed to watch. This means they might find rare particles or new physics that were previously invisible. The success of this "converged architecture" sets a new standard for how future particle physics experiments will be built, showing that with the right mix of off-the-shelf technology and clever software, we can push the boundaries of what is possible in science.