A Scalable Two-Stage Back-End Architecture for 400GbE Data Acquisition Systems

In the heart of Europe, beneath the border of France and Switzerland, lies the Large Hadron Collider, a machine so vast it circles the earth's crust to smash particles together at nearly the speed of light. When these collisions occur, they generate a flood of data, a torrent of information that must be captured, sorted, and analyzed before it vanishes. This task falls to the data acquisition system, a complex network of electronics that acts as the experiment's nervous system. It must gather signals from thousands of tiny sensors scattered across a massive detector, each sensor speaking in a specialized, radiation-hardened language. The challenge is not just to listen to these whispers, but to gather them from every corner of the machine and combine them into a single, roaring stream of information that can be processed by computers. As the machine prepares for its next, more powerful phase, the volume of this data is set to explode, threatening to overwhelm the very wires and chips designed to carry it.

For decades, scientists have relied on powerful, expensive computer chips called field-programmable gate arrays to perform this heavy lifting. These chips are like universal translators, capable of decoding the unique signals from the sensors and converting them into a format that standard computers can understand. However, as the data rates climb toward hundreds of terabits per second, the traditional approach of using the most powerful, expensive chips for every single connection is becoming too costly and physically difficult to manage. The wires inside these massive chips are running out of room, and the power required to keep them cool is becoming a limiting factor. The researchers at CERN, specifically the team working on the LHCb experiment, faced a future where the number of data links would triple, requiring a new way to think about how data moves from the detector to the computer farm.

To solve this, the team proposed a two-step architecture that shifts the workload away from the most expensive components. Instead of forcing a single, massive chip to handle everything, they designed a system that breaks the job into two distinct stages. In the first stage, a mid-range computer chip, which is less expensive and more abundant, takes the incoming signals from the sensors. This chip acts as a local collector, gathering dozens of these individual streams and bundling them together into a few faster connections. It does not try to do everything at once; it simply organizes the traffic and prepares it for the next leg of the journey.

The second stage is where the innovation truly shines. The bundled data streams arrive at a specialized networking component, a device designed specifically to merge multiple high-speed channels into one massive highway. This component, known as an Ethernet physical layer device, takes three separate streams of data and fuses them into a single, incredibly fast connection capable of carrying 400 gigabits of information per second. This approach allows the system to use standard, off-the-shelf networking technology, which is produced in huge quantities for the global internet and data centers, rather than relying on custom, bespoke solutions that are rare and expensive. By separating the decoding task from the final aggregation, the design avoids the bottlenecks that plague older systems.

The researchers built a working prototype to test if this idea held up under real-world conditions. They set up a test environment that mimicked the flow of data from the detector, generating three separate streams of information, each moving at 100 gigabits per second. These streams were fed into their new two-stage system, which then combined them into a single 400-gigabit stream. The results were precise and encouraging. Over the course of testing, the system processed more than 610 million packets of data, totaling five terabytes of information. Throughout this massive transfer, the system did not drop a single packet, nor did it lose the order of the data. The speed remained steady, matching the target rates without any signs of congestion or delay. The measurements showed that the error rate was so low it was effectively zero for practical purposes, proving that the hardware could handle the immense pressure of the data flow without faltering.

Beyond the technical performance, the new design offered a significant financial advantage. The researchers calculated the cost of handling each individual data link and found that their two-stage approach was substantially cheaper than the traditional methods. By replacing expensive, high-end chips and complex network switches with their streamlined architecture, they reduced the cost per connection by nearly 29 percent compared to the standard solution. This saving comes from using mid-range chips that are cheaper to buy and a specialized networking device that aggregates the data more efficiently, eliminating the need for intermediate layers of expensive hardware. The study suggests that this method is not just a theoretical possibility but a practical, scalable solution that can be deployed as the data demands of particle physics continue to grow.

The work confirms that it is possible to build a data acquisition system that is both powerful and affordable by leveraging the mass production of modern networking technology. The two-stage design successfully decouples the sensitive, radiation-hardened front end from the high-speed back end, allowing each part to operate at its optimal efficiency. The researchers demonstrated that standard data streams, without the need for specialized, high-latency protocols, can sustain the massive throughput required for next-generation experiments. This finding offers a clear path forward for the LHCb experiment and other large-scale scientific endeavors, ensuring that as the machines get bigger and the data gets heavier, the systems designed to capture them can keep pace without breaking the bank.