A Modular Zero-Dead-Time Data Acquisition and Real-Time GPU Processing Platform for High Throughput Physics Experiments

This paper presents a modular, software-defined platform that integrates high-bandwidth PCIe digitizers with consumer GPUs to achieve continuous, zero-dead-time data acquisition and real-time processing for high-throughput physics experiments, successfully demonstrated in the WISPLC dark matter search with sustained 1 GB/s throughput and negligible data loss.

The Gist

Imagine you are trying to record a symphony orchestra playing at lightning speed. In the old days, if you wanted to analyze the music in real-time, you needed a massive, custom-built machine (like a specialized robot) that was incredibly fast but very expensive, hard to program, and difficult to change if you wanted to listen to a different instrument.

This paper introduces a new way to do this recording and analysis using a "modular" approach. Instead of a custom robot, the team built a system using standard, high-speed computer parts (like those found in gaming PCs) combined with a clever software program. Here is how it works, broken down into simple concepts:

1. The Problem: The "Traffic Jam"

In high-speed physics experiments, data comes in faster than a highway during rush hour.

• The Old Way: Traditional systems use specialized hardware (FPGAs) to handle this. It's like having a dedicated, super-fast police officer directing traffic. It works perfectly, but building and changing the police officer's instructions takes months of specialized training and costs a fortune.
• The New Way: This team realized they could use a standard computer's graphics card (GPU)—the same kind used for playing video games—to do the heavy lifting. It's like hiring a team of thousands of efficient, off-the-shelf workers instead of one expensive, custom-built robot.

2. The Solution: A "Zero-Dead-Time" Pipeline

The biggest fear in recording fast data is "dead time." This is a tiny fraction of a second where the system stops recording to process what it just heard. If you miss a beat, the data is ruined.

The authors built a system that claims to have zero dead time.

• The Analogy: Imagine a conveyor belt in a factory. Usually, when the belt stops to let a worker pack a box, the belt stops, and the next box waits.
• Their Trick: They built a system where the conveyor belt never stops. While one worker (the GPU) is packing the current box, another worker is already grabbing the next box, and a third is preparing the next one. They use a "callback" system, which is like a timer that says, "Hey, as soon as you have a full box of data, process it immediately, then get back to the belt instantly."
• The Result: They proved that over a 10-minute recording, they didn't miss a single "beat" of data. The system is so precise that if it did miss data, it would be less than one-trillionth of the total time.

3. The Hardware: A Custom "Soundproof Box"

Since they are using powerful computer parts (GPUs) that can create electrical noise, they had to be careful.

• The Shield: They built a custom aluminum box (a Faraday cage) to hold the sensitive recording card. Think of this like a soundproof booth for a singer. It keeps the "noise" from the computer's fans and power supplies from messing up the delicate physics signals they are trying to hear.
• Cooling: Because the box is tight, they added fans and heat sinks to keep the electronics from getting too hot, ensuring the recording stays stable for weeks at a time.

4. The "Three-Headed Monster" (Multi-GPU Setup)

To handle the massive amount of data, they didn't just use one graphics card; they used three.

• The Assembly Line: They split the work into three stages, like an assembly line in a car factory:
  1. GPU 1: Converts raw numbers into physical voltage (like translating a foreign language).
  2. GPU 2: Does the complex math (Fast Fourier Transforms) to turn the sound into a frequency spectrum (like turning a song into a sheet music score).
  3. GPU 3: Averages the results and calculates statistics.
• The Trade-off: Moving data between these three cards takes a little extra time (like passing a car part down a long line), but it allows them to use much more memory than a single card could hold. This lets them see very fine details in the data.

5. Real-World Success: The "Dark Matter" Hunt

They tested this system in a real experiment called WISPLC, which is looking for "dark matter" (invisible particles that make up most of the universe).

• The Win: Before this system, the experiment would have generated so much raw data that they would have needed to store 21 Terabytes every single day.
• The Fix: Because their system analyzes the data as it comes in (averaging it out immediately), they only needed to store the final, summarized results. This dropped their storage needs from 21 TB a day to less than 20 TB a month.
• Stability: The system ran continuously for a full month without crashing, overheating, or losing data.

Summary

The paper claims to have built a flexible, cheaper, and easier-to-update alternative to expensive, custom-built scientific hardware. By using standard computer parts and smart software, they created a "zero-dead-time" recording system that can handle massive data streams, analyze them instantly, and store only the important bits. They proved it works by successfully running a month-long dark matter experiment without a single glitch.

Technical Summary

Technical Summary: A Modular Zero-Dead-Time Data Acquisition and Real-Time GPU Processing Platform

Problem Statement
Modern high-throughput physics experiments generate escalating data volumes that challenge traditional data acquisition (DAQ) pipelines. Conventional systems often rely on offline processing, which is insufficient for flexible transient signal searches and storage management. While hardware-based solutions using Field Programmable Gate Arrays (FPGAs) or Radio Frequency Systems on a Chip (RFSoCs) offer deterministic latency and high efficiency, they present significant development bottlenecks. These include the requirement for specialized hardware description language (HDL) expertise (e.g., VHDL, Verilog), long compilation times, complex debugging cycles, and high entry costs. Furthermore, the rigid nature of fixed hardware architectures makes iterative algorithm development difficult in experimental environments.

Methodology
The authors propose a modular, software-defined DAQ platform that integrates high-bandwidth PCIe analog-to-digital converters (ADCs) with standard consumer-grade Graphics Processing Units (GPUs). The system is designed to achieve continuous, zero-dead-time data acquisition and real-time processing using NVIDIA CUDA.

• Hardware Architecture: The core system utilizes Spectrum Instrumentation SPCM series digitizers (M4i.4420-x8 and M2p.5941-x8) paired with a multi-threaded CPU and multiple CUDA-capable GPUs. To mitigate electromagnetic interference (EMI) from host components, the ADC card is physically isolated within a custom 10 mm-thick aluminum Faraday cage equipped with active cooling. The platform supports a triple-GPU configuration (two NVIDIA RTX 2080 Ti and one RTX A4000) to overcome the VRAM limitations of single consumer cards, enabling higher resolution bandwidths.
• Software Architecture: The system employs a multi-threaded C/C++ architecture using POSIX Threads (pthreads). It features a callback-driven design where a dedicated ADC worker thread manages the hardware FIFO ring buffer. When available data exceeds a user-defined threshold, a user-provided callback function is invoked. This abstraction allows users to implement custom processing logic (e.g., FFTs, filtering, statistical averaging) without managing low-level memory wrapping or hardware drivers.
• Processing Pipeline: The real-time processing pipeline involves converting interleaved int16 ADC data to floating-point voltages, applying windowing functions, and computing Fast Fourier Transforms (FFTs) using the cuFFT library. In the multi-GPU setup, the workload is distributed sequentially across three GPUs: type conversion on the first, FFT computation on the second (which holds the largest VRAM), and statistical averaging on the third. This sequential distribution avoids the complexity of strided memory copies across physical devices, though it introduces host RAM transfer overhead.

Key Contributions

1. Modular Software-Defined Architecture: Realization of a DAQ system that enables continuous high-throughput signal acquisition and real-time processing using standard PCIe digitizers and consumer GPUs, eliminating the need for specialized FPGA firmware.
2. Callback-Driven Design: Implementation of a flexible framework where users define processing logic via callbacks, enabling the real-time computation of derived statistics (power, variance, spectral averages) and easy adaptation to different signal processing algorithms.
3. Zero-Dead-Time Validation: In-situ demonstration of continuous acquisition capabilities, validated through phase continuity tests and long-term stability runs.
4. Multi-GPU Workload Distribution: A strategy to distribute FFT and averaging tasks across multiple GPUs to achieve high resolution bandwidths despite individual VRAM constraints.

Results

• Performance Benchmarks: The platform sustains continuous processing at sampling rates up to 500 MSa/s, managing data throughputs of 1 GB/s. At the maximum rate, the total real-time processing consumes approximately 80% of the available DAQ time, primarily due to inter-GPU memory transfer latencies routed through host RAM. At the lower 124 MSa/s rate used in the WISPLC experiment, processing usage drops to roughly 20%.
• Zero-Dead-Time Verification:
  • File Boundary Analysis: Tests injecting sinusoidal signals at various frequencies (9 Hz to 19.8 MHz) confirmed phase continuity across file boundaries. Statistical comparison with Monte Carlo simulations showed agreement with the zero-dead-time model, ruling out data loss during buffer wraparounds.
  • End-to-End Validation: A 10-minute continuous acquisition of a 3 MHz signal constrained fractional data loss to below $5.6 \times 10^{-12}$ (equivalent to a hypothetical dead time of <3.33 ns), which is less than a single sample interval at 125 MSa/s.
• Long-Term Stability: The system operated continuously for one month without crashes, memory corruption, or performance degradation. The custom cooling solution maintained stable ADC temperatures (~48°C), preventing thermal drift.
• Application Impact: In the WISPLC dark matter experiment, the platform operated at 124 MSa/s with a 0.1 Hz resolution bandwidth. Real-time spectral averaging reduced daily data storage requirements from 21 TB to less than 20 TB per month.

Significance
The paper establishes that high-performance, software-defined DAQ platforms utilizing consumer GPUs can effectively replace specialized, rigid hardware pipelines (like FPGAs) for medium to large-scale laboratory setups. By leveraging standard C/C++ and CUDA programming, the system significantly lowers the entry barrier for high-throughput experiments, allowing for rapid iteration of signal processing algorithms without hardware redesigns. The demonstrated ability to maintain zero dead-time at high sampling rates while performing complex real-time averaging offers a flexible, cost-effective alternative to traditional digital baseband converter (DBBC) systems and FPGA-based solutions. The authors note that while the current implementation addresses specific physics needs, the architecture is adaptable for broader applications, including signal synthesis and network analysis, provided appropriate RF hardware is integrated.