AURORA: A High Performance DAQ Framework for Next-Generation Rare-Event Search Experiments

This paper introduces AURORA, a scalable, high-performance, and experiment-agnostic distributed data acquisition framework designed to meet the demanding bandwidth requirements of next-generation rare-event search experiments like PandaX-xT, achieving a benchmark throughput of over 3 GB/s.

The Gist

Imagine you are running the world's most sophisticated security camera system for a giant, invisible vault. This vault is the PandaX-xT experiment, a massive tank of liquid xenon designed to catch the ghostly particles of Dark Matter.

The problem? The vault has thousands of ultra-sensitive cameras (over 3,000 of them!). When a particle hits the xenon, it creates a tiny flash of light. The cameras need to record these flashes instantly. But here's the catch: the data is coming in so fast that it's like trying to drink from a firehose while holding a tiny cup. If you spill even a drop, you might miss the discovery of a lifetime.

The old system (used in the previous PandaX-4T experiment) was like a reliable delivery truck, but it was too slow for the new, massive firehose. It started to choke when the data got too heavy.

Enter AURORA. Think of AURORA not just as a computer program, but as a super-efficient, high-speed logistics network designed to catch that firehose, organize the water, and pour it into storage tanks without spilling a single drop.

Here is how AURORA works, broken down into simple concepts:

1. The Frontline: The "Data Collectors" (DAQ Readers)

Imagine you have 3,000 cameras. You can't have one person trying to watch all of them. So, AURORA hires a team of specialized workers called DAQ Readers.

• The Job: Each worker stands next to a group of cameras (digitizers). Their only job is to grab the raw video feed the moment it happens.
• The Trick: They don't stop to think about what the video means. They just grab it, put it in a temporary bucket, and shout, "Got it!" to the next station. They are fast, automated, and never get tired.

2. The Hub: The "Traffic Controller" (The Collector)

This is the brain of AURORA. All the workers shout their data to a central hub called the Collector.

• The Problem: The workers are shouting at different speeds. Worker A sends a packet at 10:00:01, but Worker B is slow and sends a packet from 10:00:00 after Worker A's. If you just dumped them into a box, the video would be scrambled.
• The Solution (The Magic): AURORA uses a clever trick called Multi-Level Buffering.
  • Imagine the Collector has a giant conveyor belt with 100 slots.
  • Instead of sorting every single piece of data immediately (which would be too slow), it puts data into slots based on time windows (e.g., "Everything that happened between 10:00:00 and 10:00:01 goes in Slot 1").
  • It waits a few seconds to make sure no late-arriving data is missing.
  • Then, it takes the oldest slot, sorts the data perfectly inside that slot, and moves it to the next stage.
  • Why this is cool: It's like a post office that doesn't try to sort every single letter the second it arrives. Instead, it fills up a "10:00 AM" bin, waits a moment, sorts that bin perfectly, and then ships it out. This keeps the line moving even when the mail truck is overloaded.

3. The Storage: The "Vault" (Disk Writing)

Once the data is sorted and organized, it needs to be saved to a hard drive.

• AURORA has a dedicated "writer" who takes the perfectly sorted data and slams it onto a super-fast NVMe SSD (a type of ultra-fast hard drive).
• Because the data was already sorted by the "Traffic Controller," the writer doesn't have to stop and rearrange anything. They just write, write, write.

4. The Safety Net: "No Data Lost"

In the old system, if the network got clogged, data would get dropped (like a letter falling on the floor).

• AURORA is built with Redundancy and Flow Control. It has huge buffers (waiting rooms) that can hold massive amounts of data.
• If the storage drive gets slow, the buffers fill up, but the workers keep working. The system is designed to handle bursts of data (like when scientists turn on a bright calibration light) that are twice as fast as normal. It can handle up to 1.6 Gigabytes per second (that's like downloading 300 HD movies every second!) without breaking a sweat.

5. The "Remote Control" (HTTP Interface)

Scientists don't want to type complex code to start the experiment. AURORA has a simple Web Interface.

• Think of it like a smart home app. The scientists can click a button on their phone or laptop to say "Start Recording," "Stop," or "Check Status."
• The system talks back, telling them exactly how much data it's collecting and if everything is healthy.

Why Does This Matter?

The PandaX-xT experiment is looking for the "Holy Grail" of physics: Dark Matter. These particles are so rare that they might only hit the detector once in a while.

• If the data system misses a single flash because it was too slow or crashed, that's a missed discovery.
• AURORA ensures that every single flash is caught, organized, and saved perfectly.

In summary: AURORA is a high-speed, self-correcting data highway. It takes a chaotic flood of information from thousands of sensors, organizes it into neat, time-ordered packages, and delivers it to storage faster than a human could blink, ensuring that the next great discovery in physics isn't lost in the noise.

Technical Summary

1. Problem Statement

The next-generation PandaX-xT dark matter experiment requires a significant upgrade in Data Acquisition (DAQ) capabilities compared to its predecessor, PandaX-4T.

• Scale: The detector will feature up to 8,000 readout channels (initially ~3,000 for PandaX-20T), a massive increase from previous iterations.
• Data Volume: The system must sustain a continuous data throughput of 1.0 GB/s during normal operation and handle short-term bursts up to 1.6 GB/s during calibration runs.
• Limitations of Legacy Systems: The existing PandaX-4T DAQ system, designed for 800 MB/s, failed to sustain rates around 600 MB/s during prototype testing for the larger setup. This highlighted an urgent need for a more robust, scalable, and high-throughput solution capable of ensuring zero data loss and maintaining strict time-ordering for physics analysis.

2. Methodology: The AURORA Framework

The authors developed AURORA (Adaptable Unified Real-time Online Readout Architecture), a high-performance, distributed DAQ framework. The design prioritizes modularity, low latency, and efficient resource utilization.

System Architecture

• Distributed Topology: The system separates data acquisition from aggregation.
  • Frontend (daq_reader): Runs on multiple independent DAQ servers. Each server connects to up to 28 digitizer boards via optical links (SiTCP). It reads raw data, buffers it, and streams it to the central node.
  • Aggregation (collector): A central server receives data streams from all daq_reader instances via 10 Gbps SFP+ links. It is responsible for sorting, buffering, and writing data to disk.
• Multi-Stream Handling: The architecture supports logical data streams (e.g., TPC, VETO). The collector maintains separate buffering and sorting queues for each stream to preserve internal data ordering while allowing concurrent processing.
• External Services:
  • PostgreSQL: Stores configuration parameters (thresholds, trigger logic).
  • InfluxDB: Receives real-time monitoring metrics (bandwidth, status).
  • Kafka: Publishes metadata (run/file info) to trigger downstream automated processing pipelines.

Key Technical Innovations

1. Multi-Level Buffering & Deferred Sorting:
   • Instead of sorting immediately upon receipt, the collector uses a BufferManager with fixed-size "timed buffers" (100 ms windows, 512 MB capacity).
   • Incoming data blocks are placed into these buffers based on their embedded timestamps.
   • Sorting occurs after a buffer is filled and a time delay (6 seconds) has passed to account for network jitter and clock drift. This decouples data reception from disk I/O, preventing back-pressure.
2. Asynchronous Producer-Consumer Model:
   • Built on the Asio C++ library, the system uses thread pools for network I/O and data processing.
   • daq_reader uses asynchronous send operations, while collector uses a dedicated file-writing thread. This ensures that network reception and disk writing do not block each other.
3. Clock Synchronization:
   • To handle drift between the digitizer hardware clock and the collector's system clock, the system periodically queries the global trigger board. It calculates the time difference ($\delta t$) and applies micro-adjustments ($\pm 1$ or $2$ ms) to the processing window, ensuring accurate time-ordering.
4. State-Machine Control:
   • The system operates via a strict state machine (IDLE $\to$ INITIALIZING $\to$ STARTING $\to$ RUNNING $\to$ STOPPING). Commands (INIT, START, STOP) are only executed when all components acknowledge the transition, preventing race conditions.

3. Key Contributions

• Scalable Architecture: A modular design that allows horizontal scaling. If a single collector cannot handle the load, the system can deploy multiple collectors, each handling a subset of streams, with data remaining globally aligned via timestamps.
• High-Performance I/O Pipeline: The separation of data reception, in-memory sorting, and disk writing allows the system to sustain throughput far exceeding the design target.
• Experiment-Agnostic Design: While built for PandaX, the framework is adaptable to other experiments provided the raw data format includes reliable timestamps. Only the data parsing layer requires modification.
• Integrated Automation: Full integration with Kafka for metadata and automated downstream processing, enabling real-time monitoring and alert generation.

4. Results and Performance Evaluation

The framework was evaluated through benchmarks, stability tests, and real-world deployment.

• Throughput Benchmarks:
  • Copy to Timed Buffer: Achieved >14 GB/s (measured via Google Benchmark), confirming network reception is not a bottleneck.
  • Copy to Output Buffer: Achieved >3 GB/s, identifying this as the limiting factor but still well above the 1.6 GB/s target.
  • Disk Write: The NVMe SSD subsystem sustained 5.47 GB/s (XFS formatted).
  • Conclusion: The system can sustain >3 GB/s total throughput, double the required 1.6 GB/s.
• Stability Test:
  • On the PandaX-20T prototype (1,400+ channels), the system ran continuously for 58 hours without errors or data corruption, exceeding the 24-hour design goal.
• Real-World Validation (PandaX-4T):
  • Deployed during high-rate calibration runs (AmC, DD/DT generators, $^{220}$Rn injection).
  • Sustained average rates of 800 MB/s and peak rates of 900 MB/s with zero interruptions.

5. Significance

AURORA represents a critical enabler for the next generation of rare-event search experiments.

• Scientific Impact: It allows the PandaX-xT experiment to fully utilize its massive detector volume (40+ tonnes of xenon), significantly advancing the sensitivity frontier for dark matter detection and neutrinoless double-beta decay searches.
• Engineering Benchmark: It demonstrates that distributed, asynchronous architectures can successfully handle multi-GB/s data rates with zero data loss, a requirement for future large-scale particle physics experiments.
• Reusability: The framework provides a proven, high-performance reference design for the broader physics community, particularly for experiments requiring high channel counts and strict time-correlation between subsystems.