The High Level Trigger and Express Data Production at STAR

To support the BES-II program at RHIC, the STAR experiment implemented a complementary dual real-time framework combining a High Level Trigger for online event selection and an Express Data Production system for rapid, near offline-quality reconstruction, enabling prompt physics analysis and demonstrating scalability for future high-luminosity experiments.

The Gist

Imagine the STAR experiment at the Relativistic Heavy Ion Collider (RHIC) as a massive, high-speed photography studio. Their goal is to take pictures of the universe's most extreme moments: smashing heavy atoms together to recreate the "soup" of particles that existed just after the Big Bang.

The problem? They are taking billions of photos per second. If they tried to save every single photo to a hard drive and look at them later, they would run out of storage space instantly, and the scientists would have to wait years to see if they captured anything interesting.

To solve this, the STAR team built a two-part "Smart Camera System" called the High Level Trigger (HLT) and the Express Data Production (xProduction). Here is how it works, using simple analogies:

1. The High Level Trigger (HLT): The "Super-Fast Editor"

Think of the HLT as a team of incredibly fast, super-smart editors sitting right next to the camera.

• The Problem: The camera (the detector) captures a chaotic blur of millions of particles every time two atoms collide. Most of these are boring background noise.
• The Solution: The HLT looks at the raw data instantly (in real-time). It uses advanced algorithms (like a "Cellular Automaton," which is like a digital game of "connect the dots") to quickly trace the paths of particles.
• The Decision: Within milliseconds, the HLT decides: "Is this a boring collision? Delete it. Is this a rare, exotic event (like a hypernucleus)? Keep it!"
• The Result: It filters out 99% of the junk, keeping only the "golden tickets." It also acts as a quality control inspector, checking if the camera is focused correctly while the show is happening.

Analogy: Imagine a concert where 10,000 people are shouting. The HLT is a sound engineer who instantly silences the crowd noise and only records the singer's voice if they hit a perfect high note.

2. The Express Data Production (xProduction): The "Instant Developer"

Once the HLT saves the "golden tickets," they are still in a raw, messy format (like a negative film). Usually, scientists would have to wait months to "develop" these photos in a darkroom (offline processing) to see the final picture.

The xProduction system is like a magic instant-developer.

• How it works: While the HLT is still busy filtering the next batch of collisions, the xProduction system runs in parallel on the same computer cluster. It takes the filtered "golden tickets" and processes them immediately.
• The Speed: Instead of waiting months, it produces high-quality, analysis-ready results in hours.
• The Output: It creates compact, easy-to-read files (called "picoDsts") that are small enough to share instantly but contain all the important physics details.

Analogy: If the HLT is the editor picking the best photos, xProduction is the photo booth that prints a high-quality, framed picture of those best photos while you are still standing in line, so you can show your friends immediately.

Why Does This Matter? (The "Hypernucleus" Story)

The real magic of this system is that it allows scientists to find rare, exotic particles called hypernuclei.

• The Challenge: Hypernuclei are like "ghosts" in the data. They are extremely rare and decay almost instantly. In the past, scientists might have collected data for a year, processed it, and then realized, "Oh no, we missed the rare events because our filters were too strict," or "We have to wait two years to confirm if we found them."
• The Breakthrough: With this new system, during the 2021 run, the team was able to spot a specific type of hypernucleus (called $^{5}_{\Lambda}\text{He}$) with incredible statistical confidence while the experiment was still running.
• The Impact: They didn't just find it; they could analyze its properties (like its mass and how it decays) almost immediately. This is like finding a needle in a haystack, measuring the needle's weight, and telling the world about it before you've even finished sorting the hay.

Summary: The "Dual-Engine" Approach

The paper describes a brilliant dual-engine architecture:

1. Engine A (HLT): The Gatekeeper. It runs fast, filters the noise, and ensures the data stream is clean. It tells the scientists, "The camera is working, and here are the interesting events."
2. Engine B (xProduction): The Analyst. It takes those interesting events and turns them into scientific results immediately, without waiting for the slow, traditional offline process.

The Bottom Line:
This system bridges the gap between "taking the photo" and "understanding the photo." It allows the STAR experiment to handle massive amounts of data, spot rare cosmic mysteries in real-time, and give physicists the feedback they need to adjust their experiments on the fly. It's the difference between waiting years for a movie review and getting a live, high-definition critique the moment the movie ends.

Technical Summary

1. Problem Statement

The STAR experiment at the Relativistic Heavy Ion Collider (RHIC) faced significant challenges during the Beam Energy Scan Phase-II (BES-II) program. The primary issues were:

• Data Volume and Latency: The upgrade to the detector and readout systems increased the data rate (up to ~4 kHz for minimum-bias Au+Au collisions). The traditional offline workflow (archiving raw data to High Performance Storage System (HPSS) followed by reconstruction months or years later) introduced unacceptable latency for real-time operational feedback and rapid physics analysis.
• Rare Signal Detection: The physics goals included studying rare signals like hypernuclei and hyperons. Waiting for offline processing delayed the ability to verify data quality or adjust trigger strategies for these rare events.
• Resource Constraints: Existing "FastOffline" solutions were limited to small fractions of data (~5%) and lacked the flexibility for rapid, physics-level reconstruction (e.g., hypernuclei yield checks) due to strict software controls and shared resource limitations.

2. Methodology

To address these challenges, STAR developed a dual real-time framework consisting of two complementary systems running on a dedicated multi-core CPU cluster (27 nodes, ~1,200 cores, with optional Xeon Phi coprocessors):

A. High Level Trigger (HLT)

• Role: Operates online within the Data Acquisition (DAQ) chain as the final software stage (Level-4).
• Function: Performs rapid tracking, vertexing, and event filtering in real-time.
• Algorithms:
  • Cellular Automaton (CA) Track Finder: Optimized for the TPC's 24-sector geometry to reconstruct charged particle trajectories with high efficiency (>95%) and low latency.
  • Kalman Filter (KF) Particle Finder: Used for reconstructing short-lived particles (hyperons, hypernuclei) and secondary vertices.
  • Multi-Vertex Finder: Specifically designed to handle pile-up collisions in fixed-target mode by clustering tracks into primary, pile-up, and beam-pipe interaction vertices.
• Calibration: Utilizes a rolling update system where calibration constants (space-charge, alignment, gain) are updated every ~15 minutes or every $10^2-10^3$ events based on live data summaries.
• Offloading: Compute-intensive tasks (like KF Particle reconstruction) are offloaded to Xeon Phi coprocessors to maintain throughput.

B. Express Data Production (xProduction)

• Role: Runs concurrently and independently of the DAQ loop on the same HLT cluster.
• Function: Processes HLT-selected events (stored in DAQ format on distributed Ceph storage) to produce "picoDst" datasets (compact ROOT trees) within hours of data collection.
• Workflow:
  • xCalibration: Generates offline-quality calibration snapshots from recent data.
  • xProduction: Reconstructs events using the same algorithms as offline but with near-offline quality, producing analysis-ready datasets.
  • xPhysics: Performs real-time validation of signals (e.g., hypernuclei yields) using the reconstructed data.
• Data Reduction: Reduces data size by two orders of magnitude (from raw MBs to compact picoDsts) while preserving essential physics content.

3. Key Contributions

• Integrated Real-Time Architecture: Successfully implemented a system where HLT handles online event selection and xProduction delivers high-quality, analysis-ready data in hours rather than months.
• Advanced Reconstruction Algorithms in Real-Time: Adapted complex offline algorithms (CA Track Finder, KF Particle Finder, Multi-Vertex reconstruction) to run efficiently on a CPU/GPU hybrid farm with strict latency constraints.
• Dynamic Calibration: Developed a robust auto-calibration service that maintains tracking accuracy (e.g., keeping $\langle DCA_{xy} \rangle$ within ~0.01 cm of zero) despite fluctuating beam conditions and space-charge effects.
• Specialized Triggers: Introduced physics-driven triggers (e.g., "Helium Trigger" for hypernuclei) that enrich the express data stream with rare events, increasing the yield of specific particles by 30–64% compared to unbiased samples.

4. Results

The system was validated during the 2018–2021 runs, particularly in the 2021 fixed-target run at $\sqrt{s_{NN}} = 3.0$ GeV:

• Throughput: The HLT sustained processing rates of ~2 kHz for minimum-bias Au+Au collisions at 200 GeV, matching DAQ delivery rates.
• Reconstruction Quality:
  • Hyperons: Reconstructed various hyperons ($\Lambda, \Xi, \Omega$) and mesons ($K^0_S, \pi^0$) with high significance. For example, $\pi^0$ was observed with a significance of 48$\sigma$ in the express stream.
  • Hypernuclei: Achieved the first observation of the $^5_\Lambda$He hypernucleus in heavy-ion collisions with a statistical significance of 11.6$\sigma$ using the express data stream.
  • Comparison: The mass peak positions, widths ($\sigma$), and signal-to-background ratios (S/B) in the express data were nearly identical to those in the standard offline production (differences < 1%), confirming the reliability of the online calibration and reconstruction.
• Efficiency: The "Helium Trigger" enriched the express sample, containing roughly 30% more $^4_\Lambda$He and $^5_\Lambda$He, and 64% more $^4_\Lambda$H per event compared to the full dataset.
• Dalitz Plots: Successfully generated Dalitz plots for three-body decays (e.g., $^4_\Lambda$He $\to$ $^3$He + p + $\pi^-$), revealing complex structures potentially indicative of spin effects and nuclear resonances.

5. Significance

• Operational Paradigm Shift: This framework bridges the gap between immediate operational needs (beam tuning, detector QA) and long-term offline physics analysis. It allows for "prompt reconstruction," enabling physicists to verify rare signals and adjust strategies while the experiment is still running.
• Scalability and Robustness: The demonstrated ability to process hundreds of millions of events with near-offline quality establishes a model for future high-luminosity experiments (e.g., FAIR/CBM) that require both online filtering and rapid data access.
• Physics Discovery: The system enabled the timely discovery and characterization of rare exotic states (hypernuclei) that would have been delayed significantly by traditional offline workflows, accelerating the understanding of the QCD phase diagram and the properties of dense nuclear matter.
• Resource Efficiency: By utilizing idle compute resources on the HLT farm and offloading specific tasks to coprocessors, the system maximizes hardware utilization without interfering with the primary DAQ functions.