Large-scale real-time signal processing in physics experiments: The ALICE TPC FPGA pipeline

To support the ALICE Time Projection Chamber's continuous readout mode during LHC Run 3, the authors present a custom, large-scale FPGA-based pipeline that performs real-time signal processing—including common-mode correction, pedestal subtraction, and ion-tail filtering—to reduce raw data rates from over 3 TB/s to approximately 900 GB/s under extreme high-occupancy conditions.

The Gist

Imagine the Large Hadron Collider (LHC) as the world's most powerful particle smasher. Inside it, the ALICE experiment is like a giant, high-speed camera trying to take a picture of what happens when heavy lead atoms crash into each other.

In the past, this camera could only take a few snapshots a second. But for the latest run (Run 3), the scientists wanted to take 50,000 pictures every second. That's like trying to film a high-speed car race in 4K resolution, but with a camera that never stops recording, even when the cars aren't there.

The problem? The amount of data this camera generates is astronomical—about 3 Terabytes every second. That's roughly equivalent to downloading the entire internet's worth of data in a single second. No computer hard drive can store that much, and no standard processor can sort through it fast enough.

This paper describes the "brain" built to solve this problem: a massive, custom-made system of FPGAs (Field-Programmable Gate Arrays). Think of an FPGA not as a standard computer chip, but as a Lego set of logic circuits that scientists can build and rebuild instantly to do exactly what they need.

Here is how this "digital brain" works, broken down into simple steps:

1. The Raw Data Flood (The "Unfiltered Stream")

The detector (called a Time Projection Chamber or TPC) is a giant cylinder filled with gas. When particles fly through, they leave trails of electricity on thousands of tiny pads (like a giant grid of piano keys).

• The Problem: These pads pick up not just the particle signals, but also "static" (noise) and "echoes" (ions drifting slowly). It's like trying to hear a whisper in a crowded stadium while someone is blowing a trumpet nearby.
• The Solution: The system doesn't try to save everything. Instead, it processes the data in real-time, as it flows in, like a super-fast water filter.

2. The Three-Stage Filter (The "Cleaning Crew")

The FPGA pipeline acts like a highly efficient assembly line with three main stations:

• Station A: The "Common-Mode" Cleaner (The Crowd Control)

  • The Analogy: Imagine a room full of people talking. If everyone suddenly starts shouting at once, it creates a loud "hum" that drowns out individual voices. In the detector, electrical noise often affects all pads at once, creating a "hum" that shifts the baseline.
  • The Fix: The FPGA looks at the pads that should be silent (empty pads). It calculates the average "hum" and subtracts it from everyone's signal. It's like a sound engineer using a noise-canceling headset to remove the background drone so you can hear the music clearly.

• Station B: The "Pedestal" and "Tail" Cutter (The Echo Remover)

  • The Analogy: Every electronic device has a tiny "zero point" offset (a pedestal), and sometimes signals leave a long, fading tail (like a reverb in a cathedral).
  • The Fix: The system subtracts the built-in offset and uses a mathematical trick (a digital filter) to chop off those long, lingering tails. This prevents old signals from messing up new ones.

• Station C: The "Zero Suppressor" (The Trash Collector)

  • The Analogy: If you are filming a race, you don't need to save the frames where the track is empty.
  • The Fix: If a signal is too small (just noise), the FPGA throws it away immediately. It only keeps the "interesting" data. This cuts the data volume by more than half, turning 3 Terabytes down to about 900 Gigabytes per second.

3. The Packing Crew (The "Packing Tetris")

Once the data is cleaned and filtered, it needs to be packed up to be sent to the next stage of computers.

• The Analogy: Imagine you have a pile of loose bricks (data bits). You need to pack them into boxes (data packets) to ship them.
• The Fix: The FPGA is incredibly efficient at "Tetris." It shoves the remaining data bits into tight boxes with no wasted space, adding only a tiny label on the outside. This ensures the data fits perfectly into the memory of the next computers (GPUs) for final analysis.

4. The "Resync" Safety Net

Sometimes, the intense radiation in the collider can cause a tiny glitch in the connection, like a static crackle in a phone call.

• The Analogy: If a video stream freezes, you don't want to restart the whole movie; you just want to re-sync the audio and video.
• The Fix: The system has a built-in "Resync Controller." If a connection drops, it sends a quick "reset" signal, waits a fraction of a second, and then re-aligns the data streams. It's so fast that the scientists barely notice it happened.

Why This Matters

Before this upgrade, ALICE had to wait for the LHC to slow down to take data. Now, with this "FPGA Pipeline," the experiment can run at full speed, capturing 50,000 collisions per second.

This isn't just about saving space; it's about survival. Without this real-time processing, the data would be a chaotic, unmanageable mess. The FPGA system acts as a brilliant, tireless editor, cleaning, filtering, and packing the data instantly, allowing physicists to finally see the rare, beautiful physics hidden inside the chaos of the collisions.

In short: They built a super-fast, custom Lego-brain that sits between the detector and the computer, cleaning the noise and throwing away the junk in real-time, so the scientists can actually see the universe's secrets.

Technical Summary

Here is a detailed technical summary of the paper "Large-scale real-time signal processing in physics experiments: The ALICE TPC FPGA pipeline."

1. Problem Statement

The ALICE experiment at the CERN Large Hadron Collider (LHC) underwent a major upgrade for Run 3 (2022–2025) to accommodate significantly higher interaction rates, specifically up to 50 kHz in Lead-Lead (Pb–Pb) collisions. This upgrade replaced the Time Projection Chamber (TPC) readout from Multi-Wire Proportional Chambers (MWPCs) to Gas Electron Multiplier (GEM) technology, enabling continuous readout mode.

This shift introduced extreme data processing challenges:

• Data Volume: The raw data rate from the 524,160 anode pads is approximately 3.3 TB/s.
• Latency Constraints: The system must process data in real-time without buffering delays that would exceed the LHC orbit timing (89.1 µs).
• Signal Integrity: High occupancy rates cause two major detector effects that degrade data quality:
  1. Common-Mode (CM) Effect: Baseline shifts shared across groups of channels due to capacitive coupling between the pad plane and GEM foils.
  2. Ion-Tail Effect: Slowly drifting positive ions in the induction gap create exponential tails in pulse shapes, distorting signals and increasing noise.
• Resource Constraints: The processing must occur on hardware (FPGAs) with limited logic resources while handling 1,600 channels simultaneously at a 5 MHz sampling rate.

2. Methodology

The solution involves a custom, highly parallel Field-Programmable Gate Array (FPGA) pipeline implemented on Common Readout Units (CRUs). These units are based on Intel Arria 10 FPGAs and sit between the Front-End Electronics (FEE) and the Event-Processing Nodes (EPNs).

Architecture Overview

• Input: Raw data is streamed from the FEE via 6,552 optical links (GBT links) at 4.48 Gbit/s per link.
• Processing Pipeline: The User Logic (UL) on the FPGA performs a continuous, stream-based processing chain consisting of three stages:
  1. Input Stage: Decodes GBT frames, synchronizes data streams, and aligns time-bins.
  2. Processing Stage: Applies physics-specific corrections (CM correction, pedestal subtraction, ion-tail filtering).
  3. Output Stage: Performs zero suppression and dense data packing for transmission to the EPNs.

Key Algorithms

1. Common-Mode Correction (Spatial Filter):

   • Goal: Remove baseline shifts shared by pads under the same GEM stack.
   • Method: Identifies "empty pads" (channels with no physical signal) in each time-bin. It calculates a global CM value by averaging these empty pads, scaled by a per-pad factor ($k_{pad}$) to account for local capacitive coupling variations.
   • Implementation: A highly parallel algorithm using 400 parallel comparators to match samples against random reference pads. It operates on fixed-point arithmetic to maximize speed and minimize resource usage.

2. Pedestal Subtraction & Zero Suppression:

   • Subtracts the intrinsic electronic offset (pedestal) from each channel.
   • Suppresses samples below a defined threshold to reduce data volume before transmission.

3. Ion-Tail Filter (Temporal Filter):

   • Goal: Remove the exponential tail caused by drifting ions.
   • Method: Implements a recursive Infinite Impulse Response (IIR) filter.
   • Implementation: Uses 40 parallel filter cores. To maintain numerical precision, samples are converted to 32-bit floating-point for calculation and then converted back to 12-bit fixed-point. The filter uses dual-port memory to store correction factors ($q_{cor}$) for iterative updates.

4. Integrated Digital Currents (IDC):

   • Calculates cumulative sums of raw ADC samples over time windows to monitor space-charge distortions, enabling offline calibration corrections.

5. Auxiliary Control:

   • The FPGA also controls laser alignment systems, calibration pulsers, and reads out High Voltage Current Monitors (HVCM), all synchronized with the detector sampling clock.

Data Handling & Synchronization

• Data Format: The pipeline uses a compact "link data stream" format where each clock cycle carries sample values, channel IDs, and control flags (e.g., Zero, StreamActive, Rejected).
• Resynchronization: To mitigate radiation-induced link losses (observed at 50 kHz), the system performs periodic re-synchronization (every 10 Time Frames) using a "Pattern Player" to reset ADC clocks and realign data streams without interrupting data taking.
• Output Packing: Data is packed into 8 KiB packets with a Raw Data Header (RDH), utilizing a two-level bitmask scheme to efficiently represent sparse data.

3. Key Contributions

• First Large-Scale Continuous Readout: Demonstrates the world's largest FPGA-based real-time processing pipeline for a particle detector, handling 3.3 TB/s input and reducing it to <900 GB/s output.
• Novel Common-Mode Algorithm: Introduces a robust, pad-dependent common-mode correction algorithm that operates directly on streaming data, capable of handling extreme high-occupancy conditions where traditional methods fail.
• Hybrid Precision Processing: Successfully implements a mixed-precision approach (fixed-point for logic, floating-point for ion-tail filtering) within a single FPGA architecture to balance resource usage and numerical accuracy.
• Radiation Hardening Strategy: Developed a mitigation strategy for radiation-induced link losses via deterministic, periodic re-synchronization, ensuring stable operation in a high-radiation environment without requiring hardware replacement.
• Integrated Control: Consolidates detector readout, signal processing, and auxiliary system control (lasers, pulsers, HV monitoring) into a single FPGA firmware stack, reducing system complexity.

4. Results

• Data Reduction: The pipeline successfully reduces the raw data rate from 3.3 TB/s to approximately 900 GB/s for Pb–Pb collisions at 50 kHz, meeting the bandwidth constraints of the downstream EPN farm.
• Signal Quality:
  • Common-Mode Correction: Validation data shows the effective removal of baseline shifts. Without correction, negative outliers of several tens of ADC counts were observed; with correction, the baseline remains stable near zero.
  • Ion-Tail Filtering: Results in a ~1% reduction in data size by suppressing residual signal tails, consistent with theoretical expectations.
• Stability: The system has operated stably since the start of Run 3. Despite radiation-induced link losses, the periodic re-synchronization mechanism ensures that data acceptance loss is negligible (<1% per hour), maintaining physics data quality.
• Resource Utilization: The design utilizes approximately 71% of the available Adaptive Logic Modules (ALMs) and 52% of the DSP blocks on the Intel Arria 10 FPGA, demonstrating efficient use of hardware resources for such a complex task.

5. Significance

This work represents a paradigm shift in high-energy physics data acquisition. It moves the boundary of real-time processing from simple data formatting to physics-aware online conditioning.

• Scalability: The architecture proves that FPGAs can handle the massive bandwidths required for future high-luminosity collider runs (Run 4 and beyond).
• Physics Impact: By enabling continuous readout at 50 kHz, the ALICE TPC can capture rare physics events and study heavy-ion collisions with unprecedented statistics and granularity.
• Technological Benchmark: The successful implementation of complex, multi-channel, real-time algorithms (like the common-mode correction) sets a new standard for detector readout systems, influencing the design of future experiments at the LHC and other facilities.
• Robustness: The ability to maintain high-fidelity data acquisition in a harsh radiation environment through software-defined mitigation strategies highlights the maturity of FPGA-based detector control systems.