🤖 machine learning

SparsePixels: Efficient Convolution for Sparse Data on FPGAs

The paper introduces SparsePixels, an FPGA-based framework that achieves constant-latency inference for sparse data by selectively computing only on active pixels, demonstrating a 73-fold speedup over standard dense CNNs for neutrino interaction identification with minimal performance loss.

Original authors: Ho Fung Tsoi, Dylan Rankin, Vladimir Loncar, Philip Harris

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ho Fung Tsoi, Dylan Rankin, Vladimir Loncar, Philip Harris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Busy Bee" vs. The "Empty Field"

Imagine you are a security guard at a massive stadium (the FPGA chip) tasked with checking every single seat in the stands to see if anyone is holding a ticket (processing an image).

In a standard computer system, the guard has to walk down every single row, check every single seat, and ask, "Is there a person here?" even if 99% of the seats are empty. This is how standard AI (Convolutional Neural Networks) usually works: it scans the entire image, pixel by pixel, regardless of whether there is anything interesting in it.

In particle physics experiments (like those at CERN or neutrino detectors), the "images" are often huge grids of data, but the actual "action" (like a particle track) only happens in a tiny, scattered few spots. It's like looking for a single ant in a field the size of a football stadium. If your guard has to check every blade of grass, it takes a long time. In these experiments, decisions must be made in microseconds (millionths of a second). If the guard takes too long, the event is missed forever.

The Solution: The "Smart Scout" (SparsePixels)

The authors of this paper created a new system called SparsePixels. Instead of sending a guard to check every seat, they send a Smart Scout.

Here is how the Scout works:

The Quick Scan: The Scout first does a super-fast sweep of the stadium. They don't check every seat; they just look for movement or noise.
The Shortlist: As soon as they spot a seat with a person (an "active pixel"), they write down the location on a small list. They ignore the empty seats completely.
The Focused Work: The Scout then takes this short list of only the interesting seats and does all the detailed analysis on just those few spots.

Because the Scout only works on the few seats that actually have people in them, the job gets done incredibly fast.

The Magic Trick: Constant Speed

The most clever part of this system is that the speed is always the same, no matter how empty the stadium is.

Standard AI: If the stadium is 99% empty, the guard still walks the whole way. If it's 100% full, the guard walks the whole way. The time varies based on the crowd.
SparsePixels: The system has a rule: "We will only ever check a maximum of 20 seats."
- If the image has 5 interesting spots, the system checks 5 spots and pads the list with "dummy" spots to reach 20.
- If the image has 15 interesting spots, it checks 15 and pads to 20.
- If the image has 20 spots, it checks 20.

Because the system always does the exact same amount of work (checking up to 20 spots), the time it takes to finish is constant. It doesn't matter if the input is sparse or dense; the "initiation interval" (the time between processing one image and the next) never changes. This is crucial for the strict timing requirements of particle physics.

The Results: Speeding Up by 73 Times

The researchers tested this on an FPGA (a specialized computer chip used in these experiments) using real data from neutrino detectors.

The Old Way (Standard CNN): To process a single image of a neutrino interaction, the standard system took about 48.6 microseconds.
The New Way (SparsePixels): By only looking at less than 1% of the pixels (the active ones), the new system finished the same job in 0.665 microseconds.

That is a 73x speedup.

They also found that while the new system ignored 99% of the data, it didn't lose much "intelligence." It only lost a tiny bit of accuracy (less than 2%), which is a small price to pay to get a machine 73 times faster.

Summary Analogy

Think of it like reading a book to find a specific word.

Standard CNN: You read every single letter on every single page, even the blank margins and the spaces between words, just to be sure.
SparsePixels: You quickly skim the page, find the words that look like they might contain the target, and only read those specific words. You ignore the rest of the page entirely.

The paper proves that for "sparse" data (where most of the image is empty space), this "skim and focus" method allows computers to make decisions fast enough to keep up with the fastest experiments in the universe.

Technical Summary: SparsePixels: Efficient Convolution for Sparse Data on FPGAs

1. Problem Statement

Standard Convolutional Neural Networks (CNNs) deployed on Field-Programmable Gate Arrays (FPGAs) for particle physics experiments often suffer from high inference latency and long initiation intervals. This inefficiency arises because standard convolution requires densely processing every input pixel, regardless of whether the pixel contains semantic information. In many modern detector applications, such as muon tracking in gas chambers or neutrino identification in Liquid Argon Time Projection Chambers (LArTPCs), input data is spatially sparse. For instance, a muon track may occupy less than 0.3% of a 2D image, and neutrino tracks in LArTPCs can have as few as 0.01% nonzero pixels.

Current FPGA implementations, such as those using the hls4ml library, typically rely on streaming Input/Output (I/O) to manage resource constraints. While streaming reduces resource usage, it processes pixels sequentially, leading to latencies that scale linearly with image size (e.g., scanning thousands of empty pixels). This makes standard CNNs infeasible for real-time trigger systems in high-energy physics experiments, which require decision latencies in the microsecond or sub-microsecond range to handle collision rates of 40 MHz.

Existing sparse convolution methods for GPUs (e.g., MinkowskiEngine, SpConv) often allow the set of active pixels to "dilate" across layers, increasing computational cost, or they rely on irregular data flows that result in input-dependent runtimes. In systems with strict latency bounds, variable runtime is unacceptable.

2. Methodology: The SparsePixels Framework

The authors introduce SparsePixels, a framework designed to implement sparse convolution on FPGAs that maintains a constant, input-independent runtime. The core philosophy is to selectively retain and compute only on a small, pre-specified subset of active pixels while ignoring the rest, ensuring the total operation count remains fixed for every input.

2.1 Core Architecture

The framework consists of five specialized sparse layers designed for High-Level Synthesis (HLS):

Sparse Input Reduction: The first layer identifies active pixels (those exceeding a threshold) and extracts them into compact arrays. To avoid data-dependent write addresses (which cause scheduling issues in HLS), the authors employ a recursive binary reduction tree. This hardware-efficient scheme scans the input in a fixed pattern to populate two arrays: a feature array (afeat) and a coordinate hash array (ahash). The number of retained pixels is capped at a hyperparameter, $N_{max}^{active}$ .
Sparse Convolution: This layer performs sparsity-preserving convolution. Crucially, an output pixel is computed only if the corresponding input pixel at the same location is active. This prevents the "dilation" of the active set seen in other sparse methods. The algorithm iterates over the fixed-size sparse arrays rather than the full image grid. The iteration count depends on $N_{max}^{active}$ , not the kernel size $K$ , as kernel positions are determined via offset checks rather than explicit loops over all $K^2$ positions.
Sparse Activation: Applies element-wise nonlinearity to the sparse feature array.
Sparse Pooling: Downsamples the sparse representation by mapping coordinates and accumulating features for pixels landing in the same pool.
Sparse Flattening: Converts the final sparse representation back into a dense array for connection to standard dense layers (MLP).

2.2 Implementation Details

Constant Runtime: Because the system operates on arrays of fixed size ( $N_{max}^{active}$ ), padded with invalid flags if the input has fewer active pixels, the inference latency is independent of the actual input sparsity.
Training: A Python library was developed to train these sparse CNNs using Quantization-Aware Training (QAT) with QKeras as the backend. The training mimics the HLS behavior, including the fixed $N_{max}^{active}$ constraint.
Hardware Target: The design targets AMD/Xilinx Alveo FPGAs (xcu250) using Vitis HLS, operating at 200 MHz.

3. Key Contributions

Sparse Convolution Method: A novel approach for FPGAs that adaptively selects active pixels but enforces a fixed computation budget ( $N_{max}^{active}$ ), ensuring constant latency regardless of input sparsity.
Proof-of-Concept Implementation: A demonstration of significant FPGA inference speedups on spatially sparse image data with comparable model performance to standard CNNs.
Open-Source Tooling: A publicly released Python library for training sparse CNNs and C++ HLS modules for deployment, facilitating adoption in the particle physics community.

4. Experimental Results

The authors evaluated SparsePixels on three datasets: a sparsified MNIST variant, MicroBooNE neutrino detection, and LHC jet flavor tagging. Models were compact (~4k parameters) and compared against standard CNN baselines implemented with stream I/O.

Performance vs. Latency: On the MicroBooNE dataset (63×63 images, ~4k pixels), a standard 8-bit CNN required 48.665 µs (9,733 clock cycles) for inference. The SparsePixels "sparse-large" variant (computing on only 20 active pixels, <1% of the input) achieved an inference time of 0.665 µs (133 clock cycles). This represents a 73× speedup with a negligible performance loss (<2% drop in ROC AUC).
Scalability: Similar speedups were observed for MNIST (up to 41×) and LHC jet tagging (up to 112×).
Resource Utilization: While sparse CNNs utilize more resources (due to loop unrolling for parallelism) than streaming standard CNNs, the resource usage remains well within on-chip budgets (e.g., ~20% of LUTs for the largest models).
Initiation Interval (II): The II for sparse CNNs is determined solely by the sparse input reduction layer (the first layer), as subsequent layers operate in parallel on the fixed-size sparse arrays.

5. Significance and Limitations

Significance:
The paper claims that SparsePixels enables the deployment of CNNs in ultra-low-latency regimes (sub-microsecond) required for modern particle physics triggers, where standard CNNs are currently infeasible due to latency. By trading a small, controllable amount of accuracy for massive speedups, the framework allows for more sophisticated online algorithms in experiments like the High-Luminosity LHC.

Limitations and Future Work:

Parallelization Constraints: The current implementation fully unrolls loops to minimize latency, which limits the maximum input size and model complexity due to FPGA resource exhaustion. Future work aims to introduce flexible control over parallelization to trade latency for resource efficiency.
Hyperparameter Tuning: The choice of $N_{max}^{active}$ is currently empirical. The authors propose making this parameter (and potentially the threshold) trainable in future versions.
Quantization: The current backend uses uniform per-layer quantization. Future upgrades plan to adopt High-Granularity Quantization (HGQ) for finer control.

The authors conclude that SparsePixels offers a viable path for efficient data readout in experiments with microsecond-level latency requirements, bridging the gap between the need for complex ML models and the strict timing constraints of real-time hardware triggers.