Coupling a Recurrent Neural Network to SPAD TCSPC Systems for Real-time Fluorescence Lifetime Imaging

Technical Summary: Coupling a Recurrent Neural Network to SPAD TCSPC Systems for Real-Time Fluorescence Lifetime Imaging

Problem Statement
Fluorescence Lifetime Imaging (FLI) is a critical diagnostic tool in biological and medical research, offering advantages over intensity imaging such as insensitivity to excitation fluctuations and the ability to measure microenvironmental parameters. However, existing FLI systems, particularly those based on Time-Correlated Single-Photon Counting (TCSPC), face a trade-off between processing speed, accuracy, and robustness. Traditional TCSPC systems require the accumulation of photon arrival timestamps into histograms before lifetime estimation can occur. This process generates massive data volumes, necessitating powerful PCs (often with GPUs), creating bottlenecks in data transfer and storage, and hindering miniaturization for clinical applications like non-invasive monitoring. Furthermore, common fast estimation methods like the Center-of-Mass Method (CMM) are highly sensitive to background noise, while accurate iterative methods like Least-Squares (LS) fitting are computationally expensive and slow.

Methodology
The authors propose a paradigm shift toward Edge Artificial Intelligence (Edge AI) by coupling a Single-Photon Avalanche Diode (SPAD) TCSPC system directly with a Recurrent Neural Network (RNN). The core innovation is the elimination of the histogramming step; instead, raw photon timestamps are fed directly into the neural network for immediate processing.

• System Architecture: The system utilizes a custom 32×32 SPAD sensor (Piccolo) with on-chip Time-to-Digital Converters (TDCs) providing 50 ps temporal resolution. This sensor is coupled to a Xilinx Kintex-7 FPGA.
• Neural Network Models: The study evaluates three RNN variants: Simple RNN, Gated Recurrent Unit (GRU), and Long Short-Term Memory (LSTM). These models are trained on synthetic datasets generated to simulate fluorescence decay, instrument response functions (IRF), background noise, and dark counts. The training data covers lifetimes from 0.2 to 5 ns with laser repetition frequencies of 20 MHz.
• Training Strategy: Unlike previous neural network approaches that use histograms as input, this method processes timestamps sequentially. A weighted Mean Square Percentage Error (MSPE) loss function is employed, assigning higher importance to later timesteps to reflect the accumulation of information as more photons arrive.
• Hardware Implementation: To enable real-time operation, the trained models are quantized (weights to 16-bit fixed-point, activations to 16-bit) and implemented on the FPGA. The design features four parallel GRU cores, each managing a quarter of the sensor array, allowing for event-driven, near-sensor processing.

Key Contributions

1. Direct Timestamp Processing: The paper introduces a method to estimate fluorescence lifetimes directly from raw timestamps using RNNs, bypassing the need for histogram construction. This drastically reduces data transfer volumes (from ~80 Mb/s in raw mode to 240 kb/s in RNN mode) and hardware resource utilization.
2. Robustness to Noise: The study demonstrates that RNNs, particularly GRU and LSTM variants, maintain high accuracy even in the presence of background noise (up to 5%), a scenario where traditional CMM fails significantly.
3. Theoretical Validation: The authors derive the Cramer-Rao Lower Bound (CRLB) for the measurement, showing that the RNN-based estimators achieve near-optimal precision, comparable to the theoretical limit and superior to LS fitting in terms of photon efficiency.
4. Real-Time Hardware Deployment: The successful deployment of quantized GRU cores on an FPGA demonstrates the feasibility of video-rate FLI (up to 10 frames per second) on a miniaturized, integrated system.

Results

• Accuracy: On synthetic datasets without background noise, GRU-32 and LSTM-32 achieved Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) comparable to CMM and LS fitting.
• Noise Resilience: In the presence of 1% and 5% background noise, the LSTM-32 model significantly outperformed both CMM and LS fitting. While CMM errors increased dramatically with noise, the RNN maintained low error rates, demonstrating superior robustness.
• Generalization: A model trained exclusively on synthetic data successfully estimated lifetimes on experimental data (fluorescent beads) acquired from a commercial confocal FLIM setup, validating the transferability of the approach to real-world scenarios.
• Performance Metrics: The FPGA implementation processes up to 4 million photons per second. The system achieved real-time FLI acquisition at 5–10 frames per second, capturing the movement of fluorescent beads with accurate lifetime mapping.

Significance and Claims
The paper claims that this approach offers a promising solution for biomedical applications requiring fast, accurate, and robust FLI, such as biological imaging, diagnostics, and fluorescence-assisted surgery. By removing the histogramming bottleneck, the system enables miniaturization and reduces the computational burden, making real-time FLI feasible on portable devices. The authors emphasize that the system's flexibility allows for retraining on different datasets for other applications, such as classification, and suggests that future iterations could integrate these RNN cores directly onto SPAD arrays via 3-D stacking for true in-sensor processing. The work establishes that neural networks can achieve near-optimal estimation precision while operating at speeds and with noise tolerance that traditional methods cannot match in a real-time, embedded context.