Technical Summary: DynGhost

Problem Statement

Ghost imaging is a computational technique that reconstructs spatial information from a single-pixel (bucket) detector by correlating structured illumination patterns with scalar intensity measurements. While deep learning has advanced static scene reconstruction, two critical limitations hinder the application of ghost imaging to dynamic, real-world scenarios:

1. Lack of Temporal Modeling: Existing architectures (including CNNs and static Transformers like Ghost-GPT) treat scenes as purely static. They fail to exploit temporal coherence across frames, resulting in severe motion artifacts or the necessity to discard moving frames.
2. Inaccurate Noise Modeling: Current deep learning approaches assume additive Gaussian noise. This contradicts the physical reality of single-photon quantum detectors (e.g., SNSPDs, SPADs, SiPMs), which operate under Poissonian statistics and exhibit specific artifacts like dark counts, afterpulsing, and crosstalk. This distribution shift causes models trained on Gaussian data to fail under realistic hardware constraints.

Methodology: DynGhost Architecture

The authors propose DynGhost (Dynamic Ghost Imaging Transformer), a framework designed to address both temporal and physical modeling gaps.

1. Architecture

DynGhost processes a sequence of bucket measurements $B \in \mathbb{R}^{T \times M}$ to output reconstructed frames $\hat{X} \in \mathbb{R}^{T \times H \times W}$.

• Token Embedding: Inputs are formed by concatenating learned projections of the illumination masks ($H_i$) and bucket measurements ($b_i$), augmented with learned spatial and temporal positional encodings.
• Alternating Attention Blocks: The core innovation is a sequence of $L$ transformer blocks that alternate between two modes:
  • Spatial Attention: Models informative patterns within a single frame ($O(M^2)$ complexity).
  • Temporal Attention: Propagates information across frames for a specific pattern to exploit motion coherence ($O(T^2)$ complexity).
• Output: A projection layer maps the final token representations to the spatial image domain.

2. Training Objective

The model is optimized using a composite loss function:
$$\mathcal{L} = \mathcal{L}_{MSE} + 0.5 \mathcal{L}_{SSIM} + 0.1 \mathcal{L}_{temp}$$

• $\mathcal{L}_{MSE}$: Pixel-wise reconstruction error.
• $\mathcal{L}_{SSIM}$: Perceptual structural similarity loss.
• $\mathcal{L}_{temp}$: A temporal consistency loss enforcing that the difference between consecutive reconstructed frames matches the difference in ground truth frames, ensuring smooth motion propagation.

3. Quantum-Aware Training Framework

To bridge the gap between simulation and hardware, the authors introduce a physically accurate detector simulation pipeline:

• Detector Simulation: The training data generation incorporates Poissonian statistics for photon counting, along with specific hardware artifacts (dark count rate, dead time, afterpulsing, crosstalk) for SNSPDs, SPADs, and SiPMs.
• Variance-Stabilizing Normalization: Recognizing that Poisson noise is heteroscedastic (variance scales with the mean), the authors apply the Anscombe transform ($f(n) = 2\sqrt{n + 3/8}$) or the Freeman–Tukey transform. These transforms convert Poisson-distributed counts into approximately unit-variance Gaussian variables, aligning the input distribution with the implicit assumptions of the MSE loss function.

Key Contributions

1. First Dynamic Ghost Imaging Transformer: DynGhost is the first architecture to explicitly exploit motion coherence via spatial-temporal attention and temporal consistency losses, yielding significantly smoother predictions for dynamic scenes.
2. Quantum Detector Evaluation & Distribution Shift Mitigation: The paper presents the first framework trained and evaluated under true quantum detector simulations. It identifies that Gaussian-trained models suffer catastrophic distribution shifts on real hardware and resolves this via correct noise modeling and variance-stabilization, achieving a +33.4% SSIM gain on real hardware simulations.
3. Operational Regime Characterization: Through benchmarking seven normalization strategies and various detector types, the authors define the specific operating regime where quantum models decisively outperform classical alternatives: $\ge$100 photons/measurement, Dark Count Rate (DCR) < 10,000 Hz, and efficiency > 60%.

Experimental Results

Experiments were conducted on three benchmarks: Moving MNIST, Moving CIFAR-10, and the Kvasir Endoscopy dataset (real-world infrared video).

• Performance vs. Baselines: DynGhost significantly outperforms classical methods (DGI, Pseudo-Inverse, FISTA) and contemporary deep learning baselines (U-Net, CNN, static Ghost-GPT).
  • On Moving MNIST, DynGhost achieved a 45% reduction in MSE and a 16% improvement in SSIM compared to the static Ghost-GPT.
  • It operates 130× faster than iterative solvers like FISTA.
• Dynamic Robustness: The model maintains high structural fidelity (SSIM > 0.90) for object velocities up to 10 px/frame. Performance degrades gracefully at extreme velocities (50 px/frame) due to intra-frame motion blur, a physical limitation rather than an algorithmic failure.
• Noise Robustness: DynGhost maintains acceptable reconstruction quality (SSIM > 0.5) down to 15 dB SNR, significantly outperforming classical solvers which fail catastrophically at low SNR.
• Ablation Studies:
  • Removing temporal attention blocks increased MSE by a factor of 2.3, confirming temporal coherence as the primary driver of performance.
  • Relying solely on MSE loss caused a severe collapse in SSIM (from 0.917 to 0.670), highlighting the necessity of perceptual and temporal losses.
• Normalization Impact: Variance-stabilizing transforms (Anscombe, Freeman–Tukey) drastically outperformed standard normalization (Min-max, Z-score) on single-photon data, improving SSIM from ~0.73 to ~0.84 on SNSPD simulations.

Significance and Claims

The paper claims that DynGhost provides a robust, principled blueprint for deploying high-speed ghost imaging in photon-starved, real-world environments. Its significance lies in:

1. Solving the Dynamic Bottleneck: By treating temporal coherence as an implicit reconstruction prior, it renders dynamic ghost imaging solvable without discarding moving frames.
2. Bridging the Physics Gap: It demonstrates that ignoring the Poissonian nature of single-photon detectors leads to suboptimal performance, and that variance-stabilizing transforms are essential for aligning deep learning models with physical hardware constraints.
3. Defining Hardware Requirements: The work clarifies that detector efficiency is less critical than Dark Count Rate (DCR) for reconstruction quality, providing specific thresholds for effective deployment.

Limitations Acknowledged:
The authors note that the evaluation relies heavily on synthetic datasets (Moving MNIST/CIFAR), which may not fully capture real-world complexity. The quadratic scaling of temporal attention limits the model to short sequences ($T=8$), and the normalization is calibrated for a fixed photon flux, potentially degrading under variable illumination. Validation on physical real-time hardware remains an open step.