Enhancing Eye Feature Estimation from Event Data Streams through Adaptive Inference State Space Modeling

1. Problem Statement

Eye tracking using event-based cameras (Dynamic Vision Sensors or DVS) offers significant advantages over traditional frame-based cameras, including ultra-low latency, high temporal resolution (up to 10,000 Hz), and low power consumption. However, existing methods struggle with the highly variable kinematics of eye movements:

• Saccades: Rapid, ballistic movements generate dense bursts of events with a high Signal-to-Noise Ratio (SNR).
• Fixations: Periods of stillness or slow movement (e.g., smooth pursuit) generate very few events, resulting in low SNR and sparse data.

The Core Challenge: Current state-of-the-art models, particularly Recurrent Neural Networks (RNNs) and hybrid CNN-RNN architectures, rely heavily on historical context. When the eye transitions from a saccade to a fixation, the sudden drop in event density causes these models to degrade in performance. They either fail to adapt quickly enough to the sparse signal or introduce bias by relying on outdated historical context when the current signal is unreliable. Furthermore, increasing the temporal window to accumulate more events introduces unacceptable latency.

2. Methodology: Adaptive Inference State Space Model (AISSM)

The authors propose a novel Bayesian architecture called the Adaptive Inference State Space Model (AISSM). Unlike standard RNNs that treat past and present information uniformly, AISSM dynamically weights the contribution of the current observation versus historical context based on the reliability of the current signal.

Key Components:

1. Adaptive Inference Mechanism:

   • The model maintains a latent state representation using a discrete, categorical distribution rather than a single continuous vector.
   • It computes a Posterior ($q_\theta(s_t|o_t)$) representing the current event frame and a Prior ($p_\nu(s_t|h_t)$) representing the historical state derived from a recurrent transition model.
   • Instead of collapsing these distributions (as in Variational Inference), the model treats them as distinct sources of information: the posterior for the present and the prior for the past.

2. Dynamic Confidence Network (DCN):

   • A dedicated module ($f_\lambda$) estimates a confidence score ($\hat{\alpha}_t$) for the current observation.
   • This score is derived from two metrics calculated within a Region of Interest (ROI) around the eye feature:
     • Signal-to-Noise Ratio (SNR): The ratio of events inside the ROI to those outside.
     • Event Density (ED): The number of events normalized by the ROI area.
   • The DCN outputs a weight $\alpha \in [0, 1]$.

3. Adaptive State Fusion:

   • The final latent state ($\bar{s}_t$) is a linear combination of the posterior ($s^q_t$) and the prior ($s^p_t$):
     $$\bar{s}_t = \hat{\alpha}_t \cdot s^q_t + (1 - \hat{\alpha}_t) \cdot s^p_t$$
   • Logic: When event density/SNR is high (e.g., during a saccade), $\hat{\alpha}_t \to 1$, relying on current data. When density is low (e.g., during fixation), $\hat{\alpha}_t \to 0$, relying on the historical prior to maintain tracking stability.

4. Long-Horizon Training Technique:

   • To address the "stateless training" problem where RNN states are reset every batch (losing long-term dependencies), the authors propose storing the model's posterior, prior, and recurrent states directly within the dataset for each training instance.
   • During training, the model retrieves the specific state for the current instance, updates it, and saves it back, ensuring continuity across gaze behaviors without resetting the temporal context at every batch.

3. Key Contributions

• Novel Architecture (AISSM): A Bayesian State Space Model that dynamically adjusts the reliance on current vs. historical data based on signal quality, solving the latency-accuracy trade-off in variable kinematic scenarios.
• Dynamic Confidence Network: A learnable module that explicitly estimates signal reliability (SNR and density) to drive the adaptive weighting mechanism.
• Long-Horizon Training: A training strategy that preserves temporal continuity in RNN-based models by storing state information within the dataset, improving convergence and stability.
• SOTA Performance: The system achieves state-of-the-art results on event-based eye tracking benchmarks while maintaining a parameter count comparable to baselines (~500k parameters).

4. Experimental Results

The model was evaluated on the 3ET+ dataset (a large-scale, real-world event-based eye tracking benchmark) against several baselines, including CNN, CNN-GRU, CNN-Mamba, CB-ConvLSTM, and MambaPupil.

• Metrics: Performance was measured using P5, P10, and P15 (percentage of predictions within 5, 10, and 15 pixels of the ground truth) and normalized distance error.
• Quantitative Results:
  • AISSM (Ours): Achieved 46.85% P5, 72.38% P10, and 85.41% P15.
  • Best Baseline (CB-ConvLSTM): Achieved 29.86% P5, 57.40% P10, and 71.94% P15.
  • Error Distance: AISSM achieved the lowest normalized error (0.01) compared to 0.03 for most baselines.
• Qualitative Analysis:
  • Visualizations show that during transitions from saccade to fixation, CNN-GRU models suffer rapid performance drops as event density decreases.
  • In contrast, AISSM successfully increases its reliance on the historical prior as the current signal degrades, maintaining a high number of "hits" (accurate predictions) even during low-density fixation periods.
  • The Long-Horizon training technique was shown to stabilize accuracy over time, preventing the divergence seen in models trained without it.

5. Significance and Future Work

• Impact: This work bridges the gap between the high-speed capabilities of event cameras and the robustness required for real-world eye tracking. It demonstrates that adaptive inference, inspired by Kalman filtering principles but implemented via deep learning, is superior to static recurrent models for non-stationary biological signals.
• Efficiency: The approach allows for low-latency tracking without the computational overhead of large temporal windows or complex transformer architectures.
• Limitations & Future Directions:
  • Latency: The current implementation uses two networks (AISSM + DCN), which may introduce inference latency. Future work could explore observation uncertainty-aware confidence to merge these into a single network.
  • Representation: The use of categorical distributions for latent states can be computationally expensive. Future research might investigate disentangled representations to improve training and sampling efficiency.

In summary, the paper presents a robust solution for event-based eye tracking by introducing an adaptive mechanism that intelligently balances current sensory input with historical context, significantly outperforming existing methods in dynamic and sparse signal environments.