An Angular-Temporal Interaction Network for Light Field Object Tracking in Low-Light Scenes

Technical Summary: An Angular-Temporal Interaction Network for Light Field Object Tracking in Low-Light Scenes

Problem Statement
Existing object tracking models primarily rely on RGB camera techniques, which capture 2D intensity information. In complex low-light scenes, intensity information is heavily attenuated, leading to degraded tracking performance due to a lack of discriminative visual cues. While light field cameras capture structured 4D data (spatial and angular information) that enhances object appearance and geometric structural descriptions even in low light, current methods struggle to effectively utilize this data. Specifically, high-dimensional light fields contain redundant appearance cues that tightly couple with sparse geometric structural cues. Existing representations, such as standard Epipolar Plane Images (EPIs), still retain this redundancy, limiting the model's ability to explore geometric structural cues in the angular domain. Furthermore, modeling temporal cues for light field tracking is challenging; standard spatio-temporal methods often aggregate information from non-geometric structures, weakening target-background distinction. Additionally, the scarcity of light field datasets makes purely supervised learning insufficient for effective temporal feature modeling.

Methodology
The paper proposes a framework centered on a novel representation and a specialized network architecture:

1. Light Field Epipolar-Plane Structure Image (ESI):
   The authors introduce the ESI representation to explicitly define the geometric structure within the light field. By capitalizing on the abrupt changes in light ray angles within the epipolar plane (specifically at object boundaries), the method identifies "geometric structure points." This is achieved by calculating the first-order derivatives (gradients) of horizontal and vertical EPIs and projecting the maximum gradient amplitudes onto a 2D plane. This process removes high-dimensional redundant appearance information, leaving a representation that clearly delineates target contours, particularly in low-light conditions.

2. Angular-Temporal Interaction Network (ATINet):
   ATINet is a hybrid architecture designed to learn angular-aware representations from the ESI. It employs a dual-stream feature extraction process:

   • Appearance Stream: Inputs ESI into a CNN backbone.
   • Angular-Temporal Stream: Inputs ESI into a Transformer-based framework enhanced by a Geometry Adaptive Selection (GAS) module.
     The baseline Transformer utilizes multi-head self-attention to model intra-frame and inter-frame correlations. The GAS module acts as a plug-and-play component that adaptively partitions ESI embeddings into geometric structure points (crucial for motion perception) and non-geometric points. It constructs a relation matrix to selectively aggregate information, ensuring that the network focuses on sparse geometric cues rather than redundant appearance data.

3. Self-Supervised Learning (SSL):
   To address the limited availability of light field datasets, the authors introduce a self-supervised loss function. This loss utilizes a masking strategy (inspired by DropMAE) where a decoder reconstructs masked embedding features. Crucially, the decoder employs adaptive spatial-attention dropout to constrain intra-frame interactions while encouraging inter-frame interactions. This forces the network to prioritize temporal motion cues and learn effective representations for temporal matching without requiring extensive labeled data.

4. Dataset Construction (R8LUT):
   The authors constructed the R8LUT dataset, a large-scale light field dataset for object tracking. Using a Raytrix R8 camera, they recorded 26 light field video sequences in low-light environments containing diverse objects (e.g., fish, toy cars, industrial nuts). The dataset includes 173 sequences for Single Object Tracking (SOT) and 26 sequences for Multiple Object Tracking (MOT), all annotated with bounding boxes and ESI representations.

Key Contributions

• Novel Representation: The proposal of the Light Field ESI, which explicitly delineates geometric structure points by exploiting abrupt light ray changes, effectively reducing redundancy and enhancing visual expression in low-light scenes.
• ATINet Architecture: The development of an Angular-Temporal Interaction Network that integrates a Geometry Adaptive Selection (GAS) module to learn discriminative features from geometric structural and angular-temporal cues.
• Self-Supervised Optimization: The introduction of a self-supervised loss to optimize angular-aware representations, facilitating effective feature learning in the temporal domain despite data scarcity.
• Benchmark Dataset: The creation of the R8LUT dataset, providing the first large-scale resource for light field SOT and MOT tasks in low-light scenarios.
• Extension to MOT: The extension of the proposed method to Multiple Object Tracking (AMTrack) by integrating GAS into a re-identification framework.

Results
Extensive experiments were conducted on the R8LUT dataset:

• Single Object Tracking (SOT): ATINet achieved state-of-the-art performance, outperforming 24 existing methods (including hybrid and pure Transformer trackers) in Success, Precision, and Normalized Precision metrics. Visual comparisons demonstrated superior ability to track deforming and similar objects in low-light conditions compared to RGB-based trackers.
• Multiple Object Tracking (MOT): The extended AMTrack method achieved the highest MOTA (87.5) and IDF1 (85.4) scores among compared state-of-the-art methods, demonstrating effective long-term trajectory maintenance and low miss detection rates.
• Ablation Studies: Results confirmed that the combination of ESI, GAS, and SSL yields optimal performance. Specifically, the ESI representation was shown to be superior to alternative gradient aggregations (mean, sum, max), and the GAS module was proven essential for modeling both intra-frame and inter-frame correlations.

Significance
The paper claims that its primary significance lies in addressing the specific challenges of light field object tracking in low-light environments, a scenario where traditional RGB methods fail. By decoupling geometric structural cues from redundant appearance information and modeling them through an angular-temporal interaction network, the proposed method enables more robust target discrimination. The introduction of the R8LUT dataset fills a critical gap in available resources for this domain. The authors assert that their approach validates the effectiveness of high-quality light field angular-temporal modeling, achieving state-of-the-art results on both SOT and MOT benchmarks while maintaining computational efficiency through the lightweight design of the GAS and SSL modules.