Learn Temporal Consistency For Robust Satellite Video Detector

Technical Summary: Learn Temporal Consistency For Robust Satellite Video Detector

Problem Statement
Satellite Video Object Detection (SVOD) is critical for applications ranging from disaster response to smart cities. However, existing SVOD methods face three primary limitations:

1. Scope Limitation: Most approaches focus exclusively on moving objects (Moving Object Detection), failing to capture the complete patterns of static or intermittently moving objects.
2. Granularity and Orientation: Current methods often rely on coarse-grained categories and Horizontal Bounding Boxes (HBBs). This is insufficient for satellite videos where objects are inherently arbitrarily oriented and require fine-grained recognition (e.g., distinguishing between different aircraft types).
3. Temporal Discontinuity: While image-based oriented object detectors exist, they treat satellite videos as disjointed frames, ignoring rich temporal contexts. This leads to fragmented object descriptions and timing inconsistencies. Conversely, existing video detectors are typically designed for general videos with HBBs and struggle with the significant scale variations and arbitrary orientations found in satellite imagery.

Methodology: The TCL Framework
The authors propose Temporal Consistency Learning (TCL), a novel video object detection framework designed specifically for oriented and fine-grained objects in satellite videos. The framework integrates three core modules:

1. Temporal and Fine-Grained Feature Aggregation (TFA):

   • Goal: To enhance fine-grained features by leveraging information from adjacent frames.
   • Mechanism: The Oriented Region of Interest (ORoI) features of the current frame and neighboring frames are partitioned into $N_b$ local blocks (e.g., separating an aircraft into wings, body, and tail). The module models relations between corresponding blocks across frames using a weighted attention mechanism. This aggregates temporal information to produce enhanced, consistent fine-grained features ($\Omega^{t+}$).

2. Structure Encoding (SE):

   • Goal: To improve fine-grained recognition by incorporating structural information alongside appearance.
   • Mechanism: The prediction head receives enhanced object features. After processing through fully connected layers, the module explicitly stacks structural information—specifically width ($w$), height ($h$), and aspect ratio ($w/h$)—with the feature vector. This combined representation is used to predict both the object's location (oriented bounding box parameters: $x, y, \theta, w, h$) and its fine-grained category.

3. Temporal Consistency Constraint (TCC):

   • Goal: To enforce feature consistency for the same object across consecutive frames, reducing noise and environmental variation impacts.
   • Mechanism: A loss function is formulated to minimize the Euclidean distance between features of the same object across frames while maximizing the distance between different objects.
   • Components:
     • $L_{fine}$: Constrains features based on fine-grained labels (same label $\to$ distance 1; different label $\to$ distance 0).
     • $L_{obj}$: Extends the constraint to distinguish between foreground and background features, preventing the model from overlooking objects with weak appearances.
   • Dynamic Thresholds: The loss utilizes dynamic thresholds ($\delta$) that adapt to the feature distribution of specific categories.

Key Contributions

1. Novel Framework: TCL is presented as the first video object detection framework tailored specifically for oriented and fine-grained objects in satellite videos, capable of mining temporal contexts.
2. Temporal Consistency Strategies: The introduction of TFA and TCC strategies allows the model to capture complete and consistent patterns of objects throughout the entire video sequence, addressing the limitations of frame-by-frame processing.
3. Structure Information Utilization: The SE module is designed to encode both appearance and structural information (dimensions and aspect ratios), enhancing precision in fine-grained recognition where appearance alone may be distorted or weak.
4. Benchmark Leadership: The framework demonstrates superior performance on the SAT-MTB benchmark, the largest satellite video dataset, setting a new state-of-the-art.

Experimental Results
Experiments were conducted on the SAT-MTB dataset, which includes 12 fine-grained categories (e.g., wide-bodied aircraft, naval vessels, trains) and supports both Oriented Bounding Box (OBB) and Horizontal Bounding Box (HBB) detection tasks.

• OBB Detection Performance: On the SAT-MTB-OBB sub-dataset, TCL achieved a 47.7% mAP, a 4.8% improvement over the baseline (ReDet, 42.9%). It significantly outperformed other state-of-the-art oriented detectors (e.g., R3Det at 42.4%, Oriented RCNN at 40.4%). Notable improvements were seen in small instances (e.g., Other Ships, OS) and objects with large scale variations.
• HBB Detection & Generalization: When applied to the SAT-MTB-HBB sub-dataset and compared against general video detectors (e.g., FGFA, DFF) and image-based detectors, TCL (using Faster R-CNN as a base) achieved 34.2% mAP. This surpassed existing video detectors and improved upon the base image detector by 7.2%, demonstrating the framework's ability to enhance existing image-based detectors through temporal context mining.
• Ablation Studies:
  • TFA: Improved mAP by 3.8% (42.9% $\to$ 46.7%).
  • SE: Improved mAP by 2.6% (42.9% $\to$ 45.5%), particularly for objects with unique scales.
  • TCC: Improved mAP by 3.5% (42.9% $\to$ 46.4%) and resulted in faster convergence with smoother loss curves.

Significance
The paper claims that TCL represents a paradigm shift in satellite video object detection. Unlike image-based detectors that fragment temporal structures, TCL offers a holistic perspective that mines rich temporal contexts to detect both moving and static objects with high precision. By successfully integrating temporal consistency with oriented and fine-grained detection, the framework addresses the specific challenges of scale variation, arbitrary orientation, and weak appearance inherent in satellite imagery. The authors note that the framework is flexible, readily accommodating existing image-based detectors to enhance their performance, and establishes a new benchmark for the field.