VC-FeS: Viewpoint-Conditioned Feature Selection for Vehicle Re-identification in Thermal Vision

Technical Summary: VC-FeS for Vehicle Re-identification in Thermal Vision

Problem Statement
Vehicle Re-identification (ReID) in thermal vision faces significant challenges compared to the RGB domain. Thermal images lack color information and fine texture details, which are critical discriminative features in standard ReID tasks. Furthermore, vehicles and vessels exhibit high intra-class similarity (e.g., identical models) and significant inter-viewpoint variability (front, side, rear). Existing methods, often pre-trained on RGB data, struggle in the thermal domain because they rely on color and texture patterns that are absent in grayscale thermal imagery. Additionally, the variability in viewpoint adds complexity, as feature vectors differ drastically depending on the camera angle, and determining which sides of a vehicle are visible in a single image remains a challenging pre-processing step.

Methodology
The proposed system, VC-FeS (Viewpoint-Conditioned Feature Selection), addresses these issues through a two-subsystem framework: object tracking and viewpoint-conditioned re-identification.

1. Object Tracking and Detection:
   The system adapts the TraDeS algorithm for the thermal domain. It utilizes a single backbone (DLA-34) to perform joint detection and tracking. Two key modules optimize performance:

   • Cost Volume based Association (CVA): Generates embeddings and derives object motions to improve tracking accuracy.
   • Motion-guided Feature Warper (MFW): Enhances the feature vector of the current frame based on the tracking history of past frames, propagating tracking cues to improve detection in occluded scenarios.
     The system outputs tracked bounding boxes which are then cropped for ReID.

2. Foreground Extraction:
   Since thermal images lack color contrast, conventional segmentation (e.g., GrabCut) fails. The authors employ a custom encoder-decoder architecture with residual connections to generate foreground masks. This model is trained on annotated thermal masks to accurately isolate vehicles from the background despite similar intensity distributions.

3. Viewpoint-Conditioned Feature Selection:
   The core ReID algorithm extracts features from the masked foreground using a Dino-ViT (Vision Transformer) pre-trained on ImageNet. The extracted features are mapped into four latent spaces via parallel linear layers: Global, Front, Side, and Rear.

   • ArcFace Mapping: Each latent space utilizes ArcFace to map feature vectors onto a hypersphere, maximizing inter-class variance and minimizing intra-class variance.
   • Area Ratio Calculation: A decoder breaks the foreground mask into side-specific masks (front, side, rear) to calculate the Area Ratio (AR) for each viewpoint.
   • Weighted Distance Calculation: The system calculates L2 distances between the query and gallery features in each of the four spaces. These distances are weighted by the corresponding Area Ratios (Eq. 2). This mechanism ensures that features from visible viewpoints contribute to the final score while suppressing contributions from non-visible viewpoints (where the AR is zero), effectively handling self-occlusion.
   • Loss Functions: The model is trained using a combination of Identity Classification Loss (Cross-Entropy) and Triplet Loss to learn discriminative shape features.

Key Contributions

• Novel ReID Algorithm: The authors propose a viewpoint-conditioned algorithm that constructs confidence scores by fusing features from separate latent spaces (front, side, rear, global) weighted by viewpoint visibility. This approach adapts RGB-pre-trained ViT extractors to the thermal domain by focusing on shape rather than color.
• Thermal Maritime Dataset: The paper introduces a new public dataset containing video footage and images of maritime objects (small and large vessels) from various angles, annotated with COCO standards for detection, tracking, and ReID. This is claimed to be the first public dataset of its kind for thermal maritime surveillance.
• Thermal Domain Adaptation: The work demonstrates the successful adaptation of the TraDeS tracking algorithm to the thermal domain, establishing a baseline for tracking vessels, ships, vehicles, and humans without color features.

Results
The system was evaluated on the RGBNT100 (IR vehicle) dataset and the authors' proprietary thermal maritime dataset.

• ReID Performance: The proposed method outperformed the state-of-the-art SPAN method.
  • On RGBNT100, it achieved a 19.7% improvement in mAP.
  • On the thermal maritime dataset, it achieved a 12.8% improvement in mAP.
  • In the authors' dataset, the method achieved a Top-1 accuracy of 81.82% and mAP of 63.08%, surpassing SPAN's 78.37% Top-1 and 55.88% mAP.
• Tracking Performance: The adapted TraDeS algorithm achieved a 61.2% MOTA score in the IR domain, comparable to its performance in the RGB domain (63.5% on MOT17), with a processing speed of 15 fps suitable for real-time tracking.
• Ablation Studies: Results indicated that using multiple viewpoint spaces (rather than just the largest view) improved Top-1 scores by an average of 15% across datasets. The use of ArcFace mapping and ViT-based feature extraction was shown to be critical for organizing features by orientation and improving discrimination.

Significance and Claims
The paper claims that by constructing viewpoint-conditioned feature vectors and performing area-specific feature comparisons, the system effectively leverages existing RGB-pre-trained ViT extractors while adapting them to the specific challenges of the thermal domain (lack of color/texture). The authors emphasize that this is the first time maritime vessel re-identification has been conducted in the thermal domain. The proposed method is presented as a robust solution for surveillance applications where color information is unavailable, establishing a new baseline for thermal ReID and maritime tracking. The authors plan to release their thermal maritime dataset to the public to facilitate further research in this under-explored area.