Technical Summary: A Lightweight Appearance-Guided Association Module for Underwater Multi-Fish Tracking

Problem Statement

The paper addresses the challenge of multi-object tracking (MOT) for densely stocked fish in fixed-camera underwater aquaculture environments. While continuous trajectory data is essential for population estimation and behavior analysis, tracking individual fish is hindered by:

• Visual Similarity: Individuals of the same species often share nearly identical appearances.
• Complex Dynamics: Frequent direction changes, crossings, mutual occlusions, and substantial spatial overlaps occur.
• Environmental Degradation: Underwater conditions introduce illumination variations, turbidity, and reduced contrast, which degrade image quality.

Limitations of Existing Methods:

• Motion/IoU-based trackers (e.g., SORT): Rely on spatiotemporal cues (Kalman prediction and Intersection over Union). When fish overlap or cross, their spatial and motion states become indistinguishable, leading to high rates of identity switches (IDSW) and trajectory fragmentation.
• Deep Appearance-based trackers (e.g., DeepSORT): While they use deep learning for re-identification, they incur significant computational overhead. Furthermore, deep features trained on pedestrian datasets often fail to generalize to underwater scenes with low contrast and partial occlusions, and they struggle when detection regions are contaminated by adjacent fish.

The core problem is the need for a real-time, lightweight association method that can exploit subtle appearance differences to complement spatiotemporal cues without the heavy computational cost of deep neural networks.

Methodology

The authors propose a Lightweight Appearance-Guided Association Module integrated into the Simple Online and Realtime Tracking (SORT) framework. The method operates in a "tracking-by-detection" paradigm without modifying the underlying object detector.

1. Handcrafted Appearance Feature Extraction

Instead of using deep learning, the module extracts three types of low-cost, handcrafted features from each detected fish region:

• HSV Color Histogram: The region is converted from BGR to HSV. The H, S, and V channels are quantized into 16, 8, and 8 bins respectively, resulting in a 1024-dimensional vector.
• Gradient-Orientation Histogram: Based on the Histogram of Oriented Gradients (HOG) principle, the grayscale region is divided into a 2×2 grid. Unsigned gradient orientations in [0°, 180°) are quantized into 9 bins per sub-region, creating a 36-dimensional vector.
• Morphological Descriptor: A 3-dimensional vector capturing the aspect ratio ($w/h$), logarithmic area ($\log(w \times h)$), and normalized diagonal length.

These are concatenated into a unified 1063-dimensional appearance vector ($f_i$).

2. XGBoost-Based Identity-Matching Model

The association between an existing track and a current detection is formulated as a binary classification problem (same fish vs. different fish).

• Input: A 2128-dimensional feature vector ($x_{ij}$) constructed by concatenating:
  • The reference appearance feature of the existing track.
  • The appearance feature of the current detection.
  • The IoU between the predicted and detected bounding boxes.
  • The normalized center distance between them.
• Model: An XGBoost classifier (50 trees, max depth 3) estimates the probability ($p_{ij}$) that the pair corresponds to the same fish.
• Training Strategy: Positive samples are pairs of the same fish in adjacent frames (filtered to exclude high-overlap occlusions). Negative samples are pairs of different identities in the same frame.

3. Adaptive Data Association

The core innovation lies in how the appearance probability is fused with the standard SORT cost matrix:

• Adaptive Weighting: The association cost ($C_{ij}$) is a weighted sum of the IoU cost and the appearance cost.
  • $w_{IoU} = \max(0, 1 - \alpha \cdot IoU)$
  • $w_{XGB} = 1 - w_{IoU}$
  • $C_{ij} = w_{IoU}(1 - IoU) + w_{XGB}(1 - p_{ij})$
• Logic: When the IoU is low (spatial overlap is small), the system relies primarily on motion/spatial cues. As IoU increases (indicating potential overlap or ambiguity), the weight shifts toward the XGBoost appearance probability to resolve identity conflicts.
• Efficiency: Pairs with IoU < 0.1 are discarded immediately without XGBoost inference to save computation.

Key Contributions

1. Lightweight Module: Proposes a module combining color, gradient, and morphological features with XGBoost, avoiding the computational cost and domain gap issues of deep re-identification networks.
2. Adaptive Fusion Strategy: Introduces a mechanism to adaptively incorporate appearance probabilities into the SORT cost matrix based on spatial overlap, allowing appearance cues to supplement spatiotemporal information specifically when ambiguity is high.
3. Performance Validation: Demonstrates significant improvements in identity preservation on the Multiple Fish Tracking 2025 (MFT25) dataset while maintaining real-time processing speeds.

Experimental Results

Experiments were conducted on seven sequences from the MFT25 dataset using a Leave-One-Video-Out (LOVO) cross-validation strategy.

• Identity Switches (IDSW): The proposed method reduced total IDSW from 1,513 (SORT) to 833, a 44.94% reduction.
• Identity F1 Score (IDF1): Improved from 53.75% (SORT) to 65.49%.
• MOTA: Increased from 98.52% to 98.82%.
• Speed: Excluding object detection, the tracking speed exceeded 700 FPS (compared to 714 FPS for SORT), demonstrating that the added appearance processing incurs negligible overhead.
• Ablation Studies:
  • Using XGBoost only as a post-processing correction (V1) reduced IDSW by only 5.49%.
  • Integrating probabilities into the cost matrix before Hungarian assignment (V2) was crucial, achieving the full 44.94% reduction.
  • The method showed transferability when applied to the ByteTrack framework, reducing IDSW by 42.64% relative to the baseline ByteTrack.
• Visual Analysis: Qualitative results showed that while SORT frequently swapped identities after severe overlap, the proposed method maintained consistent identities, reducing the number of predicted identities assigned to a single ground-truth fish (e.g., from 14 to 1 in sequence BT-001).

Significance and Claims

The paper claims that the proposed module successfully addresses the trade-off between identity consistency and computational efficiency in underwater fish tracking. By leveraging subtle, handcrafted appearance features and an efficient gradient boosting classifier, the method significantly improves identity preservation during frequent crossings and occlusions without requiring deep learning infrastructure.

The authors state that these results indicate the module's potential for real-time fish monitoring in intelligent aquaculture, providing reliable trajectory data for downstream tasks such as population estimation and abnormal behavior recognition. The work is presented as a practical solution for scenarios where deep appearance models are too costly or unreliable due to underwater imaging conditions.