Fusion or Confusion? Assessing the impact of visible-thermal image fusion for automated wildlife detection

Technical Summary: Fusion or Confusion? Assessing the impact of visible-thermal image fusion for automated wildlife detection

Problem Statement
Effective wildlife monitoring is critical for biodiversity conservation, yet conventional methods (direct observation, camera traps) face limitations regarding accessibility, observer bias, and efficiency. While aerial imagery using visible (VIS) sensors combined with deep learning offers a promising alternative, relying on a single sensor modality often fails to detect cryptic species or distinguish targets from complex backgrounds. Thermal infrared (TIR) imagery provides complementary information, particularly for detecting heat signatures, but it suffers from lower spatial resolution and different fields of view (FOV) compared to VIS sensors.

The core challenge addressed in this study is the difficulty of aligning and fusing VIS and TIR data for automated detection. Misalignment due to differing FOVs and resolutions can degrade model performance. Furthermore, while early fusion (combining images at the input level) and late fusion (combining model outputs) are known concepts, there is a lack of published research comparing these specific methods on synchronous aerial VIS-TIR imagery for wildlife detection, particularly using state-of-the-art object detection models.

Methodology
The authors conducted a case study focusing on the Great Blue Heron (Ardea herodias) and their nests in six heronries across Quebec, Canada. Data was acquired using a DJI Matrice 300 drone equipped with a Zenmuse H20T sensor, capturing synchronous VIS (4056 × 3040 pixels) and TIR (640 × 512 pixels) images.

The methodology involved three primary stages:

1. Image Alignment: Due to the significant FOV difference (VIS is ~5x larger than TIR), the VIS images were center-cropped. A deep learning-based feature matching framework, LightGlue, was employed to automatically align VIS-TIR pairs. Pairs with insufficient keypoints (≤45) or high alignment error (>68 px) were discarded, resulting in a final dataset of 755 aligned image pairs.
2. Annotation: Objects were labeled with bounding boxes. In VIS images, classes included "occupied nest," "empty nest," and "isolated individual." In TIR images, where nests are rarely visible, all herons (isolated or inside nests) were labeled as a single "heron" class.
3. Model Training and Fusion Strategies: The study utilized YOLO11n, the latest YOLO architecture at the time of analysis. Three approaches were compared on the same test set:
   • VIS-only: A baseline model trained solely on visible imagery.
   • Early Fusion: VIS and TIR bands were fused at the input level. The VIS image was converted to YCbCr color space, and Principal Component Analysis (PCA) was applied to the R, G, B, and TIR bands. The first principal component (weighted heavily toward Red and TIR) replaced the luminance (Y) channel, creating a new fused RGB image for training.
   • Late Fusion: Independent models were trained on VIS and TIR data. Their detection outputs (bounding boxes and scores) were geometrically intersected. A Classification and Regression Tree (CART) algorithm was then trained on features including class scores, Intersection over Union (IoU) values, and one-hot encoded decisions to make the final classification. This approach also handled non-intersecting boxes by retaining single-modality detections.

Key Results
The study evaluated performance using the F1 score, precision, and recall across the test dataset (316 occupied nests, 35 empty nests, 20 isolated individuals).

• Baseline Performance: The VIS-only model achieved an overall F1 score of 56.6% (macro-averaged), with a strong performance on the majority class (occupied nest: 90.2%) but significantly lower scores for minority classes (empty nest: 44.0%; isolated individual: 35.7%). The TIR-only model achieved an F1 score of 85.4% for the "heron" class.
• Early Fusion: This approach improved the detection of minority classes compared to the VIS-only model (e.g., isolated individuals increased to 55.6%). It reduced false negatives (missed detections) to 4% but slightly increased false positives (9%). The overall F1 score was 64.3%.
• Late Fusion: This method excelled in detecting the majority class, achieving the highest F1 score for occupied nests (93.0%) and the lowest rate of missed detections (3%). However, it generated a higher rate of false positives (14%) compared to other methods. A notable finding was the CART classifier's ability to identify false positives from both sources with 90% recall, effectively filtering out uncorrelated errors.
• Data Loss: The strict alignment requirements for early fusion resulted in the discard of 61% of the available image pairs, primarily due to a lack of matching features in water backgrounds. Late fusion was less sensitive to pixel-perfect alignment, allowing for the utilization of more data.

Significance and Claims
The paper claims to be the first to successfully apply and compare late fusion against early fusion for synchronous aerial VIS-TIR wildlife detection. The authors assert that while fusion methods generally offer added value over single-source models, the choice of method involves trade-offs:

• Late Fusion is highlighted as particularly effective for the primary target (occupied nests) and offers greater operational robustness by tolerating imperfect image alignment, thereby reducing data loss.
• Early Fusion shows promise for minority classes but is constrained by the need for precise pixel-level alignment.
• Operational Limitations: The authors modestly conclude that for Great Blue Heron surveys specifically, the added value of fusion may not yet justify the operational complexity and data loss, given the high performance of VIS-only models for this specific species in daylight. They suggest that fusion techniques might be more beneficial for cryptic species (e.g., white-tailed deer) or nocturnal/crepuscular surveys where TIR provides critical information not available in VIS.
• Future Directions: The study suggests that using aircraft-mounted, very high-resolution visible sensors (wide FOV) could be a more practical compromise for operational surveys than current drone-based fusion approaches, which are limited by narrow FOVs and alignment constraints.

The research underscores the potential of deep learning for automated wildlife monitoring while realistically addressing the technical hurdles of multi-sensor data integration in real-world survey contexts.