Technical Summary: AnyThermal and TartanRGBT

1. Problem Statement

Thermal imagery offers robustness against lighting conditions and weather, making it critical for resilient autonomy in search and rescue, autonomous driving, and surveillance. However, unlike RGB data, thermal data suffers from a severe scarcity of large-scale, diverse datasets. Existing thermal feature extractors typically rely on task-specific training with small-scale data or adapt pre-trained RGB backbones. These approaches result in models that are limited to specific environments and tasks. Furthermore, while knowledge distillation from visual foundation models (e.g., DINOv2) to thermal domains has been explored, prior work has been constrained by a lack of diverse training data, often relying on single-environment datasets, which limits the generalizability of the resulting thermal encoders.

2. Methodology

A. AnyThermal: Task-Agnostic Thermal Encoder

The core contribution is AnyThermal, a thermal backbone designed to capture robust, task-agnostic features.

• Architecture: AnyThermal is based on the DINOv2 Vision Transformer (ViT-B/14).
• Training Strategy (Knowledge Distillation): The authors employ a teacher-student distillation framework.
  • Teacher: A frozen DINOv2 network initialized with pre-trained weights, processing RGB images.
  • Student: A trainable DINOv2 network (AnyThermal), initialized with the same weights, processing thermal images (converted from grayscale to 3-channel).
  • Loss Function: A contrastive loss is applied to the CLS token features of the final layer. This aligns the global semantics of corresponding RGB-thermal pairs without requiring low-level cues like color. This approach relaxes constraints on exact image alignment or synchronization, making it suitable for datasets where perfect registration is unavailable.
• Training Data: The distillation utilizes a combination of five datasets spanning four distinct environments: Urban (ViVID++, STheReO, Freiburg), Aerial (Boson Nighttime), Indoor, and Off-road.

B. TartanRGBT Platform and Dataset

To address the diversity gap in existing RGB-Thermal (RGB-T) data, the authors developed:

• TartanRGBT Platform: An open-source, hardware-synchronized data collection payload. It integrates a ZEDx stereo RGB camera and two FLIR Boson 640+ stereo thermal cameras, all time-synced via a trigger pulse from a capture card. The system runs on an NVIDIA Jetson AGX Orin and is housed in a custom 3D-printed case with active cooling.
• TartanRGBT Dataset: A diverse, balanced dataset containing 16,943 synchronized RGB-T pairs collected across four environments: residential/campus (urban), indoor, off-road, and trails/parks.
• Data Processing: The dataset includes stereo RGB, stereo thermal, and IMU data. For training, the authors generate registered RGB-T pairs using FoundationStereo to estimate dense depth from RGB, which is then used to back-project and align thermal pixels.

C. Downstream Task Adaptation

AnyThermal is evaluated as a feature extractor combined with task-specific heads:

• Cross-Modal Place Recognition (VPR): Uses a SALAD head trained with triplet margin loss.
• Thermal Segmentation: Uses a two-layer non-linear MLP head trained with Dice loss.
• Monocular Thermal Depth Estimation: Adapts the MiDaS framework, replacing the EfficientNet backbone with AnyThermal and utilizing multiscale patch features.

3. Key Contributions

1. AnyThermal: A task-agnostic thermal feature extractor that achieves state-of-the-art performance across diverse environments and tasks without task-specific pre-training of the backbone.
2. TartanRGBT Platform: The first open-source data collection platform for simultaneously capturing synchronized stereo RGB and stereo thermal images.
3. TartanRGBT Dataset: A diverse, balanced dataset covering indoor, aerial, off-road, and urban settings, designed to bridge the data diversity gap for thermal research.

4. Results

The authors evaluate AnyThermal on zero-shot and task-specific benchmarks, demonstrating significant improvements over existing baselines:

• Cross-Modal Place Recognition: On diverse zero-shot datasets (CART, MS2, OdomBeyondVision), AnyThermal with a VPR head outperforms all baselines, including frozen DINOv2, ImageBind, and SGM. Notably, it achieves up to 36% improvement in Recall@1 compared to baselines in certain environments. The results highlight that frozen RGB extractors are suboptimal for thermal queries and that distillation on diverse data is crucial.
• Thermal Segmentation: On the MF-Net dataset, AnyThermal achieves a state-of-the-art mIoU of 53.47%, outperforming the previous best (MCNET at 51.95%) while running 3.6x faster (6.79 FPS vs 1.88 FPS) on an NVIDIA ORIN AGX.
• Monocular Depth Estimation: On the MS2 dataset, AnyThermal achieves the lowest error metrics (AbsRel: 0.0883) compared to EfficientNet-lite3 and frozen DINOv2 backbones.
• Data Diversity vs. Scale: An ablation study on pre-training data scaling reveals that simply adding more data from similar domains (e.g., more urban data) yields diminishing returns. In contrast, adding the diverse TartanRGBT dataset consistently improves performance across all tasks and environments, confirming that data diversity is more critical than scale for building robust thermal feature extractors.

5. Significance and Claims

The paper claims that AnyThermal successfully bridges the gap between the abundance of RGB foundation models and the scarcity of thermal data. By leveraging knowledge distillation across a diverse set of environments, the authors demonstrate that a single thermal backbone can generalize effectively to tasks like place recognition, segmentation, and depth estimation without needing task-specific training of the backbone itself.

The introduction of the TartanRGBT platform and dataset is presented as a foundational step to lower the barrier for the research community to collect high-quality, synchronized RGB-T data. The authors assert that their work establishes a new standard for thermal perception, moving away from narrow, environment-specific models toward universal, robust representations. They conclude that while performance gains from scaling data exist, the primary driver for generalization is the diversity of the training environments, a principle validated by the superior performance of models trained on the combined, diverse dataset including TartanRGBT.