Technical Summary: Thermo-VL

Problem Statement

Vision-Language Models (VLMs) have achieved significant success in open-ended image understanding and visual question answering (VQA) through instruction tuning on large-scale RGB datasets. However, these models inherit the failure modes of visible-spectrum sensing. Under low illumination, glare, haze, or smoke, RGB imagery often loses critical scene evidence, rendering standard VLMs brittle in safety-critical scenarios such as nighttime driving, search-and-rescue, and low-visibility inspection.

While thermal infrared (IR) imagery preserves complementary scene structure (e.g., humans, animals, warm vehicles) when visible cues degrade, current approaches to integrating thermal data into VLMs are suboptimal. Common workarounds involve translating thermal images into pseudo-RGB before feeding them into RGB-trained VLMs. This translation-based pipeline introduces a generative bottleneck, risks hallucinating visible-style details, and suppresses modality-specific information not naturally represented in RGB space. Furthermore, existing RGB-thermal datasets (e.g., KAIST, FLIR-ADAS) are primarily designed for detection or tracking and lack the language-supervised supervision required for instruction tuning and open-ended reasoning.

Methodology

The authors present Thermo-VL, a wavelength-aware VLM built upon the Molmo-7B-D backbone. The architecture is designed to integrate aligned RGB and thermal inputs without destabilizing the pretrained RGB–language interface or requiring expensive retraining of the entire backbone.

Architecture

1. Dual Visual Encoders:

   • RGB Stream: Uses the frozen Molmo pretrained vision tower ($\phi_R$).
   • Thermal Stream: Uses a ViT-L/14 encoder ($\phi_T$) initialized from a Masked Autoencoder (MAE) pre-trained on a pooled multi-band infrared corpus (KAIST, FLIR-ADAS, OSU/OTCBVS, NIRSCENE). To maintain compatibility with Molmo's 3-channel input expectation, single-channel thermal images are replicated across three channels.
   • Parameter Efficiency: During training, the RGB encoder and the Language Model (LLM) remain frozen. Only the last $N_\ell=4$ transformer blocks of the thermal encoder, the thermal LayerNorm, and the fusion module are updated.

2. Text-Guided Dual-Attention Fusion Module:
   The core innovation is a fusion block that conditions thermal features on both the language prompt and the RGB context before injecting them into the frozen RGB stream.

   • Input: Aligned RGB tokens ($R$), thermal tokens ($T$), and prompt embeddings ($P$).
   • Dual-Attention: The thermal tokens are refined via two parallel attention paths:
     • Text-Guided Path: $T_{txt} = \text{MHA}(\bar{T}, \bar{P}, \bar{P})$
     • RGB-Guided Path: $T_{rgb} = \text{MHA}(\bar{T}, \bar{R}, \bar{R})$
   • Refinement: These paths are merged with original thermal features to produce a refined representation $\hat{T}$.
   • Gated Residual Injection: Instead of replacing RGB tokens, the model predicts a residual $\Delta R$ from $\hat{T}$ and modulates it with token-wise gates $\alpha$. The final fused tokens are computed as:
     $$R^\dagger = R + \alpha \odot \Delta R$$
     This design ensures the fusion is query-aware and preserves the pretrained RGB–language interface while allowing targeted thermal evidence injection.

Training Objective

The model is optimized using a composite loss function:
$$L = L_{LM} + \lambda_{align}L_{align} + \lambda_{contr}L_{contr} + \lambda_{gate}L_{gate}$$

• $L_{LM}$: Standard autoregressive cross-entropy loss for the language modeling task.
• $L_{align}$: Token-level L2 alignment between RGB and thermal features to enforce cross-modal consistency.
• $L_{contr}$: Contrastive loss (InfoNCE) to align global semantic representations of RGB, thermal, and text.
• $L_{gate}$: Entropy regularization on the gating mechanism to prevent gate saturation (i.e., preventing the model from ignoring one modality entirely).
• Augmentation: A block-wise RGB masking strategy is employed during training, randomly masking 10% of the RGB image area to reduce over-reliance on the visible spectrum.

Key Contributions

1. Wavelength-Aware VLM: Introduction of Thermo-VL, which integrates paired RGB and thermal inputs via a text-guided dual-attention fusion module with token-wise gated residual injection, keeping the RGB tower and LLM frozen.
2. Parameter-Efficient Adaptation: A fine-tuning strategy that updates only the last few blocks of the thermal encoder and the fusion parameters, regularized by token-level alignment, contrastive supervision, and gate entropy.
3. Unified RGB-Thermal Language Supervision:
   • Dataset: A pixel-aligned RGB-thermal instruction-tuning dataset containing ~19,500 image pairs and ~54K QA pairs, constructed from public datasets and synthetic thermal augmentation (using F-ViTA) on PixMo-cap-qa data.
   • Benchmark: Thermo-VL-Bench, a manually screened, binary VQA benchmark with ~3,148 image pairs and ~15K QA pairs, designed for low-light and cross-spectrum reasoning.

Experimental Results

Experiments were conducted on Thermo-VL-Bench and the external RGB-Th Bench.

• Thermo-VL-Bench:

  • RGB+Thermal: Thermo-VL achieved 88.96% accuracy, a substantial improvement over the Molmo baseline (75.79%) when both modalities were provided.
  • Thermal-Only: Improved from 81.20% (Molmo) to 82.23%.
  • RGB-Only: Performance was preserved at 88.67%, matching the baseline, demonstrating that the fusion mechanism does not degrade existing RGB capabilities.
  • Overall: Thermo-VL achieved the highest combined score (86.37%) among all compared models (including InternVL, Qwen2.5-VL, and LLaVA-Next).

• RGB-Th Bench:

  • Thermo-VL matched the Molmo baseline on RGB-only text tasks (77.25%) but significantly improved performance on RGB-Thermal text tasks, rising from 60.84% to 69.15%.

• Ablation Studies:

  • Removing text-guided attention caused the largest performance drop (88.96% $\to$ 82.34%), confirming the importance of question-aware fusion.
  • Removing the gated residual or RGB-guided attention also degraded performance, validating the design's role in preserving the RGB interface while integrating thermal data.
  • Auxiliary losses ($L_{contr}$ and $L_{gate}$) were shown to be critical for cross-modal consistency and preventing modality collapse.

Significance and Claims

The paper claims that explicit, prompt-conditioned multispectral fusion allows thermal cues to complement RGB data effectively when fused inside the model, rather than being appended at the input level or translated to pseudo-RGB. Thermo-VL demonstrates that it is possible to extend powerful pretrained VLMs to thermal perception without destabilizing their pretrained RGB-language interface.

The authors position Thermo-VL as a research prototype rather than a deployment-ready system. They acknowledge limitations, including the assumption of spatially aligned inputs, the use of model-generated QA supervision which may introduce artifacts, and the fact that Thermo-VL-Bench is a manually screened, model-generated benchmark rather than a fully human-authored one. The work is intended to support research in multispectral vision-language learning and instruction tuning under low-light conditions, rather than serving as a standalone basis for safety-critical decision-making.