Technical Summary: UNIV – Unified Foundation Model for Infrared and Visible Modalities

1. Problem Statement

Recent advances in foundation models have yielded impressive results in single-modality tasks for both visible (RGB) and infrared (IR) domains. However, a significant "cross-domain gap" persists when applying these models across modalities. The authors identify the root cause as a pattern shortcut: models trained on a single modality tend to rely on superficial sensor-specific cues (e.g., color and brightness in RGB, or thermal intensity in IR) rather than underlying semantic structures.

Consequently, when an IR-trained model is applied to RGB data, it often mislocalizes attention based on brightness cues, and vice versa. Existing approaches lack a unified encoder capable of producing semantically consistent features for both modalities, limiting generalization in diverse perception settings (RGB-only, IR-only, or joint RGB-IR scenarios).

2. Methodology

The paper proposes UNIV, a unified foundation model designed to bridge the gap between infrared and visible modalities. The core of UNIV is a self-supervised strategy called Patch Cross-modal Contrastive Learning (PCCL).

2.1. Core Architecture

UNIV utilizes a trainable encoder ($E_{train}$) and a frozen, pre-trained RGB encoder ($E_{frozen}$) initialized from an ImageNet-pretrained model (specifically MCMAE).

• Input: Paired visible RGB and infrared images.
• Feature Extraction: The trainable encoder extracts features ($F^V$ for visible, $F^I$ for infrared) for both modalities.
• Goal: To learn a unified feature space where features of the same semantic class cluster together across modalities, while features of different classes are pushed apart.

2.2. Attention-Guided Patch Anchors

To guide the learning without ground-truth labels, UNIV leverages the semantic priors encoded in the frozen RGB encoder ($E_{frozen}$).

• Pseudo-Label Generation: The last-layer attention map from $E_{frozen}$ applied to the visible modality is used to determine semantic similarity between image patches.
• Binary Masking: This attention map is converted into a binary pseudo-label matrix ($M^A$) using three rules:
  1. Self-similarity: Diagonal elements are set to 1.
  2. Correlation Selection: For each patch, the top $m$ most correlated patches (based on attention scores) are marked as positive (1) until a cumulative threshold $\gamma$ is met.
  3. Dissimilarity Suppression: All other entries are set to 0.
• Anchors: The features from the frozen encoder serve as semantic anchors ($F^A$) against which the trainable encoder's outputs are aligned.

2.3. Patch Cross-modal Contrastive Learning (PCCL)

The optimization strategy aligns the trainable encoder's representations with the semantic anchors defined by $M^A$.

• Objective: The model minimizes the dissimilarity between negative pairs (different classes) and maximizes the similarity between positive pairs (same class) across modalities.
• Loss Function: The authors formulate the loss as a Binary Cross-Entropy (BCE) loss between the pseudo-label matrix $M^A$ and the sigmoid of the similarity matrix derived from the trainable features.
  • $L_{IA}$: Aligns Infrared features to the anchors.
  • $L_{VA}$: Aligns Visible features to the anchors (to prevent catastrophic forgetting of RGB performance).
  • Total Loss: $L_{PCCL} = \alpha \cdot L_{IA} + \beta \cdot L_{VA}$.
• Mechanism: This process pulls semantically relevant patches together across modalities and pushes unrelated ones apart, effectively reconstructing a unified feature space that prioritizes semantic structure over sensor-specific patterns.

2.4. MVIP Dataset

To support this cross-modal learning, the authors introduce MVIP (Multi-scene Visible Infrared Pairs), a comprehensive benchmark containing 98,992 precisely aligned image pairs.

• Sources: Curated from five public datasets: FLIR-Align, KAIST, DroneVehicle, LLVIP, and LasHeR.
• Coverage: Diverse scenarios including urban driving, surveillance, and aerial drone views, covering both daytime and nighttime conditions.

3. Key Contributions

1. Patch Cross-modal Contrastive Learning (PCCL): A self-supervised method that adaptively reduces the cross-modal feature gap while enhancing inter-class discriminability by using attention-guided pseudo-labels.
2. MVIP Dataset: The largest visible-infrared benchmark to date, providing a solid foundation for robust cross-modal research with nearly 100k aligned pairs.
3. State-of-the-Art Performance: UNIV achieves superior results on infrared tasks while maintaining competitive accuracy on RGB tasks.

4. Experimental Results

Extensive experiments were conducted on semantic segmentation and object detection tasks.

4.1. Infrared Semantic Segmentation (MSRS-IR)

• Performance: UNIV (LoRA variant) achieved 76.0 mIoU, surpassing the previous state-of-the-art (InfMAE) by +1.7 mIoU.
• Efficiency: This improvement was achieved while updating only 1.8M parameters (approx. 2% of the total 88.7M parameters) via LoRA.
• Full Pre-training: The full-parameter variant reached 76.6 mIoU.

4.2. Infrared Object Detection (M3FD-IR)

• Performance: UNIV (LoRA) achieved 56.9 mAP and 88.7 AP50, outperforming InfMAE by +0.7 mAP.
• Observation: The authors note that full-parameter pre-training yielded slightly suboptimal detection results compared to LoRA, attributing this to over-optimization of patch similarity at the expense of semantic richness within individual patches, which is critical for detection.

4.3. RGB Downstream Tasks

• Preservation of RGB Performance: UNIV maintained competitive performance on RGB tasks, achieving 79.0 mIoU on MSRS-RGB (matching the baseline) and 51.2 mIoU on ADE20K (retaining 99% of the original MCMAE capability).
• Significance: This demonstrates that UNIV successfully constructs a unified space without causing "modality collapse" or catastrophic forgetting of the original RGB capabilities.

4.4. Feature Space Analysis

• Alignment: Earth Mover's Distance (EMD) analysis showed that UNIV significantly improved cross-modal alignment, with same-category features clustering closer than different-category features.
• Separability: t-SNE visualizations confirmed that UNIV achieves better inter-class separability compared to single-modality models like MCMAE or InfMAE.

5. Significance and Claims

The paper claims that UNIV offers a general and scalable solution for building a unified cross-modal semantic feature space. Its primary significance lies in addressing the "pattern shortcut" problem, where models fail to generalize across modalities due to reliance on superficial sensor cues.

By integrating PCCL, UNIV reconstructs an effective unified semantic feature space that achieves:

1. Robust Cross-modal Alignment: Ensuring consistent feature representations for the same semantic objects regardless of the sensor modality.
2. Strong Inter-class Semantic Separability: Enhancing the model's ability to distinguish between different object classes.

The authors emphasize that this approach allows for efficient adaptation (via LoRA) to infrared tasks without sacrificing the model's original capabilities on visible tasks, providing a practical pathway for deploying robust perception systems in diverse weather and illumination conditions. The introduction of the MVIP dataset is also highlighted as a critical resource for future research in this domain.