Technical Summary: AdaFuse

1. Problem Statement

Cross-modal image fusion, particularly between infrared (IR) and visible (VIS) images, faces two primary challenges:

1. Imbalanced Feature Contributions: Fused images often suffer from one modality (typically the high-energy infrared) dominating the process, leading to color distortion, oversaturation, and the loss of visible texture details.
2. Computational Inefficiency: Existing deep learning approaches, such as those using residual connections or dense concatenation, often result in a significant increase in intermediate tensor dimensions. This leads to high computational costs and complex channel concatenation operations, hindering efficiency without necessarily improving fusion quality.

Additionally, traditional methods struggle with "highlight contamination," where thermal energy from high-temperature regions diffuses indiscriminately into surrounding non-thermal areas, creating pseudo-halo artifacts and weakening target saliency.

2. Methodology: AdaFuse

The authors propose AdaFuse, a cross-modal image fusion algorithm based on Negative Value Calibration and Adaptive Hierarchical Activation. The framework is designed to dynamically allocate computational resources and precisely control thermal feature diffusion.

2.1 Adaptive Hierarchical Activation

Instead of processing all image features through a uniform deep network, AdaFuse employs a hierarchical feature perception strategy:

• Channel Analysis: The algorithm analyzes the mean intensity deviation of the RGB channels in the visible image.
• Dynamic Routing:
  • Deep Path: Channels with high mean deviation (indicating significant grayscale information or potential color cast) are routed to a deep network branch for high-level semantic feature extraction.
  • Shallow Path: Channels with low deviation are processed by a shallow network to preserve foundational details.
• Residual Connection: The output of the shallow network serves as a connecting layer, aligned with the deeply processed features. This design replaces traditional ResNet layers, simplifying channel concatenation and reducing the number of intermediate tensors.

2.2 Negative Value Calibration and Diffusion Control

To address the issue of thermal energy diffusion (highlight contamination), the paper introduces a Negative Value Calibration mechanism:

• Thermal Segmentation: A threshold segmentation extracts a thermal target mask ($M_{IR}$) from the infrared image.
• Symmetric Calibration: For high-temperature pixels, the algorithm defines a thermal window and a symmetric non-thermal background window. It applies a Distance-Decay Modulation where the enhancement of thermal responses is restricted to the target region.
• Suppression: Pixels in the surrounding non-thermal background are subjected to a symmetric suppression response ($1 - \beta \cdot F_{IR}$), where $\beta$ is a learnable suppression coefficient based on distance from the thermal boundary. This prevents the "halo" effect and maintains clear gradient contrasts between thermal and non-thermal regions.

2.3 Network Architecture and Discriminators

• PTS Generator: A Precise Thermal Source (PTS) module uses a lightweight Convolutional Block Attention Module (CBAM) within a local window to generate a thermal saliency mask, guiding the fusion process.
• CA-Net Discriminator: A dual-branch discriminator (RGB branch and Gradient-aware branch) incorporates a Cross-Modal Attention (CA-Net) mechanism.
  • The visible grayscale features act as the Query ($Q$), while infrared features act as Key ($K$) and Value ($V$).
  • This allows the discriminator to adaptively weigh gradient constraints in infrared-dominant regions (e.g., thermal edges) while preserving texture details in visible-dominant regions.

2.4 Loss Functions

The training objective is guided by a comprehensive loss system:

• Adaptive Activation Loss ($L_{AAM}$): Dynamically balances the output between infrared and visible features based on local content weights ($\alpha$).
• Precise Thermal-Source Segmentation Loss ($L_{mod}$): Comprises three terms:
  • Center Constraint: Ensures thermal targets remain aligned with infrared responses.
  • Neighborhood Constraint: Manages the transition structure around targets to prevent abrupt changes.
  • Background Constraint: Ensures distant background regions rely primarily on visible features to prevent unnatural darkening or over-enhancement.
• Edge Loss: Explicitly strengthens the gradient contrast at thermal boundaries.

3. Key Contributions

1. Adaptive Computational Allocation: The introduction of a hierarchical activation strategy that routes features based on information density, significantly reducing computational load (FLOPs) compared to methods that process all channels uniformly.
2. Negative Value Calibration: A novel mechanism to suppress the uncontrolled diffusion of thermal energy, effectively eliminating pseudo-halo artifacts and preserving background fidelity.
3. Cross-Modal Discriminator (CA-Net): A discriminator design that utilizes cross-modal attention to provide modality-specific supervision, enhancing both structural fidelity and texture preservation.
4. Comprehensive Loss Framework: A multi-term loss function that jointly optimizes for thermal saliency, gradient preservation, and color balance.

4. Experimental Results

The method was evaluated on the MSRS and M3FD datasets, comparing against state-of-the-art methods including DDcGAN, SwinFusion, CrossFusion, DenseFusion, and FreeFusion.

• Quantitative Performance:
  • On the MSRS dataset, AdaFuse achieved a 7.83% increase in Entropy (EN) compared to the best baseline.
  • Structural Similarity (SSIM) improved by 4.23%, and the gradient-based fusion performance ($Q_{AB/F}$) increased by 14.96%.
  • On the M3FD dataset, the method demonstrated superior generalization. Specific results for individual images included an SSIM of 0.8916 (image 04037) and 0.8932 (image 01443), with VIF scores of 0.8463 and 0.8584 respectively.
• Qualitative Performance:
  • Visual results show better balance between thermal target enhancement and visible detail preservation.
  • The method effectively reduced color casts (e.g., excessive green channel dominance) and prevented the "halo" artifacts seen in other methods.
  • In smoke and low-light scenarios, AdaFuse maintained target distinguishability while preserving background structural details better than competitors.
• Efficiency:
  • The adaptive routing mechanism allows the computational cost to vary based on input complexity. For images with low inter-channel deviation, the cost is significantly lower.
  • The average inference time was 0.0065 seconds per image, outperforming several heavy-weight baselines like SwinFusion (0.0344s).

5. Significance and Claims

The paper claims that AdaFuse provides a methodological reference for broader tasks, such as hyperspectral and medical image fusion, due to its framework of hierarchical perception and precise control.

The authors assert that their approach:

• Significantly outperforms existing methods in both computational efficiency and fusion quality.
• Solves the specific problem of imbalanced modal contributions and intermediate tensor bloat.
• Offers a robust solution for complex environments (night, fog, rain) by effectively handling thermal target diffusion without sacrificing visible texture.
• Provides a favorable balance between infrared saliency enhancement and visible texture preservation, making it suitable for downstream vision tasks like object detection where clear target contours and background fidelity are critical.

The study concludes that the proposed framework effectively improves structural clarity and visual quality, offering strong support for practical applications in intelligent perception systems.