Technical Summary: CMS2-Fusion

1. Problem Statement

Infrared and visible image fusion aims to integrate the thermal radiation information from infrared sensors with the rich texture and spatial details of visible cameras into a single comprehensive representation. While deep learning methods have advanced this field, existing approaches typically follow an "encode-then-fuse" paradigm. In these methods, infrared and visible modalities are processed through separate network branches (using CNNs, Transformers, or Mamba-based encoders) with no information exchange during the encoding phase. Cross-modal interaction is deferred until the final fusion stage, usually via simple concatenation or averaging.

The authors argue that this separation fails to exploit the full potential of State Space Models (SSMs), specifically Mamba. In Mamba, hidden states naturally capture spatial context through state transitions. By keeping modalities isolated during encoding, existing methods miss the opportunity to learn semantically coupled representations at the state-transition level, potentially limiting the model's ability to resolve inherent differences between modalities (e.g., low resolution in infrared vs. low visibility in visible light under poor illumination).

2. Methodology: CMS2-Fusion

The paper proposes CMS2-Fusion (Cross-Modal State Space Coupling), a framework that shifts the paradigm from "encode-then-fuse" to "encoding-as-coupling." The core innovation is a mechanism that allows the hidden states of both modalities to dynamically interact during the encoding process within Mamba blocks.

Key Components:

• CoupledVisionMambaBlock:

  • Instead of independent pipelines, the infrared ($I_{IR}$) and visible ($I_{VIS}$) feature maps are projected into a shared state space.
  • The state updates for each modality are modified to include a term from the other modality, scaled by a learnable coupling coefficient ($\lambda$).
  • The update equations are:
    $$h_{IR} = \text{Mamba}(P_{IR} I_{IR}) + \lambda \cdot P_{VIS \to IR} (P_{VIS} I_{VIS})$$
    $$h_{VIS} = \text{Mamba}(P_{VIS} I_{VIS}) + \lambda \cdot P_{IR \to VIS} (P_{IR} I_{IR})$$
  • Here, $\lambda$ is initialized to 0.3 and learned during training to adaptively control interaction strength. $P$ represents learnable projection matrices.
  • This design adds only a small constant overhead compared to standard Mamba, maintaining linear complexity $O(L)$.

• Parameter Sharing:

  • Within each block, the two modalities share the core Mamba SSM parameters (state transition matrix $A$ and selective projection parameters).
  • Input projections ($P_{IR}, P_{VIS}$) and cross-modal projections ($P_{VIS \to IR}, P_{IR \to VIS}$) remain modality-specific.
  • This strategy reduces the total parameter count by 13% compared to a decoupled baseline.

• Architecture:

  • Encoder: A three-stage pyramid structure where each stage consists of coupled Mamba blocks. Patch merging reduces spatial resolution while increasing channel depth.
  • Fusion Module: Located at the deepest feature level (16x16 resolution), it uses a spatial attention mechanism to weight features based on local "activeness" (L2 norm of feature vectors), automatically assigning higher weights to thermal salient regions for infrared and texture-rich regions for visible.
  • Decoder: A skip-connected decoder reconstructs the image to original resolution. Skip connections use simple averaging of coupled features, as semantic alignment has already occurred during encoding.

• Loss Function:

  • The model is trained with a composite loss: $\mathcal{L}_{total} = \mathcal{L}_{L1} + 10\mathcal{L}_{edge} + 2\mathcal{L}_{SSIM}$.
  • $\mathcal{L}_{L1}$ preserves brightness/contrast.
  • $\mathcal{L}_{edge}$ ensures gradient magnitude preservation.
  • $\mathcal{L}_{SSIM}$ enforces structural similarity.

3. Key Contributions

1. Cross-Modal State Space Coupling: The paper introduces a cross-modal coupling mechanism within Mamba encoding blocks, enabling interaction at the hidden-state level during encoding rather than deferring it to the fusion stage. While prior Mamba-based fusion methods exist (e.g., CrossMamba), they adhere to an independent encoding paradigm; this work specifically targets state-space-level coupling within the encoding phase.
2. Parameter-Efficient Design: By sharing SSM parameters across modalities, the model achieves a 13% reduction in parameters (8.4M vs. 9.6M) compared to a decoupled Mamba baseline, while reducing overfitting risk.
3. Paradigm Shift: The work proposes a shift from independent encoding to coupled encoding, demonstrating that state transitions can be used to model thermal-texture dependencies directly.

4. Experimental Results

Experiments were conducted on the MSRS dataset (1,083 training pairs, 361 testing pairs).

• Quantitative Performance:

  • $Q_{abf}$ (Quality of fusion): CMS2-Fusion achieved 0.7704, surpassing the classical CNN method DenseFuse (0.4525) by 70.3% and the decoupled Mamba baseline (0.7589) by 1.51%.
  • Information Metrics: Mutual Information (MI) increased by 9.2% (3.0548) and Entropy (EN) by 11.2% (6.6281) compared to DenseFuse.
  • Detail Preservation: Spatial Frequency (SF) improved by 87.7% and Average Gradient (AG) by 85.2% over DenseFuse, indicating superior edge and texture preservation.
  • SSIM: The metric showed a minor decrease from 0.6693 (DenseFuse) to 0.6562. The authors attribute this to the inherent trade-off between preserving thermal saliency and visible structural fidelity, noting that the value remains within a high range. Compared to the decoupled baseline, the SSIM improved by 0.94% (from 0.6501 to 0.6562).

• Ablation Studies:

  • Parameter Sharing: Moving from a decoupled baseline to a coupled model without interaction improved $Q_{abf}$ by 0.0085 and reduced parameters by 13%.
  • Interaction: Enabling the learnable coupling coefficient ($\lambda$) provided an additional $Q_{abf}$ gain of 0.0030.
  • Coefficient Convergence: The learnable $\lambda$ converged to approximately 0.3646, indicating an optimal balance between modality-specific retention and cross-modal interaction.

• Efficiency:

  • Inference time: 41.3 ms per image on an NVIDIA RTX 4090 (comparable to the 42.7 ms baseline).
  • FLOPs: 46.1G, lower than the baseline (48.3G).

5. Significance and Claims

The paper claims that CMS2-Fusion represents a paradigm shift in multi-modal fusion. By introducing cross-modal coupling at the state-transition level, the method moves beyond the limitations of "encode-then-fuse" architectures. The authors assert that this approach allows the network to learn more semantically coupled representations earlier in the pipeline, resulting in fused images that better preserve both thermal saliency and texture fidelity.

The work distinguishes itself from existing Mamba-based fusion methods (such as CrossMamba, FusionMamba, etc.) by being the first to apply state-space cross-modal interaction specifically during the encoding phase. While prior works utilize Mamba for feature extraction, they maintain independent branches until the fusion stage; CMS2-Fusion integrates the modalities directly within the state transitions of the encoder. The authors demonstrate that sharing SSM parameters and enabling dynamic state interaction yields superior performance with fewer parameters, making it a strong candidate for edge deployment.

Future work suggested by the authors includes layer-adaptive and spatially-adaptive coupling, as well as extending the framework to other multi-modal tasks like medical image fusion.