💻 computer science

FMRFusion: Frequency-Aware Multi-View Representation Learning for Heterogeneous Image Fusion

FMRFusion is a novel frequency-aware multi-view representation learning network that enhances infrared and visible image fusion by integrating multi-scale structural perception, bilinear frequency decomposition, cross-view complementary interaction, and flow matching to effectively capture distinct modal characteristics and produce high-quality composite images, particularly in nighttime scenarios.

Original authors: Tao Zhoua, Yunlong Liu, Qinghui Chen, Zekai Zhang, Minlong Sun, Changlin Biana, Dagang Li, Wenmin Wang, Jinglin Zhang

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tao Zhoua, Yunlong Liu, Qinghui Chen, Zekai Zhang, Minlong Sun, Changlin Biana, Dagang Li, Wenmin Wang, Jinglin Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take the perfect photo of a scene at night. You have two cameras:

The "Night Vision" Camera (Infrared): It sees heat. It can spot a person or a car in total darkness because they are warm. However, the picture it takes looks like a blurry, gray ghost. You can see where things are, but you can't see what they look like (no colors, no textures).
The "Daylight" Camera (Visible): It sees light and color. If there's a streetlamp, it captures beautiful details, colors, and textures. But if it's too dark, the picture is just black noise.

The Goal: Combine these two photos into one "Super Photo" that has the clarity of the daylight camera and the ability to see in the dark from the night vision camera.

The Problem: Previous methods tried to mash these two photos together using a single, blunt tool. It was like trying to mix oil and water by just stirring them with a spoon. The result often lost important details (like the texture of a car's paint) or got confused about what was actually a person and what was just background noise.

The Solution: FMRFusion
The authors of this paper built a new, smarter system called FMRFusion. Think of it as a master chef who doesn't just mix ingredients; they separate them, taste them individually, and then recombine them perfectly. Here is how it works, using simple analogies:

1. The "Frequency" Filter (Separating the Soup)

Imagine the two photos are a bowl of soup. Some parts are the "broth" (the big picture, the background, the general shape), and some parts are the "spices and chunks" (the tiny details, edges, and textures).

Old way: You tried to blend the whole soup at once.
FMRFusion way: It uses a special sieve (called Frequency Decomposition) to separate the broth from the spices.
- Low Frequency (The Broth): This captures the shared background (like the road or the sky). Since both cameras see the same road, this part is combined gently.
- High Frequency (The Spices): This captures the unique details. The "Night Vision" camera has the "heat spices" (the person's body heat), and the "Daylight" camera has the "texture spices" (the car's shiny paint). FMRFusion keeps these separate so they don't get lost.

2. The "Dual-Branch" Kitchen (Two Chefs, One Dish)

Instead of one chef trying to do everything, FMRFusion has two separate cooking stations (branches):

Chef A looks only at the Night Vision photo.
Chef B looks only at the Daylight photo.
They prepare their ingredients separately so the "heat" doesn't ruin the "color" and vice versa. Only at the very end do they bring their dishes together. This prevents the information from getting "muddy" or confused.

3. The "Cross-View" Handshake

Once the ingredients are prepped, the two chefs need to talk to each other.

The paper calls this Cross-View Complementary Interaction.
Imagine Chef A (Night Vision) says, "Hey, there's a person right there in the dark!" and Chef B (Daylight) says, "Got it! I'll make sure their coat looks detailed and colorful."
They explicitly share this specific information so the final image knows exactly where the person is and what they look like.

4. The "Polishing" Step (Flow Matching)

Even after mixing the ingredients, the photo might look a little rough or "coarse."

The paper uses a technique called Flow Matching. Think of this as a high-tech photo editor that takes the "rough draft" and slowly, step-by-step, smooths it out.
It learns how to transform a blurry, low-quality sketch into a crisp, high-definition masterpiece, ensuring the final result looks natural and sharp.

What Did They Prove?

The authors tested this "Super Chef" system on many different datasets:

Nighttime Scenes: They showed that their method creates clearer, more natural-looking night photos than previous methods, keeping both the heat signatures and the textures.
Medical Images: They tried it on MRI and PET scans (which are like different types of "medical night vision" and "medical daylight"). It successfully combined the structural details of one scan with the functional details of another.
Multi-Focus Photos: They tested it on photos where some parts are in focus and others are blurry. Their method managed to make the whole image sharp.
Finding Objects: They used their "Super Photo" to help a computer find people and cars. The computer found them more accurately when using the FMRFusion photo than when using just the infrared or just the visible photo.

In Summary:
FMRFusion is a smart system that stops trying to force two different types of photos to blend instantly. Instead, it separates them into "big picture" and "tiny details," lets them shine individually, helps them share their best features, and then polishes the final result to create a perfect, all-seeing image.

Technical Summary: FMRFusion

Problem Statement

Infrared and visible image fusion (IVIF) aims to generate a composite image that retains significant target information from thermal radiation while preserving detailed textures from reflected light. Despite recent advances, existing methods face three primary limitations:

Incomplete Feature Decoupling: Traditional CNN-based and Multi-Scale Transform (MST) methods often apply identical operators to both modalities, causing the loss of modality-specific features through shared encoding pipelines.
Local-Global Feature Tradeoff: CNNs excel at local context but struggle with long-range dependencies, while Vision Transformers offer global modeling at prohibitive computational costs.
Suboptimal Fusion Mechanisms: Many deep learning approaches concatenate source images prematurely into a single-path network, leading to information interference rather than structured feature interaction.

Furthermore, current methods often struggle to jointly model global contextual information and preserve fine-grained details, particularly when decoupling complementary representations across heterogeneous modalities in low-quality imaging scenarios.

Methodology

The authors propose FMRFusion, a frequency-aware multi-view representation learning network designed to address these challenges. The architecture integrates Convolutional Neural Networks (CNNs) and Transformers within a dual-branch hybrid framework.

1. Dual-Branch Hybrid Architecture

The network employs independent branches for infrared and visible inputs to avoid feature interference during encoding:

Shallow Feature Extraction: A Shallow DRSformer Encoder (SDE) extracts initial features. The DRSformer block utilizes a learnable parameter $K$ to dynamically control sparsity, filtering irrelevant information during feature interaction.
Feature Decomposition: The extracted features are decomposed into Low-Frequency (Base) and High-Frequency (Detail) components:
- Transformer Blocks (TRB): Utilize spatial self-attention (specifically the LT block) to extract low-frequency information and model long-range dependencies.
- Multi-Scale Structural Perception Module (MS-SPM): Designed to capture fine-grained local structures. It comprises two sub-modules:
  - Gradient-Aware Block (GAB): A ResBlock variant that computes gradient magnitudes using Sobel operators to preserve structural edges.
  - Enhanced Attention Block (EAB): Decomposes large convolutional kernels (e.g., $21\times21$ ) into smaller depthwise kernels ( $5\times5$ or $7\times7$ ) to capture rich contextual information efficiently.

2. Cross-View Complementary Interaction (CVCI)

To explicitly model the relationship between infrared thermal structures and visible color/texture, a Lighten Cross-Attention (LCA) module is introduced in the fusion layer:

Cross-Attention Block (CAB): Uses a symmetric dual-branch structure where one branch generates queries while the opposite modality provides keys and values, ensuring refinement via complementary information.
Intensity Enhancement & Color Denoising: Based on Retinex theory, the module decomposes features into illumination and reflectance components to enhance infrared intensity and suppress chromatic noise in visible features.

3. Flow-Matching Based Refinement

A semantic-guided Flow Matching module is employed as a post-processing refinement strategy (Stage III):

Mechanism: It treats the coarse fused image ( $F_1$ ) and semantic masks as a conditional prior. A conditional U-Net predicts a velocity field to transport a Gaussian noise prior toward the target high-quality representation.
Objective: This iterative process progressively refines the fused features, learning the transformation from coarse data to high-quality representations to enhance fidelity and perceptual quality without requiring ground-truth fused images (self-supervised).

4. Three-Stage Training Strategy

The model is trained in three distinct stages to optimize different aspects of the fusion process:

Stage I (Decomposition): Optimizes feature extraction and decomposition using reconstruction, decomposition, and gradient consistency losses.
Stage II (Fusion): Focuses on intensity preservation and feature fusion.
Stage III (Enhancement): Uses Flow Matching loss alongside intensity, gradient, and semantic consistency constraints to refine the final output.

Key Contributions

The paper summarizes its main contributions as follows:

Dual-Branch Hybrid Architecture: Combines CNN efficiency with Transformer global modeling capabilities in independent branches to prevent feature interference.
Frequency-Decoupled Fusion Mechanism: Introduces a novel module that explicitly separates and enhances shared structural information (low-frequency) and modality-specific details (high-frequency).
Flow Matching Based Refinement: Proposes a paradigm to iteratively enhance fused features by mapping coarse data to high-quality representations, improving fidelity and robustness.

Experimental Results

Extensive experiments were conducted on four infrared-visible datasets (MSRS, TNO, Roadscene, M3FD) and validated on medical image fusion (MRI-CT, MRI-PET, MRI-SPECT) and multi-focus image fusion tasks.

Quantitative Performance: FMRFusion achieved superior or state-of-the-art performance across multiple metrics (Entropy, Standard Deviation, Spatial Frequency, Visual Information Fidelity, etc.) on all tested datasets. For instance, on the MSRS dataset, it achieved the highest scores in EN (6.907), SD (48.539), SF (13.486), and AG (4.538).
Qualitative Performance: Visual comparisons demonstrated that the method effectively integrates thermal radiation with texture details, minimizing background information loss and distinguishing foreground targets better than competitors like SwinFusion, CDDFuse, and TDFusion.
Generalization: The model showed consistent performance when applied directly to medical image fusion and multi-focus fusion tasks without retraining, validating its cross-task generalization.
Downstream Task: In object detection experiments using YOLOv5 on the MSRS dataset, fused images produced by FMRFusion yielded higher mAP scores for "Person" and "Car" categories compared to single-modality inputs and other fusion methods.

Significance and Claims

The paper claims that FMRFusion addresses critical challenges in heterogeneous data fusion by:

Alleviating Information Loss: Explicit high/low-frequency decomposition prevents the loss of structural and intensity details common in single-branch architectures.
Balancing Local and Global Modeling: The hybrid strategy integrates gradient-sensitive filtering with global dependency modeling, ensuring robust structural consistency and fine-detail enhancement even in degraded imaging conditions.
Enhancing Cross-View Interaction: The Cross-View Complementary Interaction Module explicitly facilitates the exchange of complementary information between radiative intensity and reflected structural cues.
Progressive Refinement: The Flow Matching mechanism provides a continuous transformation for refining fused results, improving saliency and contrast.

The authors conclude that the method demonstrates strong performance on standard benchmarks and consistent cross-task generalization, validating its effectiveness for multi-view image fusion in complex scenarios, particularly nighttime environments.