CAWM-Mamba: A unified model for infrared-visible image fusion and compound adverse weather restoration

The paper proposes CAWM-Mamba, a unified end-to-end framework that jointly performs infrared-visible image fusion and compound adverse weather restoration using a Weather-Aware Preprocess Module, Cross-modal Feature Interaction Module, and Wavelet Space State Block to outperform existing methods in handling multiple simultaneous degradations while enhancing downstream perception tasks.

The Gist

Imagine you are trying to take a perfect photo of a city street at night to help a self-driving car navigate. You have two cameras:

1. The Thermal Camera (Infrared): It sees heat. It can spot a person in the dark or a car engine even if it's pitch black. But, the picture looks like a fuzzy, low-resolution ghost. It lacks details like the color of the car or the texture of the road.
2. The Regular Camera (Visible): It sees color and sharp details. But, if it starts raining, snowing, or gets foggy, the picture gets blurry, dark, or covered in streaks.

The Problem:
Most existing technology tries to fix these problems separately. It might try to clean the rain off the regular camera first, and then blend it with the thermal camera second. Or, it might only know how to handle rain, but if it's raining and foggy at the same time (a "compound" disaster), the system gets confused and fails. It's like trying to fix a car with a flat tire and a broken engine by only knowing how to change a tire.

The Solution: CAWM-Mamba
The authors of this paper created a new AI system called CAWM-Mamba. Think of it as a super-smart, all-in-one photo editor that doesn't just fix the picture; it understands why the picture is bad and fixes everything in one single, smooth motion.

Here is how it works, using simple analogies:

1. The Weather Detective (WAPM)

Before the system even starts blending the images, it puts on a detective hat. It looks at the blurry, rainy, or foggy picture and says, "Ah, I see rain streaks here, and heavy fog there."

• What it does: It creates a "weather map" (an embedding) that tells the rest of the system exactly what kind of mess it's dealing with. This helps the system know how hard to scrub the image clean.

2. The Handshake (CFIM)

Now, the system brings the Thermal Camera and the Regular Camera together. Usually, these two don't speak the same language. One talks in "heat," the other in "colors."

• What it does: This module acts like a translator and a handshake. It forces the two cameras to agree on where objects are. It takes the sharp edges from the regular camera and the heat signatures from the thermal camera and locks them together perfectly, so you don't end up with a ghostly car floating in the middle of the road.

3. The Magic Sieve (Wavelet Space State Block)

This is the most clever part. Imagine you have a bucket of mixed-up sand, pebbles, and water (the image with rain, fog, and snow).

• Old methods try to scoop everything out with a big spoon, often missing the small pebbles or leaving water behind.
• CAWM-Mamba uses a special sieve (Wavelet Decomposition). It separates the image into different "frequencies":
  • Low Frequency (The Big Picture): This catches the big, blurry stuff like fog. The system knows fog is "slow" and "blurry," so it smooths it out.
  • High Frequency (The Tiny Details): This catches the sharp, fast stuff like rain streaks or snowflakes.
• The Secret Sauce (Freq-SSM): Rain usually falls in straight lines (vertical or diagonal). Snow falls in flakes. The system has a special tool that knows exactly which direction the rain is falling and only scrubs that direction, leaving the rest of the image alone. It's like using a vacuum with a nozzle that only sucks up dust bunnies without sucking up your rug.

Why is this a big deal?

• It handles the "Double Trouble": Most AI breaks if it's raining and foggy. CAWM-Mamba handles both at the same time because it separates the problems into different "buckets" (frequencies) and fixes them individually.
• It's Fast and Efficient: It uses a new type of AI brain (Mamba) that is much faster and lighter than the old, heavy Transformers. It's like switching from a gas-guzzling truck to a sleek electric sports car.
• It actually helps robots: When they tested this on self-driving car tasks (like finding pedestrians or cars), the AI using CAWM-Mamba saw much more clearly than the others. It didn't just make a pretty picture; it made a safer picture for machines to drive with.

In a nutshell:
CAWM-Mamba is a unified, end-to-end system that acts like a master chef. Instead of washing the vegetables, chopping them, and then cooking them in three different pots, it has one magical pot that washes, chops, and cooks everything perfectly at the same time, even if the ingredients are muddy, frozen, and foggy all at once.

Technical Summary

1. Problem Statement

Context: Infrared (IR) and Visible (VI) image fusion is critical for applications like autonomous driving and UAV monitoring, as it combines the thermal sensitivity of IR with the high-resolution texture of VI.
The Gap: Existing fusion methods generally assume ideal, clean input images. In real-world scenarios, adverse weather (haze, rain, snow) severely degrades image quality.
Specific Challenge:

• Single vs. Compound: Current state-of-the-art (SOTA) methods typically handle only single types of degradation (e.g., just rain or just haze) or rely on a two-stage pipeline (restore then fuse), which often fails when multiple degradations coexist (e.g., rain + haze).
• Limitations: Two-stage approaches suffer from error accumulation and misalignment. Single-degradation models lack the capability to model the complex, overlapping frequency characteristics of compound weather.
• Goal: Develop a unified, end-to-end framework that simultaneously performs image restoration and fusion under compound adverse weather conditions without manual preprocessing.

2. Methodology: CAWM-Mamba

The authors propose CAWM-Mamba, an end-to-end deep learning framework based on the Mamba architecture (State Space Models) and Wavelet transforms. The network consists of three core modules:

A. Weather-Aware Preprocess Module (WAPM)

• Function: Processes the degraded visible image to enhance features and extract global weather context.
• Mechanism:
  1. Extracts initial features via convolution.
  2. Uses Global Average Pooling (GAP) and channel attention (Sigmoid) to generate an enhanced feature map.
  3. Key Innovation: Generates a Global Weather Embedding ($F_{embedding}$), a fixed-dimensional vector representing the overall weather condition. This embedding acts as a prior to guide subsequent fusion and restoration steps.

B. Cross-modal Feature Interaction Module (CFIM)

• Function: Aligns and fuses features from IR and VI modalities while respecting their unique characteristics.
• Mechanism:
  1. Modality Independence: Processes IR and VI features separately initially.
  2. Multi-scale Pooling: Uses Max Pooling on IR features to capture thermal contours and Avg Pooling on VI features to capture texture and global brightness.
  3. Adaptive Attention: Generates cross-modal attention weights. The IR features guide the enhancement of VI textures, while VI spatial details refine IR boundaries.
  4. Fusion: Combines these weighted features to ensure precise structural and texture alignment before restoration.

C. Wavelet Space State Block (WSSB)

This is the core restoration engine, designed to decouple different degradation types based on their frequency characteristics.

• Wavelet Decomposition: Uses Discrete Wavelet Transform (DWT) to split the image into Low-Frequency (LL) and High-Frequency (LH, HL, HH) subbands.
  • LL: Contains global structure and brightness (affected mainly by haze).
  • High-Freq: Contains directional details (affected by rain streaks and snow).
• Freq-SSM (Frequency-Selective State Space Model):
  • Low-Freq: Processed by a standard 2D-SSM (Mamba) to capture global structural dependencies.
  • High-Freq: Uses a novel Directional Alignment Mechanism. Unlike standard SSMs that scan indiscriminately, Freq-SSM aligns its scanning trajectory with the specific subband direction (e.g., vertical scanning for LH subbands). This efficiently models anisotropic degradations (like rain streaks) without redundancy.
• Common Degradation Space Mechanism (CDSM): A lightweight residual block with channel attention that maps features from different weather types into a unified latent space. This allows the model to learn shared degradation patterns, improving generalization to unseen compound scenarios.
• Weather-Guided Modulation: The global weather embedding from WAPM is injected into the WSSB to dynamically adjust feature weights, acting as a gate to filter residual noise specific to the detected weather intensity.

D. Loss Functions

The model is trained with a composite loss function:

• Wavelet Loss ($L_{wavelet}$): Enforces consistency in both low and high-frequency subbands.
• Pixel/Perceptual Losses: Includes brightness, color consistency, perceptual (MobileNetV3), gradient, and SSIM losses to ensure high-fidelity visual quality.

3. Key Contributions

1. Unified Framework: First end-to-end model to jointly handle compound adverse weather (e.g., rain+haze) and IR-VI fusion, eliminating the need for separate restoration and fusion stages.
2. Novel Architecture: Introduction of the WSSB with Freq-SSM, which leverages wavelet decomposition and directional state-space scanning to efficiently handle anisotropic, multi-frequency degradations.
3. Weather Embedding: The WAPM module explicitly extracts global weather information to guide the network, enhancing robustness against complex environmental interference.
4. Generalization: The CDSM enables the model to learn a unified representation of degradation, allowing it to generalize well across single and compound weather types, as well as clean scenarios.

4. Experimental Results

The model was evaluated on the AWMM-100K benchmark (compound weather) and three standard datasets (LLVIP, MSRS, M3FD).

• Quantitative Performance:
  • Compound Weather: CAWM-Mamba achieved the 1st rank in average metrics across all three compound scenarios (Haze&Rain, Rain&Snow, Haze&Snow), significantly outperforming SOTA methods like AWFusion, SHIP, and TarDal.
  • Single Weather: Achieved perfect Avg. Rank of 1.00 across Rain, Haze, and Snow scenarios.
  • Standard Fusion: Maintained top-tier performance on clean datasets, proving it does not sacrifice quality in ideal conditions.
• Downstream Tasks:
  • Object Detection (YOLOv7): Fusion results from CAWM-Mamba yielded the highest mAP scores, outperforming the second-best method by a statistically significant margin.
  • Semantic Segmentation (BANet): Achieved the highest mIoU, indicating superior preservation of semantic details.
• Efficiency:
  • Despite having 31.95M parameters, CAWM-Mamba has only 371.01 G FLOPs, which is approximately 1/11th of the computational cost of the Transformer-based baseline AWFusion (4167.20 G), making it highly suitable for resource-constrained devices like UAVs.

5. Significance and Impact

• Real-World Applicability: The ability to handle compound degradations (e.g., rain and fog simultaneously) addresses a critical gap in autonomous driving and surveillance, where weather conditions are rarely singular.
• Efficiency: By utilizing the linear complexity of Mamba and wavelet decomposition, the model offers a superior trade-off between performance and computational cost compared to heavy Transformer-based approaches.
• Robustness: The "Zero-Shot" generalization to real-world unseen scenes and clean scenarios demonstrates that the model learns fundamental degradation physics rather than overfitting to synthetic training data.

Conclusion: CAWM-Mamba represents a significant advancement in multimodal perception, providing a robust, efficient, and unified solution for operating in the most challenging weather conditions.