💻 computer science

EPOFusion: Exposure aware Progressive Optimization Method for Infrared and Visible Image Fusion

The paper proposes EPOFusion, an exposure-aware progressive optimization method that utilizes a guidance module, an iterative decoder, and an adaptive loss function to effectively fuse infrared and visible images in overexposed scenarios, supported by the introduction of a new high-quality overexposure dataset (IVOE).

Original authors: Zhiwei Wang, Yayu Zheng, Defeng He, Li Zhao, Xiaoqin Zhang, Yuxing Li, Edmund Y. Lam

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhiwei Wang, Yayu Zheng, Defeng He, Li Zhao, Xiaoqin Zhang, Yuxing Li, Edmund Y. Lam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a perfect photo of a scene that has two very different "cameras" looking at it:

The Visible Camera: This is like your human eye. It sees beautiful colors, textures, and details. But, if you look directly at the sun or a bright streetlight, it gets "blinded." The image turns into a white, washed-out blob where nothing is visible.
The Infrared Camera: This is like a "heat vision" eye. It doesn't care about bright lights; it sees heat. So, even if the visible camera is blinded by a glare, the infrared camera can still see the shape of a car or a person.

The Problem:
For years, scientists have tried to combine these two cameras into one "super image." But they hit a snag. When the scene is super bright (overexposed), the old methods get confused. They either:

Trust the visible camera too much: The result looks bright but loses the heat details (like a ghostly white blob).
Trust the infrared camera too much: The result keeps the heat details but looks grainy, noisy, and weirdly colored, ruining the natural look.

The Solution: EPOFusion
The authors of this paper created a new system called EPOFusion. Think of it as a smart photo editor with a "flashlight" and a "polishing station."

Here is how it works, using simple analogies:

1. The "Flashlight" (Guidance Module)

Imagine you are in a dark room trying to find a specific object, but there's a blinding spotlight shining on it. You can't see the object clearly because of the glare.

What EPOFusion does: It has a special "flashlight" (the Guidance Module) that shines specifically on the bright, overexposed spots. It tells the computer, "Hey, ignore the blinding white light here; look closely at the heat signature underneath!"
The Result: It forces the system to pay attention to the invisible details (like a car's engine heat) even when the visible image is just a white blob.

2. The "Polishing Station" (Iterative Optimization)

Old methods tried to mix the two photos in one giant, messy step. It's like trying to bake a cake, frost it, and decorate it all in one second. The result is often messy.

What EPOFusion does: It uses a "step-by-step" approach, inspired by how a sculptor chips away at stone. It starts with a rough mix and then iteratively (repeatedly) refines it.
The Analogy: Imagine cleaning a dirty window. You don't just wipe it once. You wipe, check, wipe again, and check again. With every pass, the image gets clearer, the colors get more natural, and the details get sharper. This "progressive optimization" ensures the final image looks perfect, not just "okay."

3. The "Smart Filter" (Adaptive Loss Function)

When training a computer to do this, you need to tell it what "good" looks like. But "good" changes depending on the situation.

The Problem: A rule that says "keep the colors bright" works for a normal day, but fails when the sun is too bright.
What EPOFusion does: It uses a Smart Filter (Adaptive Loss). This filter is like a chameleon.
- In normal areas, it says, "Keep the natural colors and textures."
- In the blinding bright areas, it switches gears and says, "Forget the colors for a second; make sure we see the heat and the shape!"
- It dynamically changes the rules as it works, ensuring the best balance everywhere.

4. The "Training Ground" (The IVOE Dataset)

To teach this system, you need practice photos. But existing photo collections didn't have enough examples of "blindingly bright" scenes with clear instructions on what to do.

What they did: The authors built a brand new training set called IVOE. They took normal photos and artificially added "glare" to them, then carefully marked exactly what should be seen in those bright spots. It's like creating a special "driving school" for cars to practice driving in heavy fog or blinding sun.

Why Does This Matter?

This isn't just about pretty pictures. It saves lives and improves technology:

Self-Driving Cars: If a car's camera gets blinded by the sun or a headlight, this system ensures the car still "sees" the pedestrian or the other car through the heat signature.
Security: It helps security cameras spot intruders at night, even if there are bright streetlights causing glare.

In Summary:
EPOFusion is like a super-smart editor that knows exactly when to ignore a blinding light and focus on the hidden heat details, then polishes the image step-by-step until it looks natural and clear. It solves the problem of "blinded vision" by using a smart flashlight, a step-by-step polishing process, and a flexible set of rules.

1. Problem Statement

Infrared and visible image fusion is critical for applications like autonomous driving and surveillance. However, overexposure (caused by strong daylight or intense glare) presents a significant challenge:

Visible Sensors: Suffer from pixel saturation, losing texture and gradient information in bright regions.
Infrared Sensors: Often contain thermal details in these same regions, but standard fusion methods fail to extract them effectively.
Existing Limitations:
- Generative Prior-based (GAN/Diffusion) methods: Tend to introduce background noise or global degradation, prioritizing global fidelity over semantic constraints.
- Task-driven methods: Often fail to distinguish informative features from noise, leading to redundant infrared information or degraded visual quality.
- Handcrafted Loss-guided methods: Rely on fixed loss functions that cannot dynamically capture the semantic complementarity between modalities in extreme lighting, often resulting in the loss of infrared details in overexposed areas.
Data Scarcity: Existing public datasets lack sufficient overexposed samples and precise pixel-level annotations for these regions.

2. Methodology: EPOFusion

The authors propose EPOFusion, an exposure-aware fusion framework that utilizes iterative strategies for progressive optimization. The architecture consists of three main components:

A. Exposure-Guided Fusion Framework

Input Processing: Visible images are converted from RGB to YCbCr; only the luminance (Y) channel is used to reduce redundancy. The Y channel and the Infrared image are concatenated and fed into a ConvNeXt encoder.
Guidance Module: A dedicated module predicts an infrared mask for overexposed regions. This explicitly steers the encoder's attention toward infrared information in saturated areas, ensuring the recovery of thermal details that would otherwise be lost.

B. Iterative Decoding Fusion Module

Inspired by Denoising Diffusion Probabilistic Models (DDPM), this module replaces the traditional single-step decoder with a multi-step iterative process:

Mechanism: The fusion process is modeled as a step-wise refinement. A pseudo ground truth ( $I_{pseudo} = 0.5I_{vi} + 0.5I_{ir}$ ) serves as a distribution anchor. Gaussian noise is added to this anchor, and a denoising network ( $D_\theta$ ) progressively refines the features over multiple iterations.
Multi-Scale Context Fusion (MSCF) Module: The denoised features are processed by the MSCF module, which contains two branches:
1. Contextual Fusion Branch (CFB): Uses dilated convolutions and grouped convolutions to capture long-range dependencies and global context.
2. High-Fidelity Branch (HFB): Uses standard convolutions to preserve fine-grained low-level details (edges, textures) from the source images.
- These branches are fused to reconstruct the final high-quality image.

C. Adaptive Loss Function

To handle varying exposure conditions, the authors designed a composite loss function that dynamically adjusts constraints:

Region Partitioning: The image is split into overexposed ( $M_{oe}$ ) and normal ( $M_{normal}$ ) regions based on segmentation masks.
Normal Regions: Uses standard intensity and texture losses to preserve the balance between visible and infrared modalities.
Overexposed Regions: The loss function shifts priority. It encourages the fused image to follow the infrared guidance (pseudo-target) rather than the saturated visible intensity, while emphasizing infrared gradient features to recover thermal details.
Diffusion Loss: Minimizes the discrepancy between true noise and predicted noise during the iterative process.

3. Key Contributions

EPOFusion Framework: A novel region-guided, exposure-aware iterative framework that effectively preserves infrared details in overexposed regions while maintaining visual fidelity in normal areas.
Novel Modules:
- A Guidance Module to focus attention on critical infrared features in saturated zones.
- An Iterative Decoding Fusion Module based on diffusion mechanisms for progressive feature refinement.
- A Multi-Scale Context Fusion (MSCF) module to balance global context and local details.
- An Adaptive Loss Function that dynamically constrains the fusion process based on exposure levels.
IVOE Dataset: The construction of the first Infrared-Visible Overexposure (IVOE) dataset. It includes:
- 2,315 synthetic training pairs with precise pixel-level annotations generated using the Segment Anything Model (SAM) and simulated Gaussian light spots.
- 176 real-world test pairs collected from public datasets to validate performance in natural overexposure scenarios.
Performance: Extensive experiments demonstrate that EPOFusion outperforms state-of-the-art (SOTA) methods in both visual quality and downstream task accuracy.

4. Experimental Results

The method was evaluated on the MSRS, FMB, and the new IVOE datasets against 13 SOTA methods (including GANMcC, U2Fusion, TarDal, SegMif, Dif-Fusion, etc.).

Quantitative Performance:
- On the IVOE dataset, EPOFusion achieved the highest scores in Mutual Information (MI: 5.137), Visual Information Fidelity (VIF: 0.912), and QAB/F (0.664).
- On MSRS, it secured top rankings in MI (4.579), VIF (1.064), and QAB/F (0.700).
- On FMB, it achieved the best VIF (0.930) and QAB/F (0.716).
Qualitative Performance:
- Visual comparisons show EPOFusion successfully recovers textures in overexposed skies and license plates where other methods fail (either losing infrared details or introducing noise).
- It maintains natural color gradients and avoids the "dark sky" artifacts common in generative models.
Efficiency: Despite the iterative process, the model maintains moderate complexity (33.87M parameters) and fast inference time (74.80ms on RTX 4090), balancing quality and speed.
Downstream Tasks: The fused images significantly improve the accuracy of object detection and semantic segmentation in overexposed scenarios.

5. Significance

This paper addresses a critical, often overlooked bottleneck in multi-modal fusion: extreme lighting conditions. By introducing an exposure-aware mechanism and a dedicated dataset (IVOE), the authors provide a robust solution that ensures safety-critical information (like thermal signatures of pedestrians or vehicles in glare) is not lost. The work bridges the gap between theoretical fusion models and practical deployment in real-world environments where overexposure is common, setting a new standard for robustness in infrared-visible fusion.