💻 computer science

There and Back Again: A Flexible-Frame Transformer for Multi-Exposure Fusion

This paper introduces FreeMEF, the first flexible-frame transformer for multi-exposure fusion that utilizes a recurrent state space module and a global feature guided block to seamlessly handle varying numbers of input exposures without retraining, achieving state-of-the-art performance in image restoration.

Original authors: Lishen Qu, Yao Liu, Shihao Zhou, Jie Liang, Hui Zeng, Lei Zhang, Jufeng Yang

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lishen Qu, Yao Liu, Shihao Zhou, Jie Liang, Hui Zeng, Lei Zhang, Jufeng Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a perfect photo of a scene that has both a very bright sky and a very dark shadow. If you take one picture, the sky might turn white (blown out), or the shadow might turn black (crushed). To fix this, photographers often take several pictures at once: one bright, one dark, and one in the middle. They then "fuse" these pictures together to create one image that has all the details.

This paper introduces a new computer program called FreeMEF that does this fusion job better and more flexibly than anything before it. Here is how it works, explained simply:

The Problem: The "Rigid" Camera

Most existing computer programs for this task are like rigid vending machines. They are built to accept exactly three cans (three photos). If you try to put in two cans or five cans, the machine jams. In the real world, cameras might take 2, 3, or 5 photos depending on the situation. To use current technology, you'd need to build a different machine for every number of photos, which is slow and wasteful.

Also, these old machines have a "blind spot." When a part of the photo is too bright (saturated), the computer gets confused because that bright spot looks nothing like the dark spot in the other photos. It tries to match things that look similar, but in this case, the things that don't look similar are actually the ones that need fixing.

The Solution: The "Flexible" Transformer

The authors propose FreeMEF, which acts like a smart, adaptable chef instead of a rigid machine.

1. The "There and Back Again" Strategy (Flexible Inputs)
Instead of forcing the computer to look at all photos at once (which causes confusion), FreeMEF looks at them one by one, like reading a story page by page.

The Analogy: Imagine you are trying to remember a long list of items. Instead of trying to memorize 5 items all at once, you read the first, then the second, updating your mental note as you go.
The Tech: They use a Recurrent State Space Module (RSSM). This is a special memory unit that takes the first photo, then the second, then the third, blending them into a single "global memory" of the scene. Because it does this step-by-step, it doesn't matter if you feed it 2 photos or 100. It just keeps updating its memory. This means you only need one model to handle any number of photos.

2. Solving the "Similarity Paradox" (Fixing the Blind Spot)
The paper points out a funny problem: Standard computer vision tries to find matching parts between photos. But if a face is overexposed (too bright) in one photo and normal in another, they look totally different. The computer ignores the bright face because it doesn't "match" the normal face.

The Analogy: Imagine trying to find a lost key in a dark room. If you only look for things that look like the key, you might miss it if it's covered in dust. You need a flashlight that knows where the key should be, even if it looks different.
The Tech: They created a Global Feature Guided Block (GFGB) with two tools:
- Extremity-Aware Hybrid Attention (EAHA): This is the "flashlight." It detects the "extreme" spots (too bright or too dark) and tells the computer, "Don't look for a match here; look at the other photos for the missing details." It switches strategies automatically.
- Affine-Injection Feed-Forward Network (AFFN): This is the "dimmer switch." Once the details are found, this tool adjusts the brightness and contrast to make sure the final image looks natural, not just a patchwork of different lights.

The Result

When the authors tested this "smart chef" against the "rigid machines" (other top methods):

Quality: It produced sharper, clearer images with fewer "ghosts" (blurry double-images caused by movement).
Flexibility: It worked perfectly whether given 2, 3, or 5 photos, without needing to be retrained or changed.
Efficiency: It did this while using less computer power than many competitors.

In short, FreeMEF is a new way to combine multiple photos that is flexible enough to handle any number of inputs and smart enough to fix the tricky parts of a photo that other programs usually miss.

Technical Summary: FreeMEF – A Flexible-Frame Transformer for Multi-Exposure Fusion

Problem Statement
Multi-Exposure Fusion (MEF) aims to reconstruct High Dynamic Range (HDR) images by fusing multiple Low Dynamic Range (LDR) frames captured at different exposure levels. While learning-based approaches have advanced this field, they face two critical limitations hindering practical deployment:

Fixed-Input Constraints: Existing architectures are designed for a specific, fixed number of input frames (e.g., 2, 3, or 5). In real-world scenarios where exposure strategies vary, these models require architectural modifications and retraining for different frame counts, undermining deployment efficiency.
The Similarity Paradox: Conventional attention mechanisms rely on inter-frame similarity to aggregate information. However, in HDR imaging, the regions requiring the most restoration (saturated over-exposed or under-exposed areas) exhibit extremely low pixel-wise similarity to their counterparts in normal-exposure frames. Standard cross-attention mechanisms, which prioritize high-similarity regions, fail to effectively retrieve complementary information from these critical areas, leading to poor detail recovery.

Methodology: FreeMEF
The authors propose FreeMEF, a flexible-frame transformer framework that decouples reference features from base features, enabling a single model to handle arbitrary input sequence lengths without retraining. The architecture consists of two primary components:

Recurrent State Space Module (RSSM):
- Instead of processing frames in parallel, RSSM sequentially fuses features from an arbitrary number of input exposures.
- It utilizes an Attentive State-Space Equation (ASE) to capture global spatial-temporal contexts with linear computational complexity.
- The module employs a deformable alignment strategy to handle motion between frames and updates a hidden state recurrently. This allows the network to aggregate multi-frame information into a single global fused feature ( $H_T$ ) regardless of the input count.
Global Feature Guided Block (GFGB):
- This block processes the base frame using the aggregated global feature ( $H_T$ $H_{T}$ ) as auxiliary guidance. It comprises two sub-modules designed to resolve the similarity paradox and optimize exposure regulation:
  - Extremity-Aware Hybrid Attention (EAHA): To address the similarity paradox, EAHA generates a hybrid query vector. It estimates an extremity map to identify saturated regions. In these regions, the model switches from the unreliable base frame query ( $Q_{base}$ ) to a reference query ( $Q_{ref}$ ) derived from the global history. This allows the model to utilize structural priors from the global context to restore clipped details where pixel-wise similarity is low.
  - Affine-Injection Feed-Forward Network (AFFN): Standard Feed-Forward Networks (FFNs) lack the ability to adapt to global exposure shifts. AFFN injects global radiance information into the FFN via affine modulation (scaling $\gamma$ and shifting $\beta$ ). This explicitly recalibrates the contrast and brightness of local features based on the high-dynamic-range statistics preserved in the global history.

Key Contributions

Flexible-Frame Architecture: FreeMEF is the first MEF method that conceptually decouples reference features from base features, allowing a single model to train and infer on arbitrary numbers of input frames without architectural redesign or retraining.
Recurrent Fusion Mechanism (RSSM): A novel module that progressively aggregates exposure features using state-space modeling, accommodating variable input lengths while maintaining global context.
Resolution of the Similarity Paradox: The introduction of GFGB, specifically EAHA and AFFN, effectively resolves the failure of standard attention in saturated regions and achieves precise brightness control.
Superior Performance: The method demonstrates state-of-the-art performance quantitatively and qualitatively across multiple datasets.

Experimental Results
The authors evaluated FreeMEF on three benchmark datasets: Kalantari et al., Real-HDRV, and SICE.

Quantitative Performance: On the Kalantari dataset, FreeMEF achieved a PSNR of 28.418 dB, outperforming the second-best method (AFUNet) by 1.192 dB. On Real-HDRV, it improved PSNR by 0.515 dB over the runner-up.
Generalization (Cross-Dataset/Variable Frames): In cross-dataset evaluations (training on Kalantari, testing on SICE with 2, 3, and 5 frames), FreeMEF consistently outperformed fixed-input baselines. Notably, it showed significant gains in 2-frame and 5-frame scenarios where other methods required retraining or architectural changes. For instance, on the 5-frame SICE subset, FreeMEF improved PSNR by 1.116 dB over the second-best method.
Efficiency: The model maintains low computational cost, saving approximately 39% and 56% in FLOPs compared to AFUNet and HDR-Transformer, respectively, while keeping parameter counts suitable for mobile deployment.
Visual Quality: Qualitative comparisons show superior recovery of backlit faces and moving subjects, with effective suppression of motion ghosting artifacts common in parallel fusion methods.

Significance and Claims
The paper claims that FreeMEF addresses the fundamental rigidity of current MEF systems. By moving away from fixed-input constraints and parallel fusion paradigms, the proposed "there and back again" recurrent mechanism allows for seamless adaptation to diverse real-world exposure configurations. The authors assert that their approach not only improves restoration quality in challenging saturated regions but also establishes a more efficient and versatile framework for HDR imaging, eliminating the need for multiple specialized models for different frame counts.

Limitations
The authors acknowledge that while FreeMEF performs robustly, slight color deviations can occur in fused results compared to ground truth. This is attributed to overly large gaps between exposure levels, which can cause colors to deviate from the true scene appearance.

The Problem: The "Rigid" Camera

The Solution: The "Flexible" Transformer

The Result

Technical Summary: FreeMEF – A Flexible-Frame Transformer for Multi-Exposure Fusion

More like this