💻 computer science

ExpoMotion: A Large-Scale Benchmark and A Householder Projection Network for Multi-Exposure Fusion

This paper introduces ExpoMotion, a large-scale benchmark with expert-verified ground truth for evaluating multi-exposure fusion in dynamic scenes, and proposes the Householder Orthogonal Projection (HOP) network, which utilizes Householder transformations to effectively decouple exposure alignment and ghost removal for superior artifact-free detail restoration.

Original authors: Yao Liu, Lishen Qu, Shihao Zhou, Jie Liang, Hui Zeng, Yabin Peng, Huipeng Lin, Lei Zhang, Jufeng Yang

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yao Liu, Lishen Qu, Shihao Zhou, Jie Liang, Hui Zeng, Yabin Peng, Huipeng Lin, Lei Zhang, Jufeng Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Ghost" in the Machine

Imagine you are trying to take a perfect photo of a busy street scene. You want to capture the bright sunlight hitting the buildings and the dark details in the shadows. But your camera has a limit: if you set it to see the dark shadows, the bright sun turns into a white blob. If you set it to see the sun, the shadows turn into black holes.

To fix this, photographers usually take three photos at once: one dark, one normal, and one bright. They then try to blend them together. This is called Multi-Exposure Fusion (MEF).

The Catch: If anything moves while you are taking those three photos—a person walking, a car driving, or even your hand shaking—the three photos don't line up perfectly. When you try to blend them, you get "ghosts." It looks like a transparent, blurry double-image of the moving person.

Until now, most computer programs trying to fix this have struggled because:

They were trained on boring, static photos (where nothing moves).
They didn't have a "perfect answer key" (Ground Truth) to learn from when things do move.

The Solution: A New Training Ground (ExpoMotion)

The authors realized they needed a better gym for their AI to train in. They built ExpoMotion, a massive new dataset.

The Analogy: Think of previous datasets as a practice field where the players never move. The AI learned to score goals but froze when the ball started moving. ExpoMotion is like a full-contact scrimmage with real players running, dodging, and changing speeds.
What's in it? It contains over 1,700 video sequences and nearly 11,000 images. It includes everything from controlled lab movements (like a robot arm moving) to chaotic real-world scenes (people walking in a restaurant, cars on a street).
The "Answer Key": The biggest challenge in dynamic scenes is knowing what the "perfect" photo should look like. The authors didn't just let a computer guess. They used a team of professional photographers and experts to manually curate the perfect "Ground Truth" images. They acted like art directors, ensuring the final result looked natural and free of ghosts, rather than just mathematically correct.

The New Tool: The "Householder" Mirror (HOP Network)

To solve the ghosting problem, the authors created a new AI network called HOP (Householder Orthogonal Projection).

Instead of trying to force the moving images to line up perfectly (which is like trying to glue two mismatched puzzle pieces together), HOP uses a clever mathematical trick based on mirrors.

The Analogy: Imagine you are looking at a reflection in a mirror. If you move your hand, the reflection moves. But if you have a specific type of mirror (a "Householder reflector"), you can mathematically "flip" the reflection so that the moving parts cancel each other out, leaving only the clear, static image behind.
How it works:
1. The Lighting Fixer (GPIA): First, the network makes sure the bright photo and the dark photo have similar brightness levels, so they aren't fighting each other.
2. The Ghost Eraser (HOA): This is the core magic. The network identifies the "ghost" parts (the moving artifacts) as a specific direction in math-space. It then uses a "Householder transformation" to project those ghosts out of the picture, like shining a light at a specific angle so the shadow disappears. It keeps the sharp details (like the texture of a brick wall) but throws away the blurry, moving double-images.
3. The Detail Sharpening (GGFN): Finally, it uses a special filter to make sure the edges and textures remain crisp, not blurry.

The Results: A New Champion

The authors tested their new AI against the current best methods.

The Scoreboard: On standard tests, their new method (HOP) scored higher than all previous champions. It produced clearer images with fewer ghosts.
The Real-World Test: When they tested it on photos it had never seen before (using the new ExpoMotion dataset), the old methods got confused and produced blurry, ghostly messes. The new HOP method handled the chaos beautifully, keeping details sharp and removing the ghosts.
Efficiency: It achieved these results without needing a supercomputer; it is actually quite efficient compared to other heavy-duty AI models.

Summary

In short, the paper says: "We built a massive, realistic training library called ExpoMotion because old libraries were too boring. Then, we built a new AI tool called HOP that uses a mathematical 'mirror trick' to erase moving ghosts from photos while keeping the details sharp. This combination works better than anything else currently available."

Technical Summary: ExpoMotion and the Householder Projection Network

1. Problem Statement

Multi-Exposure Fusion (MEF) is a critical technique for extending the dynamic range of standard imaging sensors, which often fail to capture scenes spanning over 10 orders of magnitude (from starlight to direct sunlight). While MEF offers a direct path to high-quality images in the display domain without the secondary artifacts (e.g., halos) associated with Tone Mapping Operators used in HDR reconstruction, it faces significant hurdles in dynamic environments.

The primary challenges are:

Motion Artifacts: Camera shake and moving objects cause severe misalignments between exposure frames, resulting in "ghosting" or tearing.
Benchmark Scarcity: Existing benchmarks largely focus on static scenes. Current dynamic datasets either lack reliable Ground Truth (GT) due to real-world motion or lack realism due to synthetic generation. Furthermore, existing GT generation methods often rely on tone-mapped HDR reconstructions, which inherit contrast distortions and fail to align with human perceptual preferences.
Algorithmic Limitations: Standard approaches relying on optical flow or similarity-based attention mechanisms struggle with MEF. Flow-based warping introduces geometric distortions in occluded regions, while standard attention fails to distinguish between useful complementary details (e.g., textures in over-exposed shadows) and misalignment artifacts.

2. Proposed Methodology

The authors propose a two-pronged solution: a new large-scale dataset (ExpoMotion) and a novel network architecture (HOP).

A. The ExpoMotion Dataset

To address the lack of high-quality dynamic benchmarks, the authors constructed ExpoMotion, a dataset comprising 1,738 sequences and 10,909 images.

Acquisition Strategy: The dataset integrates three protocols:
1. Controlled Motion: Static scenes with swapped reference frames to simulate motion while maintaining artifact-free GT.
2. Laboratory Motion: Captured in a professional motion lab to manipulate precise dynamics and illumination.
3. Real-World Motion: "In-the-wild" captures for no-reference perceptual evaluation.
Ground Truth Generation: A human-in-the-loop pipeline was employed. Candidate GTs were generated via multiple algorithms and commercial software (e.g., Lightroom, Photomatix). These were then screened by raters and meticulously refined by professional photographers to ensure photorealism, artifact-free quality, and alignment with human perception.
Scale and Diversity: The dataset is significantly larger than existing benchmarks (e.g., 19× larger than Kalantari17) and covers a wider range of luminance, resolution (avg. 2563 × 1896), and semantic diversity.

B. The Householder Orthogonal Projection (HOP) Network

The authors propose HOP, which reframes MEF deghosting from a mathematical perspective, decoupling the task into Exposure Pre-alignment and Ghost Filtering.

Global Priors Illumination Alignment (GPIA):
- Goal: Rectify drastic dynamic range discrepancies before feature aggregation.
- Mechanism: Inspired by Retinex theory, GPIA extracts global illumination priors via average pooling. It computes an illumination ratio map to modulate auxiliary frames, aligning their exposure levels with the reference frame. This decouples illumination correction from motion handling.
Householder Orthogonal Attention (HOA):
- Goal: Remove motion artifacts (ghosts) while preserving high-frequency details.
- Mechanism: Instead of standard similarity-based attention, HOA models artifacts as orthogonal perturbations.
  - It constructs a conflict vector ( $v$ ) based on the difference between reference and auxiliary features.
  - It employs a Householder transformation ( $H = I - 2vv^T$ ) to geometrically reflect auxiliary features.
  - By projecting features onto a hyperplane defined by the conflict vector, the network effectively "rejects" the misaligned ghost components while preserving the signal subspace.
  - An adaptive scaling factor and an intensity gate allow the network to fine-tune rejection strength and avoid suppressing useful details.
Gradient-Guided Feed-Forward Network (GGFN):
- Goal: Enhance high-frequency detail preservation (edges, textures).
- Mechanism: This module combines a learnable depth-wise convolution branch with a fixed Discrete Laplacian Operator branch. The Laplacian branch acts as a hard-coded high-pass filter, forcing the network to attend to structural boundaries. The two branches are fused via a learnable parameter, regularizing the learning process with explicit edge information.

3. Key Contributions

ExpoMotion Dataset: A large-scale, dynamic MEF benchmark featuring diverse motion patterns (controlled, lab, and real-world) and expert-refined Ground Truth, designed specifically for rigorous deghosting evaluation.
HOP Network: A novel architecture that utilizes Householder transformations to mathematically decouple exposure harmonization and ghost filtering, offering a geometric alternative to standard attention mechanisms.
Performance: Extensive experiments demonstrating that HOP outperforms state-of-the-art methods in both quantitative metrics (PSNR/SSIM) and perceptual quality (no-reference metrics), particularly in dynamic and extreme lighting conditions.

4. Experimental Results

Quantitative Performance: On the Kalantari17, RealHDRV, and the authors' controlled-motion test sets, the HOP-B model achieved state-of-the-art results. For instance, on Kalantari17, it achieved 28.804 dB PSNR, outperforming the strong competitor Restormer by ~0.5 dB while using only ~60% of the parameters (15.69M vs. 26.10M).
Generalization: When trained on ExpoMotion and tested on external real-world datasets (Sen and Tursen), HOP-B significantly outperformed models trained on other datasets (e.g., +4.9 points in MUSIQ score compared to training on Kalantari17). This validates the dataset's ability to teach models to generalize across diverse motion and lighting scenarios.
Visual Quality: Qualitative comparisons show HOP effectively removes ghosting artifacts around moving objects (e.g., lamp poles, people) and recovers details in extreme shadows and highlights where other methods fail or introduce noise.
Ablation Studies: Removing any core component (GPIA, HOPU, or GGFN) resulted in a performance drop of approximately 0.4–1.0 dB, confirming the necessity and synergy of the proposed modules.

5. Significance and Claims

The paper claims that ExpoMotion and HOP address the critical bottleneck of dynamic scene MEF, where current methods struggle with the trade-off between detail restoration and artifact suppression.

Dataset Impact: By providing a benchmark with reliable GT for dynamic scenes, the authors enable the community to rigorously evaluate deghosting capabilities, moving beyond the limitations of static or synthetically generated data.
Methodological Shift: The introduction of Householder transformations offers a mathematically grounded approach to deghosting, treating it as a geometric projection problem rather than a simple feature matching task. This allows for more robust handling of misalignments without the geometric distortions typical of flow-based warping.
Efficiency: The proposed method achieves superior performance with high computational efficiency, making it suitable for practical deployment.

Limitations: The authors acknowledge that the current framework is designed for a three-input fusion (over-exposed, reference, under-exposed). This limits immediate applicability to devices or datasets providing a variable number of exposures, a challenge identified for future research.