Capturing Stable HDR Videos Using a Dual-Camera System

This paper proposes a novel dual-camera system with an asynchronous exposure control and a corresponding exposure-adaptive fusion network (EAFNet) to generate stable, high-quality HDR videos by decoupling temporal luminance anchoring from detail reconstruction, thereby overcoming the temporal flicker issues inherent in traditional single-camera alternating exposure methods.

The Gist

The Problem: The "Flickering Camera" Struggle

Imagine you are trying to take a video of a beautiful sunset. The sky is bright orange, but the ground is dark. If you set your camera to capture the bright sky, the ground turns into a black silhouette. If you set it to capture the ground, the sky turns into a blinding white blob.

To fix this, standard cameras use a trick called Alternating Exposure (AE). They take one photo with the shutter open a tiny bit (to see the sky) and the next photo with the shutter open longer (to see the ground), then they stitch them together.

The Catch: In a video, this happens so fast that the camera is constantly switching between "bright mode" and "dark mode."

• The Result: The video looks like a strobe light. The brightness of the whole scene jumps up and down every frame, creating a annoying "flicker." It's like trying to watch a movie where the lighting in the room keeps randomly dimming and brightening.

The Solution: The "Two-Camera Team"

The authors of this paper realized that trying to fix the flicker with software alone is like trying to stop a leaky boat by bailing water with a spoon. Instead, they changed the boat.

They built a Dual-Camera System (DCS). Think of it as a two-person photography team instead of one person trying to do everything.

1. Camera A (The Anchor): This camera never changes its settings. It takes a steady, medium-exposure video. Its only job is to be the "steady hand" that keeps the brightness consistent from frame to frame. It acts as the temporal anchor, ensuring the video doesn't flicker.
2. Camera B (The Explorer): This camera is the wild card. It rapidly switches between very dark and very bright exposures. Its job is to grab all the hidden details in the shadows and the highlights that Camera A misses.

The Magic: Because Camera A is always steady, the video never flickers. Because Camera B is always exploring, the video never loses detail. The computer then blends these two streams together to create a perfect, stable, high-quality video.

The Secret Sauce: EAFNet (The Smart Blender)

Just having two cameras isn't enough; you need a smart way to mix the footage. The authors created a special AI brain called EAFNet to do the blending. You can think of EAFNet as a Master Chef preparing a complex dish:

1. Pre-Alignment (The Prep Station): Before cooking, the chef checks the ingredients. The AI looks at the "dark" and "bright" photos and adjusts their brightness levels so they match the "steady" photo. This ensures the chef isn't trying to mix apples and oranges.
2. Asymmetric Fusion (The Taste Test): This is the most clever part. Usually, AI treats all photos equally. But here, the AI knows that the "Steady Camera" (Camera A) is the boss.
   • It uses a special attention mechanism that says: "Trust the Steady Camera for the structure and movement. Only borrow the 'dark' or 'bright' details from the Explorer Camera if they are perfect."
   • If the Explorer Camera sees a ghost (a blur caused by movement), the AI ignores it and sticks to the Steady Camera's version. This prevents the "ghosting" artifacts (where objects look like they have double outlines) that plague other methods.
3. Restoration (The Plating): Finally, the AI polishes the image, sharpening the edges and making sure the colors look natural, not washed out or neon.

Why This Matters

• No More Flicker: By separating the job of "keeping time" (Camera A) from "gathering details" (Camera B), the video stays smooth.
• No More Ghosts: The AI is smart enough to know which camera to trust when things move fast, so moving cars or people don't leave weird trails.
• Real-World Ready: Unlike expensive, bulky scientific cameras, this system uses two standard industrial cameras that can be mounted side-by-side. It's a practical, cost-effective way to get Hollywood-quality HDR video without the Hollywood price tag.

The Bottom Line

Think of traditional HDR video as a tightrope walker trying to balance a pole while juggling. It's unstable and prone to falling (flickering).

This new method is like having two tightrope walkers: one is a master of balance who never wobbles, and the other is a master of juggling who catches all the falling objects. They work together, and the result is a smooth, stable, and spectacular show.

Technical Summary

1. Problem Statement

High Dynamic Range (HDR) video acquisition is critical for capturing scenes with extreme lighting variations. While Alternating Exposure (AE) paradigms using a single camera are cost-effective, they suffer from a fundamental limitation: temporal flicker and instability.

• The Core Issue: In AE systems, the reference frame itself alternates between different exposure levels (e.g., low, medium, high). This causes the "temporal luminance anchor" to jump between frames.
• Consequences: Deep learning models trained on AE data struggle to distinguish between actual scene motion and exposure switching. This leads to feature interference, ghosting artifacts, and visible luminance fluctuations (flicker) in real-world dynamic scenes, even when using state-of-the-art neural networks.
• Hardware Limitations: Existing multi-camera HDR solutions often require strict hardware synchronization and specialized sensors, making them bulky, expensive, and unsuitable for high-frame-rate capture.

2. Methodology

The authors propose a holistic solution comprising a novel Dual-Stream Paradigm, a custom Dual-Camera System (DCS), and a specialized deep learning network called EAFNet.

A. The Dual-Stream Paradigm & Dual-Camera System (DCS)

Instead of alternating exposures on a single sensor, the system decouples temporal luminance anchoring from exposure-variant detail reconstruction.

• Hardware Setup: Two identical industrial cameras mounted side-by-side.
  • Reference Stream (Camera 1): Continuously captures fixed medium-exposure frames. This provides a stable temporal baseline, eliminating luminance jumps between frames.
  • Auxiliary Stream (Camera 2): Alternates between low and high exposures. This stream provides the necessary dynamic range (shadow and highlight details) but does not need to be temporally synchronized with the reference stream.
• Asynchronous Operation: The system relaxes strict synchronization requirements. The auxiliary stream can operate at a different frame rate or phase relative to the reference stream. The network constructs input groups by pairing the fixed reference frame with the nearest available low/high exposure pairs based on timestamps.

B. EAFNet (Exposure-Adaptive Fusion Network)

To process the asynchronous dual-stream data, the authors designed a three-stage network:

1. Pre-alignment Subnetwork:

   • Global Luminance Alignment (GLA): Maps all inputs to a consistent luminance distribution in the sRGB domain to minimize exposure-induced mismatches before feature extraction.
   • Exposure-guided Feature Selection Module (EFSM): Uses exposure priors to modulate features. It selectively emphasizes well-exposed regions and suppresses unreliable features (e.g., noise in shadows or saturation in highlights) based on the specific exposure time of each input.

2. Asymmetric Cross-Feature Fusion Subnetwork:

   • Asymmetric Cross-Attention (ACA): Unlike standard attention mechanisms that treat all inputs equally, ACA is reference-dominant. It injects reference features into the query projection to regularize attention.
   • Mechanism: In aligned regions, cross-correlation dominates. In misaligned or occluded regions (due to parallax or motion), the mechanism defaults to self-correlation (reference features), effectively suppressing ghosting artifacts from the non-reference stream.
   • Cross-Scale Guidance: Integrates coarse-scale attention maps to guide fine-scale alignment, handling large spatial offsets.

3. Restoration Subnetwork:

   • Utilizes Discrete Wavelet Transforms (DWT) to decompose features into frequency subbands.
   • Applies a Lightweight DomainPlus Block (LDPB) at low scales to correct high-frequency artifacts (ghosting) and simple convolutions at high scales for global structure refinement.
   • Reconstructs the final HDR image using Inverse Wavelet Transform (IWT).

3. Key Contributions

1. Paradigm Shift: Introduced a dual-stream generation paradigm that explicitly decouples temporal stability (fixed exposure) from dynamic range expansion (varying exposure), solving the inherent flicker problem of AE systems.
2. Hardware Innovation: Designed an Asynchronous Dual-Camera System (DCS) that enables high-frame-rate HDR capture without precise hardware synchronization, leveraging consumer-grade dual-camera availability.
3. Algorithmic Design: Proposed EAFNet, featuring:
   • GLA & EFSM: For robust feature selection under varying exposures.
   • ACA: A reference-dominant attention mechanism that prevents ghosting in dynamic scenes with parallax.
4. Comprehensive Evaluation: Validated the system on both synthetic datasets (Kalantari, Prabhakar) and a new self-captured real-world dataset with diverse lighting and motion.

4. Experimental Results

The proposed method demonstrates State-of-the-Art (SOTA) performance across multiple metrics:

• Image Quality (Static): On Kalantari and Prabhakar datasets, EAFNet achieved the highest PSNR-µ (44.69 dB and 41.80 dB respectively) and SSIM-µ, outperforming previous CNN and Transformer-based HDR deghosting methods.
• Temporal Stability (Video):
  • Flicker Suppression: The method significantly reduced Luminance Standard Deviation (LSD) and Mean Absolute Difference of Brightness (MADB) compared to AE-based methods (e.g., HDRFlow, LAN-HDR).
  • Visual Consistency: Qualitative results showed the elimination of "photometric jumping" and ghosting artifacts in fast-motion scenarios, maintaining consistent illumination across frames.
• Robustness to Parallax: Stress tests with simulated parallax (up to 100 pixels) showed that EAFNet maintained stability where CNN-based baselines failed due to large displacements, thanks to the reference-dominant attention mechanism.
• Cross-Dataset Generalization: The model showed strong transferability when trained on one dataset and tested on another, indicating it learns exposure-aware representations rather than dataset-specific statistics.

5. Significance and Impact

• Solving the "AE Paradox": The paper identifies that the flicker in AE HDR video is a structural issue of the paradigm, not just a model limitation. By decoupling the reference frame from exposure variation, they provide a fundamental solution to temporal instability.
• Practical Deployment: The asynchronous dual-camera design bridges the gap between theoretical HDR algorithms and practical deployment. It allows for high-frame-rate capture using standard hardware without complex synchronization triggers, making it suitable for future mobile devices with dual-camera arrays.
• Cost-Effectiveness: Unlike specialized HDR sensors or beam splitters, this solution uses two standard cameras, offering a low-cost, high-performance alternative for dynamic HDR video acquisition.

Limitations: The current implementation relies on Transformer-based modules, resulting in higher computational latency (approx. 4.76 FPS on RTX 4090 at 1080p), which poses challenges for real-time mobile deployment. Future work aims to optimize the model via quantization and pruning. Additionally, extreme contrast nighttime scenes with saturated light sources remain a challenge for feature alignment.