💻 computer science

InstantHDR: Single-forward Gaussian Splatting for High Dynamic Range 3D Reconstruction

The paper proposes InstantHDR, a feed-forward network that enables single-pass reconstruction of high dynamic range 3D scenes from uncalibrated multi-exposure low dynamic range images, achieving state-of-the-art synthesis quality with significantly faster inference speeds compared to existing optimization-based methods.

Original authors: Dingqiang Ye, Jiacong Xu, Jianglu Ping, Yuxiang Guo, Chao Fan, Vishal M. Patel

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Dingqiang Ye, Jiacong Xu, Jianglu Ping, Yuxiang Guo, Chao Fan, Vishal M. Patel

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

📸 The Big Problem: The "Overcooked" Photo

Imagine you are taking photos of a beautiful sunset.

If you set your camera for the bright sky, the ground looks pitch black.
If you set it for the dark ground, the sky looks like a washed-out white blob.

This is the problem with standard cameras (Low Dynamic Range or LDR). They can't see the bright sun and the dark shadows at the same time. High Dynamic Range (HDR) photography solves this by taking multiple photos at different settings and blending them together to create one perfect image that shows everything clearly.

The Catch: Doing this for 3D scenes (like a whole room or a city) usually takes a supercomputer hours to calculate, and it needs to know exactly where the camera was for every single photo. It's like trying to assemble a 3D puzzle while blindfolded, but you have to do it perfectly every time.

🚀 The Solution: InstantHDR (The "Magic Instant Camera")

The researchers at Johns Hopkins University created InstantHDR. Think of it as a "magic instant camera" for 3D worlds.

Instead of spending hours calculating and adjusting, InstantHDR looks at a few messy, multi-exposure photos and instantly (in a split second) builds a perfect 3D HDR model of the scene. You can then look at that 3D model from any angle and adjust the brightness (exposure) just like you want, without the image breaking or looking weird.

🛠️ How It Works (The "Chef's Kitchen" Analogy)

To understand how they did it, imagine a high-end kitchen trying to make a perfect soup from ingredients that are at different temperatures.

1. The Geometry Branch (The "Architect")

First, the system needs to know where things are.

Old Way: You had to measure every wall and object manually (slow and tedious).
InstantHDR Way: It uses a pre-trained "Architect" (a smart AI that has seen millions of rooms) to instantly guess the shape of the room and where the camera was standing. It doesn't need to measure; it just knows the layout.

2. The Appearance Branch (The "Master Chef")

This is the tricky part. The photos come in with different brightness levels (some are dark, some are bright). If you just mix them, you get a "ghostly" mess (like a double-exposure photo).

The Problem: A white wall looks bright in one photo and dark in another.
The Fix: The "Chef" uses Geometry-Guided Fusion. Because the "Architect" already told the Chef where the wall is, the Chef can say, "Ah, that bright patch in Photo A and that dark patch in Photo B are the same wall."
The Secret Sauce: They use a technique called Geo-Attention. Imagine the Chef has a laser pointer that instantly connects the same object across all the different photos, ignoring the brightness differences, so they can blend the textures perfectly without ghosting.

3. The Meta-Network (The "Taste Tester")

Every camera takes photos slightly differently (some are warmer, some are cooler).

Old Way: You had to tune the soup recipe specifically for this kitchen.
InstantHDR Way: They built a "Meta-Network" (a smart taste tester). It looks at the scene and instantly figures out, "Oh, this camera likes to make things look warm," and adjusts the final color automatically. It learns to adapt to any camera without needing to re-learn from scratch.

⚡ Why Is This a Big Deal? (The Speed Comparison)

The paper compares InstantHDR to the old methods using a race analogy:

The Old Methods (Optimization-based): Like a snail running a marathon. They take 25 minutes (or even 30+ minutes) to build one 3D scene. If you want to change the lighting, you have to start the race over.
InstantHDR: Like a supersonic jet. It builds the scene in less than 2 seconds.
- Speedup: It is roughly 700 times faster than the old methods for a quick look, and 20 times faster even if you want the absolute highest quality.

🧪 The Secret Ingredient: "The Fake Dataset"

One of the biggest hurdles was that there weren't enough "perfect" 3D HDR datasets to train this AI.

The Problem: You can't easily take a photo of a real room and know the true light values (HDR) because cameras clip the light.
The Solution: The team built HDR-Pretrain. They created 168 fake, photorealistic indoor scenes using a computer game engine (Blender). They simulated perfect lighting and took "photos" of them. This gave the AI a massive library of examples to learn from before it ever saw a real photo.

🌟 The Bottom Line

InstantHDR is a breakthrough because it turns a slow, expensive, and difficult process into something that happens instantly.

Before: "I need to take 50 photos, wait 30 minutes, and hope the computer doesn't crash to see this room in 3D."
Now: "I take a few photos, and poof—I have a 3D world I can walk through and adjust the lighting on instantly."

This technology could revolutionize things like autonomous driving (seeing clearly in blinding sun or pitch black tunnels), video games (real-time lighting changes), and virtual reality (making digital worlds feel as real as the real one).

1. Problem Statement

High Dynamic Range (HDR) Novel View Synthesis (NVS) aims to reconstruct 3D scenes with a broad luminance spectrum (0 to $+\infty$ ) from multi-view Low Dynamic Range (LDR) images captured at varying exposure times.

Current Limitations:

Optimization-Based Methods: Existing approaches (e.g., GaussianHDR, HDR-GS) rely on time-consuming per-scene optimization, requiring precisely calibrated camera poses and dense multi-view inputs. They struggle with sparse views and are too slow for real-time applications.
Feed-Forward Limitations: Recent feed-forward 3D reconstruction models (e.g., AnySplat) are fast but assume exposure-invariant appearance. When applied directly to multi-exposure LDR inputs, they suffer from severe ghosting artifacts due to brightness inconsistencies, fail to align pixels geometrically under large exposure variations, and cannot handle diverse Camera Response Functions (CRFs).
Data Scarcity: There is a lack of large-scale, multi-view HDR datasets required to pre-train robust feed-forward models.

2. Methodology: InstantHDR

The authors propose InstantHDR, the first feed-forward framework capable of reconstructing 3D HDR scenes from uncalibrated, multi-exposure LDR collections in a single forward pass. The pipeline consists of two main branches and a meta-network:

A. Architecture Overview

Geometry Branch (Frozen):
- Based on a pre-trained alternating-attention transformer (similar to VGGT/AnySplat).
- Estimates scene structure (depth maps) and camera poses from input images.
- Provides geometric correspondences (attention maps) to guide the appearance branch.
Appearance Branch (Trainable):
- Exposure Normalization ( $F_E$ ): Uses a FiLM layer to modulate appearance tokens based on relative log-exposure, aligning all views to a shared reference irradiance level before fusion.
- Geo-guided Cross-view Attention ( $F_A$ ): Reuses the attention maps from the frozen geometry encoder to fuse irradiance features across views. This ensures that features are matched based on geometric correspondence rather than just pixel intensity, effectively handling extreme exposure differences (e.g., matching a bright wall in one view to a dark wall in another).
- High-Resolution Upsampling ( $F_U$ ): Recovers pixel-level texture details lost during patch-based processing using a Difference of Gaussians (DoG) strategy. It adds high-frequency residuals from full-resolution LDR images to the fused low-frequency irradiance features.
Gaussian Head ( $F_G$ ):
- Combines geometry tokens and high-resolution irradiance features to predict HDR 3D Gaussian attributes: position ( $\mu$ ), opacity ( $\sigma$ ), rotation ( $r$ ), scale ( $s$ ), and HDR color embeddings ( $c^h$ ) represented as spherical harmonic coefficients encoding log-radiance.
Meta-Network ( $F_M$ ):
- A lightweight network that takes the predicted HDR Gaussians, LDR features, and exposure times as input.
- It predicts scene-specific parameters ( $\theta$ ) for a learnable tone mapper (a 2-layer MLP).
- This allows the model to adapt to different Camera Response Functions (CRFs) and render controllable LDR outputs at arbitrary target exposure times without per-scene optimization.

B. Training Strategy

Self-Supervised: The model is trained end-to-end using only multi-view LDR images with known exposure times. No HDR ground truth is required during training.
Loss Functions:
- Photometric Loss (MSE + Perceptual): Compares rendered LDR images against ground truth inputs at the same exposure.
- Geometry Consistency Loss: Enforces alignment between the frozen depth maps and the depth rendered from the predicted Gaussians (masked by a confidence map to ignore unreliable regions like sky).
Post-Optimization: An optional lightweight refinement step (1,000 iterations) can be applied to further improve quality, though the single-forward pass already yields strong results.

3. Key Contributions

InstantHDR Framework: The first feed-forward method for HDR NVS. It achieves 3D HDR reconstruction in seconds from uncalibrated multi-exposure inputs, eliminating the need for per-scene optimization.
Novel Modules:
- Geometry-Guided Appearance Modeling: Solves exposure-induced inconsistency by fusing features using geometric attention rather than naive pixel fusion.
- Meta-Network for Tone Mapping: Enables generalizable, scene-adaptive tone mapping without learning a fixed CRF.
HDR-Pretrain Dataset: To address data scarcity, the authors constructed a large-scale synthetic dataset containing 168 diverse indoor scenes rendered in Blender. It includes multi-view, multi-exposure LDRs, 32-bit HDR ground truth, depth, and normal maps, rendered under various tone-mapping operators (Standard, AgX, Filmic).

4. Experimental Results

The method was evaluated on the HDR-NeRF benchmark (synthetic and real scenes) and compared against state-of-the-art optimization-based methods (HDR-GS, GaussianHDR) and feed-forward baselines (AnySplat).

Performance (Zero-Shot):
- InstantHDR significantly outperforms AnySplat (e.g., +5.65 dB PSNR on real scenes with 8 views), eliminating ghosting artifacts.
- It achieves competitive quality to optimization-based methods in dense-view settings.
Performance (with Post-Optimization):
- InstantHDR_1K (1K iterations of refinement) surpasses GaussianHDR in sparse-view settings (4 views), achieving 22.16 dB PSNR vs. 19.26 dB.
- In dense views, it remains competitive with optimization methods.
Efficiency:
- Speed: InstantHDR is ~700× faster than optimization-based methods in single-forward mode and ~20× faster even with post-optimization.
- Time: Reconstruction takes ~1–2 seconds (single forward) or ~30–40 seconds (with refinement), compared to ~25 minutes for GaussianHDR.
Qualitative Results:
- Produces clean, exposure-controllable LDR views.
- Successfully handles extreme exposure variations (e.g., 0.5s to 32s) without ghosting.
- HDR rendering quality is comparable to optimization-based methods after refinement.

5. Significance

Real-Time HDR 3D: InstantHDR bridges the gap between high-quality HDR reconstruction and real-time inference, making HDR NVS feasible for applications like autonomous driving, digital humans, and immersive editing.
Robustness to Uncalibrated Data: By not requiring precise camera poses or dense initialization, it is more practical for real-world scenarios where data is often sparse and uncalibrated.
Generalization: The meta-network approach allows the model to generalize across different cameras and tone curves, a significant step forward from methods that overfit to specific scene CRFs.
Resource Efficiency: The introduction of HDR-Pretrain and the single-forward architecture lowers the barrier for future research in 3D HDR reconstruction.