HDR Video Generation via Latent Alignment with Logarithmic Encoding

1. Problem Statement

High Dynamic Range (HDR) imagery offers a faithful representation of scene radiance but poses significant challenges for modern generative models (specifically diffusion models).

• Distribution Mismatch: Pretrained generative models are trained on Standard Dynamic Range (SDR) data, which is bounded (typically $[0, 1]$) and perceptually compressed. In contrast, raw HDR data exists in a scene-linear, unbounded space with heavy-tailed, highly unbalanced distributions spanning several orders of magnitude.
• Limitations of Existing Approaches:
  • Direct Modeling: Attempting to model HDR in linear space directly is difficult due to the statistical mismatch.
  • New Encoders: Training a dedicated Variational Autoencoder (VAE) or encoder to map HDR to a latent space introduces architectural complexity, requires massive datasets, and risks weakening the strong visual priors already captured by pretrained models.
  • Frame-by-Frame Generation: Existing methods often process video frame-by-frame, leading to temporal flickering and inconsistency.

2. Methodology: LumiVid

The authors propose LumiVid, a framework that generates temporally coherent HDR video from SDR inputs by leveraging a pretrained video diffusion model with minimal adaptation. The core philosophy is that HDR generation does not require new architectures, but rather a representation that aligns HDR data with the model's existing learned priors.

A. Latent Manifold Alignment via Logarithmic Encoding

The primary innovation is the use of a fixed, camera-inspired LogC3 encoding to map scene-linear HDR radiance into the latent space of the pretrained VAE.

• The Mechanism: Instead of training a new encoder, the pipeline applies a differentiable LogC3 transform ($T$) to map unbounded radiance $x \in [0, \infty)$ into the VAE's expected input range $[-1, 1]$.
• Alignment Strategy: The authors minimize the Kullback–Leibler (KL) divergence in two stages:
  1. Pixel Space: Ensuring the transformed HDR distribution matches normalized SDR statistics.
  2. Latent Space: Ensuring the VAE encoder produces latents consistent with those induced by native SDR inputs.
• Result: LogC3 achieves the lowest divergence and best roundtrip reconstruction quality compared to other encodings like PQ, HLG, and ACES, allowing the frozen VAE to handle HDR content without retraining.

B. Training Pipeline with Camera-Mimicking Degradations

To teach the model to hallucinate missing radiance (details lost in clipped highlights or crushed shadows) rather than simply copying pixels, the authors introduce a specific training strategy:

• Data Sources: They utilize synthetic HDR clips (rendered from PolyHaven HDRIs) and real-world HDR film (Tears of Steel) to create paired SDR-HDR data.
• Degradation Augmentation: The SDR reference input is deliberately degraded using:
  • MP4 compression artifacts.
  • Contrast scaling (clipping).
  • Selective blurring of highlights and shadows.
• Objective: These augmentations "erase" high-fidelity details in extreme luminance regions, forcing the model to rely on its internalized visual priors (e.g., how light glints off surfaces) to reconstruct plausible HDR content.
• Architecture:
  • Backbone: A pretrained Video Diffusion Transformer (DiT) with a frozen VAE.
  • Adaptation: LoRA (Low-Rank Adaptation) adapters are applied to the DiT. Only the LoRA weights (<1% of parameters) are trained.
  • Conditioning: The model uses the AVControl framework to condition the generation on the SDR reference.

C. Inference

1. An input SDR video is VAE-encoded.
2. The DiT (with trained LoRA) denoises the latent space conditioned on the SDR input.
3. The output latents are decoded by the frozen VAE.
4. An Inverse LogC3 transform converts the result back to scene-linear float16 EXR format.

3. Key Contributions

1. Representation Alignment: Demonstrated that a fixed, camera-oriented logarithmic encoding (LogC3) naturally aligns HDR distributions with the latent space of SDR-trained generative models, bypassing the need for new encoders.
2. Generative Reconstruction Strategy: Introduced a training regime using camera-mimicking degradations to force the model to synthesize missing radiance details from priors rather than performing simple pixel mapping.
3. Efficient Video Generation: Achieved high-quality, temporally coherent HDR video generation using a native video diffusion backbone with minimal parameter updates (LoRA), avoiding the flickering issues of frame-by-frame methods.
4. Minimal Data Requirement: Proved that high-quality HDR generation is possible with limited data (~300 clips, ~8 hours of training on a single GPU) by leveraging pretrained knowledge.

4. Experimental Results

The method was evaluated on out-of-distribution benchmarks, including professional ARRI Cinema Footage (48 clips) and the UPIQ image dataset.

• Reconstruction Fidelity:
  • ARRI Video: LumiVid achieved a JOD (perceptual quality) of 7.86 and PU21-PSNR of 36.20 dB, significantly outperforming baselines like HDRTVNet (JOD 6.94) and X2HDR (JOD 3.54).
  • UPIQ Images: Despite being a video model, it generalized well to static images, achieving 30.05 dB PU21-PSNR, outperforming all image-based baselines.
• Temporal Stability:
  • LumiVid is the only method that produces native video generation with high temporal coherence.
  • It achieved a Flicker index of 0.0245 and Frame-to-Frame PSNR of 45.63 dB, whereas frame-by-frame methods (like X2HDR) suffered from severe flickering (Flicker index 0.1630).
• Ablation Studies:
  • Encoding: LogC3 outperformed PQ, ACES, and HLG in perceptual metrics (JOD) and latent alignment.
  • Augmentation: The full degradation pipeline (blur + compression + contrast) yielded the best perceptual scores (JOD 7.86), confirming that corrupting inputs forces better generative synthesis.

5. Significance

This work fundamentally shifts the paradigm for HDR generation. Instead of viewing HDR as a task requiring entirely new model architectures or massive datasets, it demonstrates that pretrained SDR models already possess the necessary knowledge of light transport and dynamic range.

• Practical Impact: The method enables professional-grade HDR video generation (float16 EXR) suitable for post-production, color grading, and display technologies with minimal computational overhead.
• General Principle: The findings suggest that for other modalities where data distributions differ from training data, the solution may lie in distribution alignment and targeted augmentation rather than architectural redesign.
• Efficiency: By requiring only ~10,000 training steps and <1% parameter updates, LumiVid offers a highly scalable and accessible solution for the video industry.