Full Dynamic Range Sky-Modelling For Image Based Lighting

This paper introduces Icarus, a deep learning-based all-weather sky model that overcomes the limitations of existing methods in handling full dynamic range and class-imbalanced solar regions to generate photorealistic, user-controllable environment maps for accurate Image-Based Lighting.

The Gist

Imagine you are trying to paint a realistic sunset on a canvas. To do it perfectly, you need to know exactly how bright the sun is, how the light filters through the clouds, and how the shadows fall on the ground.

For a long time, computer graphics had a problem: they could either paint a beautiful, fluffy cloud (but the lighting was wrong), or they could get the lighting mathematically perfect (but the clouds looked like blurry smudges). It was like having a perfect recipe for a cake but no oven, or a great oven but no recipe.

This paper introduces Icarus, a new AI system that solves this problem. Think of Icarus as a "Master Sky Chef" that can cook up a perfect, photorealistic sky in a computer, complete with the right light, shadows, and cloud textures, all while letting you move the sun and clouds wherever you want.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Blinding Sun" Dilemma

In the real world, the sun is incredibly bright compared to the blue sky. It's like comparing a candle to a spotlight.

• Old AI models tried to learn from photos of the sky, but because the sun is so much brighter than everything else, the AI got confused. It would either make the sun too dim (so the shadows were weak) or make the clouds too blurry (because it was too focused on the bright sun).
• The Result: When you put a 3D character into a scene with these old skies, the lighting looked fake. The shadows were soft and wrong, and the colors were off.

2. The Solution: The "Bracketing" Trick

Instead of trying to learn the whole sky in one giant, confusing picture, Icarus uses a clever trick called Bracketing.

Imagine you are taking a photo of a sunset. If you take one photo, the sun might be too bright (white blob) and the clouds too dark. If you take another, the clouds are clear but the sun is blown out.

• The Trick: Icarus breaks the sky down into a stack of "exposure brackets." Think of this like a set of sunglasses with different tint levels.
  • One layer is for the dark shadows.
  • One layer is for the mid-tones (the blue sky).
  • One layer is for the blindingly bright sun.
• The AI learns to paint each of these layers separately. Because it's not trying to handle the sun and the dark clouds at the exact same time, it doesn't get confused. It learns the details of the clouds in the "dark" layers and the intensity of the sun in the "bright" layers.

3. The Assembly: The "Mosaic"

Once the AI has painted all these separate layers (the brackets), it has to put them back together.

• The Glue: The paper describes a special "fusion" module. Imagine you have a mosaic puzzle where some pieces are dark and some are blindingly bright. Icarus has a smart glue that knows exactly how to blend them so you don't see the seams.
• The Result: When you look at the final image, it looks like a single, perfect photo. But underneath, it's a perfectly balanced stack of light.

4. The Superpower: User Control

The coolest part about Icarus is that it's not just a random generator; it's a tool for artists.

• The "Style" Dial: You can tell Icarus, "I want the clouds to look like a stormy day, but the sun to look like a golden sunset." It can mix and match textures.
• The "Sun" Slider: You can drag the sun to a different spot in the sky, and the AI instantly recalculates the shadows and light transmission (how light passes through glass or leaves) to match that new position.
• The "Cloud" Editor: You can tell it to "add more clouds here" or "make the sky clearer there," and it respects the physics of the light while doing it.

Why Does This Matter?

Think of Image-Based Lighting (IBL) as the "sunlight" in a video game or a movie.

• Before Icarus: If you wanted a realistic scene, you had to go outside with expensive cameras and take photos of the real sky (which is hard, slow, and you can't change the weather). Or, you used old computer models that looked fake.
• With Icarus: You can generate a perfect sky instantly. You can make a scene look like it's raining in London, then switch it to a sunny day in California, all while keeping the lighting physics accurate enough that a 3D character's shadow looks real.

In a Nutshell

Icarus is like a digital sky factory. It takes the messy, complex reality of the sun and clouds, breaks it down into manageable pieces (brackets), learns how to paint each piece perfectly, and then glues them back together into a sky that is both photorealistic (looks like a photo) and physically accurate (casts the right shadows). It gives artists the power to control the weather and the sun with a simple click, without needing to go outside or do complex math.

Technical Summary

Here is a detailed technical summary of the paper "Full Dynamic Range Sky-Modelling For Image Based Lighting" by Ian J. Maquignaz.

1. Problem Statement

The paper addresses the critical limitations of current Deep Neural Network (DNN) sky-models in generating Full Dynamic Range (FDR) environment maps for Image-Based Lighting (IBL).

• The Dynamic Range Gap: Real-world outdoor scenes have an exposure range exceeding 14 EV (Exposure Values), particularly due to the intense luminance of the solar disk compared to the sky. Existing DNN models often fail to faithfully recreate this range.
• The Class Imbalance Issue: The solar disk occupies a tiny fraction of pixels but holds the majority of the scene's luminous flux. This creates a severe class imbalance during training. Standard loss functions (like L1 or LPIPS) and aggressive tone-mapping operators (e.g., $\mu$-law) used to compress HDR data into LDR for training cause:
  • Exponential Errors: Small errors in the compressed LDR space result in massive errors in the reconstructed HDR space, particularly for the sun.
  • Visual Artifacts: Generated skies often suffer from blurred cloud formations, suppressed solar intensity, or "skewed" lighting that fails to cast accurate shadows.
• Current Workarounds Fail: Previous attempts to mitigate this, such as substituting the sun with parametric models (e.g., Hošek-Wilkie) or manually boosting HDR values, lack generalization across different weather conditions (clear, cloudy, overcast) and fail to capture realistic atmospheric attenuation.

2. Methodology: The Icarus Model

The authors propose Icarus, a novel all-weather DNN sky-model that generates photorealistic FDR environment maps without relying on aggressive tone mapping. The core innovation is a bracketing and fusion approach.

A. HDR Decomposition (Bracketing)

Instead of compressing the entire HDR image into a single LDR representation, Icarus decomposes the HDR ground truth into a set of N Low Dynamic Range (LDR) exposure brackets.

• Process: The HDR image is split into multiple LDR images, each representing a specific exposure time ($\Delta t_n$).
• Benefit: This avoids the non-linear relationship between LDR and HDR spaces introduced by tone-mapping operators. It allows the network to learn features at different intensity levels independently, mitigating the class imbalance of the solar region.

B. Model Architecture

Icarus is built upon a modified SEAN (Semantic Region-Adaptive Normalization) architecture with a novel decoder design:

1. Style Encoder & Mapper:
   • RGB-Style Encoder: Extracts style codes from a source image to allow user-controlled texture transfer (e.g., copying cloud styles).
   • RND-Style Mapper: Generates stochastic style codes based on segmentation labels, enabling unconditional generation of diverse cloud formations.
2. Multi-Head Decoder:
   • Instead of outputting a single HDR image, the decoder outputs a bracket of N LDR images ($\{\check{I}_n\}$).
   • Exposure Decay Training: To prevent the generator from "forgetting" features required for low exposures while learning high exposures, the authors employ a synchronous training strategy where exposure times are iteratively decayed from a base value to the target range over epochs.
3. Fusion Module:
   • After generation, the N LDR brackets are merged back into a single FDR image.
   • The paper proposes a DNN-based fusion module ($f_{DNN}$) that learns adaptive weights for merging, though it also supports established methods like Robertson fusion ($f_{Robertson}$).
   • The fusion is performed in floating-point precision to avoid truncation errors common in standard implementations.

C. Loss Functions & Discriminators

To ensure coherence between the generated brackets and the final HDR output, Icarus uses a dual-discriminator system:

• LDR Discriminator ($\check{D}$): Evaluates each exposure bracket independently to ensure visual quality and texture fidelity at specific exposure levels.
• HDR Discriminator ($\hat{D}$): Evaluates the entire bracket set to ensure continuity and coherence between exposures, preventing artifacts like "chromatic aberration" or feature desynchronization in the fused result.
• Losses: Includes class-selective L1 losses, LPIPS for perceptual quality, and modified hinge losses for adversarial training.

3. Key Contributions

1. FDR Sky-Modelling: The first DNN sky-model capable of learning and generating the full exposure range (14+ EV) of physically captured outdoor imagery, including accurate solar luminance.
2. Bracketing Strategy: A novel method to bypass tone-mapping limitations by training on decomposed LDR exposure brackets, solving the solar class-imbalance problem.
3. User Control: Enables intuitive user control over solar positioning, cloud formations, and atmospheric textures via style transfer (RGB-encoder) and stochastic generation (RND-mapper).
4. Robust Fusion: A custom floating-point implementation of Robertson fusion and a learnable DNN fusion module to reconstruct high-fidelity HDR images from generated LDR brackets.

4. Experimental Results

The model was evaluated against the Laval HDR Sky Database (HDRDB) and compared to the state-of-the-art AllSky model.

• Illumination Accuracy: Icarus significantly outperforms AllSky in Integrated Illumination ($\hat{I}$) and Peak Luminance ($PL_\Omega$). While AllSky often overexposes the sun (reaching 16 EV erroneously) or underexposes it, Icarus maintains stable, accurate solar intensity across sunrise-to-sunset scenarios.
• Visual Quality:
  • Metrics: Icarus achieves superior scores in FID, MiFID, and HDR-VDP-3, indicating better perceptual quality and fewer artifacts.
  • Shadows & Transmission: Renderings using Icarus environment maps show realistic light transmission through objects (e.g., glass orbs) and accurate shadow casting, which were often distorted in previous models.
• Resolution Scalability: The model performs effectively at high resolutions (512²), a regime where previous DNN sky-models typically fail due to the solar disk becoming a dominant, hard-to-learn feature.
• Ablation Studies:
  • Tone Mapping: Models trained without aggressive tone mapping (using $\gamma$-correction) showed greater stability in FDR metrics compared to those using $\mu$-law.
  • Fusion: While $f_{Robertson}$ provided the best visual quality, the proposed $f_{DNN}$ fusion module successfully learned to merge brackets, though it slightly under-represented peak luminance compared to standard methods.

5. Significance

This work represents a paradigm shift in computer graphics sky modeling:

• Bridging the Gap: It successfully bridges the gap between the photorealism of physical capture and the flexibility of generative AI.
• Practical Application: The generated maps are directly interchangeable with physically captured HDRI for Image-Based Lighting (IBL) in rendering engines (e.g., Blender), enabling realistic lighting for virtual objects in outdoor scenes.
• Scientific Impact: By accurately modeling solar intensity and atmospheric scattering without parametric substitution, the model offers a tool for scientific visualization and engineering applications requiring precise lighting simulation.
• Future Direction: The paper highlights the need for larger, more diverse FDR datasets, as current datasets (like HDRDB) are limited in geographic and temporal coverage, but the proposed architecture is designed to scale with better data.

In summary, Icarus solves the long-standing problem of generating high-dynamic-range skies with deep learning by decomposing the problem into manageable LDR brackets and fusing them back into a physically accurate HDR result, achieving unprecedented photorealism and lighting directionality.