⚡ electrical engineering

Relightable Gaussian Splatting for Virtual Production Using Image-Based Illumination

This paper proposes a virtual production framework that leverages Gaussian Splatting and known background imagery to decompose 3D scenes into fixed appearance and variable lighting components, enabling efficient, high-quality relighting and compositing without relying on low-resolution environment maps or complex physically-based rendering.

Original authors: Adrian Azzarelli, Nantheera Anantrasirichai, James Pollock, David R. Bull

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adrian Azzarelli, Nantheera Anantrasirichai, James Pollock, David R. Bull

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are filming a movie scene where an actor is standing in front of a giant, high-tech LED wall. This wall isn't just showing a background picture; it's also acting as the sun, the moon, and the streetlamps, casting realistic light and shadows onto the actor. This is called Virtual Production (VP).

The problem? In the real world, if you want to change the background from a sunny beach to a dark cave, you have to physically change the LED wall and re-shoot the whole scene. If you want to change the lighting later in the editing room, you can't, because the light is "baked" into the video.

This paper introduces a new digital trick that lets editors change the background and the lighting after the filming is done, without needing to re-shoot anything. They call this "Relightable Gaussian Splatting."

Here is how it works, broken down into simple concepts:

1. The "Foggy Glass" vs. The "Magic Window"

Traditional methods try to rebuild the entire 3D world (the actor, the floor, the air) and then try to guess how light bounces off everything. It's like trying to rebuild a house brick-by-brick just to change the color of the paint. This is slow, complicated, and often gets the physics wrong, especially when the light source (the LED wall) is right next to the actor.

The authors' method is different. Instead of rebuilding the whole house, they treat the 3D scene like a collection of millions of tiny, fuzzy dots (called Gaussians).

The Analogy: Imagine the actor is made of millions of tiny, glowing specks of dust floating in the air.
The Trick: Usually, these specks have a fixed color. The authors give each speck a tiny, invisible "remote control" (learnable parameters). This remote allows the speck to look at the background image and say, "Oh, the background is blue now, so I should look blue," or "The background is bright, so I should be brighter."

2. No "Magic 360 Cameras" Needed

Usually, to change lighting in a 3D scene, computers need a special "environment map"—a 360-degree photo of the room's lighting. But in Virtual Production, the light comes from a flat screen right in front of the camera. A 360-degree camera can't capture that flat screen correctly; it gets distorted.

The authors' method skips the 360-degree map entirely.

The Analogy: Instead of trying to memorize the whole room's lighting, each tiny speck of the actor just looks directly at the background image (the LED wall) and samples the color right where it thinks it's reflecting.
The Result: It's like the actor has a mirror on their nose that only reflects the specific part of the background they are looking at. This captures realistic reflections and shadows without needing complex physics equations or special cameras.

3. The "Two-Layer Cake" Strategy

To make this work, the system splits the scene into two layers:

The Fixed Layer (The Cake): This is the actor's actual shape and base color. It stays the same no matter what.
The Variable Layer (The Frosting): This is the lighting. It changes based on what background image you choose.

When you want to change the scene, you don't touch the actor (the cake). You just swap the background image (the frosting), and the system automatically adjusts the lighting on the actor to match.

4. The "Invisible Screen" for the Background

One of the coolest parts is how they handle the background itself.

The Problem: If you just put a new picture behind the actor, the lighting on the actor won't match the new picture perfectly because the original video had distortions from the real LED wall.
The Solution: The system "cuts out" the actor from the original video. Then, it places a simple, flat digital screen behind them. It paints the new background image onto this flat screen. Finally, it blends the actor (with their new, calculated lighting) in front of this clean screen.
The Benefit: This removes the "glare" and distortions from the original LED wall, making the final result look like the actor was actually standing in front of the new background.

Why This Matters (According to the Paper)

Speed and Cost: It's very fast to train (under 2 hours) and doesn't need super-expensive computer memory.
Flexibility: Directors can change the background and lighting in the editing room without calling the actors back to the studio.
Quality: The paper claims this method creates better-looking results than previous methods, especially for shiny or glass objects, because it doesn't get confused by the complex physics of light bouncing off near-field LED walls.

In short, this paper gives filmmakers a "digital remote control" for lighting and backgrounds, allowing them to change the mood and setting of a scene after the cameras have stopped rolling, all while keeping the lighting looking perfectly realistic.

Technical Summary: Relightable Gaussian Splatting for Virtual Production Using Image-Based Illumination

Problem Statement

Virtual Production (VP) utilizes LED walls to provide both background imagery and image-based lighting (IBL) for on-set compositing. While this enables in-camera visual effects, it creates a tight coupling between the background and the scene's illumination. This rigidity limits post-production flexibility; any deviation from the desired creative outcome often necessitates costly re-shoots or complex editing.

Conventional inverse rendering (IR) approaches attempt to decouple lighting by jointly estimating 3D geometry and lighting using environment maps. However, these methods face significant limitations in VP contexts:

Far-field Assumption: Standard environment maps assume distant light sources, whereas VP LED walls act as near-field sources, leading to inaccurate lighting effects and reconstruction errors.
Complexity and Portability: IR methods often rely on physically-based rendering (PBR) frameworks, ray-tracing (e.g., NVIDIA OptiX), and robust surface normals, requiring ground-truth depth data or auxiliary geometry. This increases computational cost and reduces portability.
Resolution Constraints: Environment maps compress illumination into spherical representations, often losing the high-resolution detail required for reflections from structured LED sources.

Existing 2D relighting techniques for LED backgrounds cannot synthesize novel 3D views or camera trajectories, failing to meet the needs of modern film production where camera motion and shot composition are often designed in post-production.

Methodology

The authors propose a geometry-implicit relighting framework specifically designed for 3D Gaussian Splatting (GS) scenes captured in VP environments. The core innovation is a parameterization that directly samples known background imagery to model lighting, bypassing the need for environment maps, explicit ray-tracing, or complex material property estimation.

1. Geometry-Free Relighting Pipeline

Instead of recovering surface normals and material properties (albedo, roughness, etc.), the method decomposes the scene into fixed appearance and variable lighting components.

Fixed Appearance ( $\tilde{c}_i$ ): Represents the object's color under a canonical lighting condition.
Variable Lighting ( $f(\cdot|I^B_k)$ ): Models the residual color caused by changes in the background texture $I^B_k$ .

The final color for a Gaussian primitive $i$ under background $k$ is computed as:
$c_{i,k} = \tilde{c}_i + \tilde{\lambda}_i \Delta c(\cdot | I^B_k)$
Where $\tilde{\lambda}_i$ is a learnable intensity/exposure parameter, and $\Delta c$ is the color change derived from sampling the background.

2. Image-Space Light Texture Sampling

To achieve view-dependent lighting without ray-tracing, each Gaussian primitive is assigned learnable UV coordinates ( $\mu_i$ ). These coordinates represent the 2D pixel location on the LED wall's image space where a reflected or refracted ray is expected to intersect.

Mipmap Sampling: To handle texture aliasing and varying distances, the background image is downsampled into $N$ mipmap levels. A learnable scaling parameter ( $\delta_i$ ) determines the appropriate mip-level bounds.
Bilinear Interpolation: The final color residual is acquired by linearly interpolating samples from the lower and upper bound mipmaps.
Implicit Reflections: By learning UV coordinates per Gaussian, the system implicitly captures reflections and refractions based on the LED wall's 2D image space, avoiding the need to compute surface normals or the LED wall's 3D pose.

3. Background Representation and Compositing

Since GS cannot reliably reconstruct the LED wall itself, the framework treats the background as a separate entity:

Masking: During training, Gaussians corresponding to the background are pruned using manually drawn masks ( $Q_j$ ).
Backplate Mesh: Post-training, the original high-resolution background image is projected onto a simple, low-polygon 2D mesh positioned behind the Gaussian scene.
Alpha Blending: The foreground GS render is alpha-blended with the background mesh render. This eliminates light distortions inherent in capturing real LED emitters and allows for arbitrary background replacement without re-capture.

4. Optimization

The system is trained using a panoptic loss combining RGB, DSSIM, and a "canonical loss" ( $L_{canon}$ ). The canonical loss ensures that the fixed appearance component ( $\tilde{c}_i$ ) preserves the object's look under a reference lighting condition, isolating the variable lighting to the residual term.

Key Contributions

Geometry-Free Relighting Pipeline: A VP-specific framework that requires no depth priors, auxiliary geometry, light transport simulation, or custom CUDA code. It decouples lighting from geometry by directly sampling the known background.
Novel GS Parameterization: A method that parameterizes each Gaussian with learnable UV coordinates and intensity values to sample image-based light sources directly in pixel space. This enables photorealistic, view-dependent lighting without environment maps.
New VP Dataset: The introduction of the first publicly available multi-camera dataset for 3D reconstruction and relighting in VP, captured under varying background content and illumination conditions using both miniature and life-size stages.

Experimental Results

The authors evaluated their method against TensoIR (a state-of-the-art IR baseline) and two proxy baselines: TIR (TensoIR adapted to a geometry-free pipeline) and DFR (a deferred rendering approach).

Objective Metrics: The proposed method achieved superior performance across all datasets in PSNR (RGB, Y, CrCb), SSIM, and LPIPS. For example, on Dataset 1, the method achieved a PSNR of 25.65 compared to TensoIR's 16.19.
Efficiency: The approach is highly efficient, requiring <3 GB RAM, <5 GB VRAM, and training times under 2 hours for 18 views. Inference speeds reach 35 FPS.
Ablation Studies: Experiments confirmed that geometric quality is not compromised by the relighting process and that the method benefits from diverse texture training data.
Subjective Study: A study involving 12 participants (including VFX artists) compared manual relighting, the model's raw output, and the model's output assisted by Auxiliary Output Variables (AOVs). The results indicated that the pipeline produces high-quality results comparable to professional manual relighting, with artists noting that the AOVs allowed them to focus on creative edits rather than low-level operations.

Significance and Claims

The paper claims to offer a practical solution for post-production flexibility in Virtual Production. By removing the dependency on specialized capture equipment (like 360° cameras for environment maps) and complex inverse rendering pipelines, the method enables:

Post-Capture Modification: Directors and artists can modify the LED background imagery and associated scene lighting after the shoot without re-shooting.
Cost Reduction: The ability to use standard cameras and non-specialized lighting conditions reduces production costs and complexity.
Workflow Integration: The method supports the rendering of useful Arbitrary Output Variables (AOVs) such as depth, lighting intensity, and unlit renders, facilitating downstream VFX workflows.

The authors emphasize that their approach is tailored specifically to the constraints and opportunities of VP, where the light source is known and controlled, distinguishing it from "in-the-wild" reconstruction methods where illumination is unknown. The framework is presented as a lightweight, portable, and efficient alternative to current heavy-weight inverse rendering techniques.