Technical Summary: LuxRemix

Problem Statement

Controlling lighting in indoor scenes is fundamental to photography, cinematography, and virtual production. However, once an image or 3D reconstruction is captured, lighting is typically "baked" into the appearance, making post-capture adjustments difficult. Existing approaches face significant limitations:

• Data-driven methods often require dense, controlled multi-lighting captures that are impractical for real-world scenarios.
• Optimization-based inverse rendering is computationally intensive and often fails to produce plausible results when lighting conditions change significantly.
• Prior-driven methods (using diffusion models) have shown promise but largely focus on objects, portraits, or simple scenes under distant illumination. They struggle with the complex, spatially varying illumination of indoor scenes with multiple near-field sources.
• Single-image methods lack 3D consistency across multiple viewpoints, while multi-view methods often lack fine-grained control over individual light sources or require controlled capture setups.

The core challenge is to enable interactive, fine-grained editing of individual light sources (switching on/off, adjusting intensity and color) in complex indoor scenes from a single multi-view capture, while maintaining 3D consistency and real-time performance.

Methodology

LuxRemix addresses this challenge through a three-stage pipeline that combines generative image decomposition, multi-view harmonization, and 3D Gaussian splatting.

1. Synthetic Data Generation

To train the necessary models, the authors generated a large-scale synthetic dataset comprising over 12,000 procedurally generated indoor scenes. Using Infinigen, they augmented these scenes with up to six controllable light sources (ceiling, wall, floor, table lamps, and environment lighting).

• Rendering: Scenes were rendered using Blender's Cycles renderer in two modes: fully lit and multiple "One-Light-At-a-Time" (OLAT) configurations where all but one light are switched off.
• Efficiency: Instead of pre-rendering all perspective views, scenes were rendered as 360° equirectangular HDR images. This allowed for dynamic sampling of diverse camera viewpoints during training, significantly reducing storage and rendering costs while maintaining data diversity.
• Ground Truth: The dataset includes ground-truth masks for light sources and their corresponding OLAT renderings, enabling supervised training for decomposition.

2. Single-Image Lighting Decomposition (LuxRemix-SV)

The first stage involves decomposing a single input image into an ambient component and multiple individual OLAT components.

• Model Architecture: The method fine-tunes a pretrained image-editing Diffusion Transformer (DiT) using Low-Rank Adaptation (LoRA).
• Conditioning: The model is conditioned on the input image, a text prompt (e.g., "switch off all lights except the selected one"), and a spatial light selection mask processed via a single-layer MLP.
• Tasks: The model learns two primary tasks:
  1. OLAT Decomposition: Isolating a single light source while turning off others.
  2. Light Removal: Turning off a specific light while keeping others active (treating the result as ambient).
• HDR Reconstruction: To handle high dynamic range, the model is prompted with three brightness levels (high, medium, low). The resulting images are merged using exposure bracketing techniques to reconstruct full HDR OLAT components.

3. Multi-View Lighting Harmonization (LuxRemix-MV)

Since single-image decomposition can produce inconsistent results across different viewpoints, a second stage propagates these decompositions consistently across all views.

• Approach: Inspired by multi-view diffusion methods (e.g., CAT3D, SEVA), this stage uses a pretrained multi-view diffusion U-Net.
• Input: The network takes the original multi-view images and sparse, partially decomposed views (from the SV stage), along with Plücker ray embeddings and reference masks.
• Process: The U-Net is fully fine-tuned to propagate the lighting decomposition from the sparse reference views to all other views, ensuring geometric and photometric consistency. This step resolves inconsistencies in shadows and lighting appearance that would arise from processing views independently.

4. Real-Time Relightable 3D Representation

The final stage encodes the decomposed, consistent lighting into a 3D Gaussian Splatting representation for real-time interaction.

• Representation: Building on standard 3D Gaussian Splatting, each Gaussian is extended with per-light HDR RGB coefficients. These coefficients store the appearance contribution of each light source (including ambient) separately.
• Training:
  1. Pretraining: A standard 3DGS model is trained on the original multi-view images to establish geometry.
  2. Light Fitting: Geometric parameters are frozen. Per-light RGB coefficients are introduced and optimized using the decomposed multi-view images.
• Optimization: The training minimizes an L1 loss for individual light renderings, a composition loss to ensure recombined lights match the original input, and a spatial smoothness loss (based on K-Nearest Neighbors) to prevent high-frequency noise and artifacts.
• Rendering: At runtime, the final image is generated by linearly combining the per-light contributions based on user-controlled intensity and color settings, enabling real-time interactive manipulation from any viewpoint.

Key Contributions

1. Single-Image Decomposition Model: A generative model that factorizes complex indoor lighting into individually controllable light sources (OLAT) and ambient light, allowing for independent manipulation of state, chromaticity, and intensity.
2. Multi-View Lighting Harmonization: A novel method to propagate lighting decompositions across multiple views, ensuring 3D consistency and resolving view-dependent artifacts.
3. Relightable 3D Gaussian Splatting: An encoding of decomposed lighting into a 3D Gaussian representation that enables real-time, interactive manipulation of individual near-field light sources from novel viewpoints.
4. Large-Scale Synthetic Dataset: The release of a dataset containing over 12,000 indoor scenes with ground-truth per-light decompositions, facilitating future research in controllable scene lighting.

Results and Evaluation

The authors evaluated LuxRemix on both synthetic and real-world datasets:

• Quantitative Performance: On a synthetic test set of 30 scenes, LuxRemix-SV outperformed baselines (ScribbleLight, Qwen-Image) in PSNR, SSIM, and LPIPS metrics. The multi-view harmonization (LuxRemix-MV) significantly improved consistency compared to independent single-view processing.
• Qualitative Results: The method demonstrated the ability to realistically isolate light sources, switch them on/off, and adjust their colors. It successfully maintained 3D consistency across views, avoiding the flickering and shadow inconsistencies seen in single-view approaches.
• Real-World Application: The method was tested on real-world captures (e.g., from Zip-NeRF and VR-NeRF), showing convincing results in decomposing and remixing lighting in complex indoor environments.
• Real-Time Performance: The 3D Gaussian representation enabled real-time interactive relighting, a capability lacking in many prior inverse rendering or NeRF-based approaches.

Significance and Claims

The paper claims to present the first method for multi-view lighting decomposition and remixing specifically for indoor scenes from a single multi-view capture. Its significance lies in bridging the gap between the fine-grained control of single-image decomposition and the 3D consistency of multi-view methods.

By leveraging the rich priors of modern diffusion models for decomposition and the geometric constraints of multi-view data for consistency, LuxRemix enables flexible post-capture editing of individual light sources without the need for controlled multi-lighting captures. The authors position this work as a foundation for future research in controllable multi-view scene lighting, with potential applications in interactive scene editing, virtual production, and immersive experiences where photorealistic, fine-grained lighting control is essential.

The authors modestly acknowledge limitations, noting that the models are trained on static synthetic indoor scenes and may not generalize perfectly to outdoor or dynamic scenes. Additionally, the training data's bias towards conical light spreads can affect decomposition accuracy for diffuse lighting, and the current model does not support editing distant global illumination via HDRIs.