💻 computer science

Lighting-Consistent Object Transfer Across Radiance Fields

This paper presents a novel 3D object transfer framework that leverages a diffusion model trained on a heterogeneous dataset to harmonize lighting inconsistencies when compositing objects between 3D Gaussian Splatting scenes, followed by a post-optimization step to consolidate the results into a high-quality, visually consistent 3D representation.

Original authors: Nicolás Violante, George Kopanas, Linus Franke, Julien Philip, George Drettakis

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nicolás Violante, George Kopanas, Linus Franke, Julien Philip, George Drettakis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a high-quality photo of a shiny red sports car parked in a sunny garage. Now, imagine you want to move that car into a photo of a dark, rainy alleyway. If you just cut the car out of the first photo and paste it into the second (like using "Copy and Paste" in a photo editor), it will look fake. The car will still look like it's in the sun, missing the wet reflections on the pavement and the shadows cast by the rain. It won't "belong" there.

This paper introduces a new tool called DOT3D (Diffusion Object Transfer in 3D) that solves this problem for 3D scenes. Instead of just pasting an object, it "re-dresses" the object to fit its new environment perfectly.

Here is how it works, broken down into simple steps:

1. The Problem: The "Uncanny Valley" of Lighting

When you take a 3D object (like a 3D model of a chair) from one scene and drop it into another, the lighting doesn't match. The object might be too bright, have the wrong shadows, or lack reflections. In the real world, light bounces off walls and floors; in a simple digital paste, it doesn't. This makes the object look like a sticker rather than a real part of the room.

2. The Solution: A "Magic Lighting Artist"

The authors built a system that acts like a highly skilled digital lighting artist. They used a type of AI called a Diffusion Model. Think of this model as an artist who has seen millions of photos of objects in different lighting conditions.

The Input: You give the AI a "naive" composite (the object pasted in with bad lighting) and a mask (a stencil showing exactly where the object is).
The Magic: The AI looks at the new room (the target scene) and asks, "How would this object look if it were actually in this room?" It then redraws the object, adding the correct shadows, reflections, and color tones to match the new surroundings.

3. The Training: Teaching the Artist

To teach this AI artist, the researchers didn't just use one type of data. They created a "training diet" made of three different ingredients:

Synthetic Data: Computer-generated scenes (like a video game) where they knew the exact lighting rules.
Generated Data: Images created by another AI to show objects in weird, creative lighting.
Real Data: Photos of real objects in real rooms.

By mixing these three, the AI learned to handle everything from shiny metal cars to soft fabric chairs in all kinds of environments.

4. The 3D Challenge: Keeping it Consistent

Here is the tricky part: The AI is great at fixing one single photo (a 2D image). But a 3D scene is made of hundreds of photos taken from different angles. If the AI fixes the photo from the left, but a different photo from the right looks slightly different, the final 3D object will look wobbly or broken when you walk around it.

To fix this, the authors added a second step called Multi-View Consolidation.

Imagine you have 200 different photos of the car in the rainy alley, all fixed by the AI.
The system then runs a "post-optimization" process. It's like a quality control inspector that looks at all 200 photos at once and smooths out the differences. It ensures that the shadow on the left matches the shadow on the right, so when you look at the 3D scene, the car looks solid and consistent from every angle.

5. The Result

The final result is a 3D scene where the transferred object looks like it has always been there. It casts the right shadows, reflects the new environment, and fits the lighting perfectly.

In short:

Old Way: Cut and paste. The object looks like a sticker with the wrong lighting.
DOT3D Way: Cut, paste, and then use a smart AI to "re-light" the object and smooth out the 3D view so it looks real.

The paper claims this method works better than previous attempts, which often failed to create realistic shadows or reflections, leaving the object looking fake. They tested it on real-world scenes (like gardens and train stations) and synthetic scenes, showing that the object blends in naturally.

Technical Summary: Lighting-Consistent Object Transfer Across Radiance Fields (DOT3D)

Problem Statement
3D Gaussian Splatting (3DGS) enables efficient capture and real-time rendering of real scenes. However, editing these representations is challenging because materials and lighting are entangled at the time of capture. While compositing objects from one captured scene into another has applications in VFX, architecture, and marketing, naive transfer (copying Gaussian primitives) results in unrealistic outputs. The primary failure mode is lighting inconsistency: the transferred object lacks shadows, reflections, and highlights appropriate for the target environment, making it appear pasted rather than integrated. Existing solutions either require costly manual 3D modeling and lighting estimation or are limited to 2D image harmonization, failing to produce multi-view consistent 3D results.

Methodology: DOT3D
The authors propose DOT3D (Diffusion Object Transfer in 3D), a pipeline that extracts an object from a source 3DGS scene, places it in a target 3DGS scene, and harmonizes the lighting to match the new environment. The process consists of three main stages:

Interactive Object Composition:
- Segmentation: The source scene is reconstructed in 3DGS. To extract an object, the method assigns a binary classification feature ( $d=1$ ) to each Gaussian primitive. These features are optimized using a binary cross-entropy loss against 2D masks (generated by BiRefNet) to segment the object in 3D.
- Transfer: Users interactively place the segmented object into a target 3DGS scene, adjusting position, orientation, and scale. This creates an initial "naive composite" where the object retains its original lighting, resulting in visual inconsistencies.
Single-View Harmonization (Diffusion Model):
- The core of the method is a fine-tuned diffusion model (based on FLUX.1-schnell) that transforms the inconsistent composite images into harmonized views.
- Input: The model takes a render of the naive composite, the object's binary mask, and the composite image itself.
- Training Data: To train the model, the authors constructed a heterogeneous dataset comprising three sources:
  - Synthetic Data: Blender-rendered indoor scenes with ground-truth global illumination, where objects are rendered under mismatched lighting and pasted onto backgrounds.
  - Generated Data: Images created by FLUX, processed with a custom object removal network (trained on synthetic data) to generate backgrounds, and relit using DiffusionRenderer to create inconsistent composites.
  - Real Data: Subsampled images from the ORIDa dataset, augmented with relighting via DiffusionRenderer.
- Process: The model operates in the latent space of a variational autoencoder, regressing the velocity to transform a noisy latent into a harmonized image that includes correct shadows, reflections, and lighting integration.
Multi-View Consolidation (3DGS Post-Optimization):
- Since the diffusion model processes views independently, the resulting harmonized images may lack 3D consistency.
- The authors propose a post-optimization step to consolidate these views back into a single 3DGS representation.
- Optimization: Starting from the naive composite 3DGS, the method optimizes only the colors (all spherical harmonic bands) of the Gaussian primitives using a perceptual loss (LPIPS).
- Constraints: View-dependent effects from the source scene are reset (higher SH bands set to zero) to prevent artifacts from unseen viewpoints. Geometry and opacity remain fixed. This step ensures the final scene is multi-view consistent while preserving the high-fidelity harmonization generated by the diffusion model.

Key Contributions

A Specialized Diffusion Model: A fine-tuned model for 2D lighting harmonization of object insertion, trained on a heterogeneous dataset combining synthetic, generated, and real data.
3DGS Post-Optimization: A consolidation strategy that transforms independently harmonized 2D views into a consistent 3DGS scene by optimizing Gaussian colors with a perceptual loss, avoiding the need for iterative dataset updates.
Interactive Pipeline: A complete workflow allowing users to extract, place, and harmonize objects across 3DGS scenes with consistent lighting, shadows, and reflections.

Results and Evaluation
The method was evaluated on a variety of real-world datasets (Mip-NeRF-360, DL3DV, Ref-NeRF, etc.) and high-quality synthetic scenes with ground truth.

Qualitative Performance: Compared to 2D methods (LBM, MV-CoLight, Nano Banana) and 3D inverse rendering approaches (Gaussian Shader, GS-IR, 3DGS-DR), DOT3D produces significantly more realistic results. Previous methods often fail to generate contact shadows, suffer from color bleeding, or produce overly saturated surfaces. DOT3D successfully integrates objects with plausible shadows and reflections.
Quantitative Performance: On synthetic scenes with ground truth, DOT3D outperforms inverse rendering baselines across all metrics (PSNR, SSIM, LPIPS, FID, KID). For instance, it achieves a PSNR of 22.53 compared to 18.66 for the next best method (Gaussian Shader).
Consistency: Ablation studies on illumination consistency across different object poses demonstrate that the method maintains consistent lighting in novel views.
Ablations: The study confirms that the synthetic Blender dataset is the most critical component for training the diffusion model. Furthermore, the single-step post-optimization approach is shown to be more effective and efficient than iterative dataset update strategies.

Significance and Claims
The paper claims that DOT3D provides a "visually compelling" solution that makes object transfer between 3DGS scenes easy to use and significantly improves quality over previous methods. By leveraging the priors of diffusion models for 2D harmonization and a lightweight post-optimization for 3D consistency, the method addresses the difficult inverse problem of lighting harmonization without requiring explicit geometry reconstruction or complex lighting estimation. The authors note that while the method produces "plausible" results rather than absolute physical accuracy, it enables realistic asset reuse across different captured environments. The work is presented as a step forward in making 3DGS representations editable for real-world applications like VFX and interior design.

1. The Problem: The "Uncanny Valley" of Lighting

2. The Solution: A "Magic Lighting Artist"

3. The Training: Teaching the Artist

4. The 3D Challenge: Keeping it Consistent

5. The Result

More like this