🤖 AI

Intrinsic decomposition and editing of 3D Gaussian splats

This paper proposes a method for intrinsic decomposition and editing of 3D Gaussian splats by modeling independent Gaussian primitives for albedo and shading, optimizing their disentanglement via data-driven predictions, and enabling texture edits on planar surfaces that maintain plausible lighting across arbitrary viewpoints.

Original authors: Alexandre Lanvin, Jeffrey Hu, Simon Lucas, Adrien Bousseau, George Drettakis

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexandre Lanvin, Jeffrey Hu, Simon Lucas, Adrien Bousseau, George Drettakis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical, ultra-realistic 3D photograph of a room. You can walk around it, look at it from any angle, and it looks perfect. But there's a catch: the colors and shadows are "baked in." If you wanted to change a red wall to blue, you'd have to repaint the whole room, including the shadows cast by the furniture. If you just painted over the shadows, they would look weird and flat.

This paper introduces a new way to build these 3D photographs using "3D Gaussian Splatting" (a fancy term for a collection of tiny, fuzzy 3D dots that create the image). The authors figured out how to separate the color of the objects from the lighting and shadows, allowing you to change the color without messing up the shadows.

Here is how they did it, explained with simple analogies:

1. The Problem: The "Baked Cake"

Usually, when you create a 3D scene, the color (albedo) and the lighting (shading) are mixed together like ingredients in a cake. Once the cake is baked, you can't take the chocolate chips (the texture) out without ruining the cake. Previous methods tried to separate them, but they often used a single set of 3D dots for everything. This is like trying to describe a high-definition texture (like a brick wall) and a soft shadow (like a cloud) using the exact same number of tiny dots. It's a compromise; you either get blurry textures or blocky shadows.

2. The Solution: Three Separate Layers

The authors decided to build the scene using three separate sets of 3D dots instead of one big mix:

The Albedo Layer (The Paint): This layer holds only the pure color and texture of the objects, like a flat painting with no shadows or highlights.
The Shading Layer (The Light & Shadow): This layer holds only the lighting information—where the sun hits and where the shadows fall. It's like a grayscale map of light.
The Residual Layer (The Glare): This captures tricky, shiny effects that change depending on where you stand, like a reflection on a wet floor or a glint on a metal spoon.

By keeping these three "paintings" separate, the system can use more dots for the detailed textures (the bricks) and fewer dots for the smooth shadows, making the whole thing more efficient and realistic.

3. How They Built It: The "Magic Video Editor"

To separate these layers in the first place, the computer needs to guess what the "pure paint" looks like before it knows where the shadows are.

The Guessing Game: The team used a powerful AI (a "video diffusion model") that acts like a super-smart art critic. You feed it a video of the room, and it predicts what the walls and objects would look like if the sun wasn't shining on them.
The Glue: Because the AI's guess might be a little fuzzy or inconsistent, the team used a "depth map" (a 3D blueprint of the room's shape) to glue the AI's predictions into the correct 3D space. This ensures that the "pure paint" layer stays consistent no matter which angle you look at.

4. The Editing Superpower

Once the scene is split into these three layers, editing becomes a breeze.

The Scenario: Imagine you want to change a wooden table to a shiny red metal table.
The Old Way: You would have to repaint the whole 3D model, and the shadows would look wrong because the "paint" and "light" were mixed.
The New Way: You simply grab the "Albedo Layer" (the paint) and swap the wood texture for red metal. The "Shading Layer" (the shadows) stays exactly where it is.
The Result: When you walk around the 3D scene, the table looks like red metal, but the shadows it casts on the floor look perfectly natural, just as if the real sun was hitting a red metal table.

5. How They Keep It Consistent

The tricky part is that if you only paint the table from one angle, the computer might get confused when you look at it from the side. To fix this, the system uses a "proxy plane" (a temporary, invisible flat sheet) to project your paint job onto the 3D dots from multiple angles at once. This ensures that when you walk around the object, the new color stays consistent and doesn't flicker or glitch.

Summary

In short, this paper teaches computers how to take a 3D photo, separate the "object color" from the "lighting," and then let you change the object's color while keeping the lighting and shadows perfectly realistic. It's like having a 3D photo where you can change the wallpaper without having to rebuild the house or repaint the shadows.

Technical Summary: Intrinsic Decomposition and Editing of 3D Gaussian Splats

1. Problem Statement

Intrinsic image decomposition, which separates image colors into diffuse albedo (texture) and shading (lighting/shadows), is a fundamental task for image editing. It allows users to modify object colors and textures without altering the underlying lighting conditions. While this has been extensively studied for 2D images, videos, and Neural Radiance Fields (NeRFs), extending it to 3D Gaussian Splatting (3DGS) presents unique challenges.

Existing approaches to intrinsic decomposition in 3DGS typically augment a single set of Gaussian primitives with intrinsic quantities (albedo, shading, residuals). However, this monolithic approach struggles because different intrinsic components require different spatial resolutions and exhibit distinct characteristics:

Albedo often contains high-frequency details (textures) but lacks geometric cues in uniform regions.
Shading contains sharp edges (hard shadows) but is often smooth elsewhere.
Residuals capture view-dependent effects.

A single set of Gaussians cannot simultaneously adapt to the varying spatial requirements of these layers, leading to suboptimal reconstruction and editing capabilities. Furthermore, editing 3DGS directly often entangles lighting and texture, making it difficult to change a surface's color without distorting shadows or highlights.

2. Methodology

The authors propose a pipeline that decouples the radiance field into three independent sets of Gaussian primitives: one for Albedo ( $G_A$ ), one for Shading ( $G_S$ ), and one for View-Dependent Residuals ( $G_R$ ). The image formation model is defined as:
$I = A \times S + R$
where $I$ is the observed image, $A$ is the albedo, $S$ is the shading, and $R$ is the residual.

2.1 Modeling

Unlike standard 3DGS which uses Spherical Harmonics (SH) for view-dependent color in all primitives, this method tailors the representation:

Albedo and Shading Gaussians ( $G_A, G_S$ ): Use simple RGB triplets ( $c \in \mathbb{R}^3$ ) as they represent view-independent quantities.
Residual Gaussians ( $G_R$ ): Use Spherical Harmonics to model view-dependent effects.
Separate Optimization: Each set is optimized independently, allowing the number and distribution of primitives to adapt to the specific frequency and geometric needs of each layer.

2.2 Reconstruction Pipeline

The reconstruction process occurs in three main stages:

Data-Driven Albedo Estimation:
- The input video is processed by DiffusionRenderer (a video diffusion model) to predict temporally consistent albedo maps for all frames.
- To handle long videos, the authors employ Brick Diffusion, staggering the denoising process across overlapping chunks to ensure smooth transitions and avoid boundary artifacts.
- The video is subsampled to create a multi-view dataset for 3D reconstruction.
Albedo Field Reconstruction ( $G_A$ ):
- The albedo Gaussians are optimized using the predicted albedo images.
- Challenge: Albedo images often lack geometric cues (piecewise constant colors), making 3D placement difficult.
- Solution: The optimization is regularized using monocular depth estimates (from Depth Anything V2) and a scale regularization term to encourage primitives to flatten and form sharp boundaries. This ensures the albedo field adheres to the scene geometry despite the lack of texture in the input albedo maps.
Shading and Residual Reconstruction ( $G_S, G_R$ ):
- With $G_A$ frozen, the shading field is optimized first using the model $I = A \times S$ .
- The residual field is then optimized to capture non-diffuse effects.
- Regularization: To prevent shading from becoming arbitrarily large in dark albedo regions, a "grayness" regularization is applied to encourage shading Gaussians to remain neutral in color.
- Initialization: Residual Gaussians are spawned at locations with high differences between the input image and the composite diffuse render.

2.3 Editing Workflow

The system enables interactive editing of the albedo layer while preserving lighting:

User Input: The user selects a surface and paints a new texture or color on a specific view.
Proxy Surface: Since 3DGS depth is unreliable for reprojection, the system fits a proxy plane to the selected surface using RANSAC on the contributing Gaussians.
Multi-View Synthesis: The edited texture is rendered onto the proxy plane from multiple nearby viewpoints to generate a set of "edited" albedo images.
Re-optimization: The albedo Gaussians ( $G_A$ ) are re-optimized using these synthesized views. New Gaussians are spawned on the proxy plane to capture high-frequency details introduced by the edit.
Result: The final scene can be navigated from arbitrary viewpoints with the new texture, while the original shading and shadows (stored in $G_S$ and $G_R$ ) remain intact.

3. Key Contributions

Separate Gaussian Representations: A novel representation of radiance fields using three distinct sets of Gaussians for albedo, shading, and residuals, allowing each layer to adapt its primitive distribution to the signal's characteristics.
Video Diffusion-Based Reconstruction: A method to reconstruct intrinsic fields from multi-view photographs using a video diffusion prior (DiffusionRenderer) to ensure temporal consistency, regularized by depth and scale constraints to handle the ambiguity of albedo reconstruction.
Intrinsic Editing Workflow: A system that allows users to edit surface textures and colors in a 3D scene while preserving realistic lighting and shadows, achieved by decoupling the intrinsic components and using proxy-based reprojection for multi-view consistency.

4. Results and Evaluation

The method was evaluated on three real-world scenes and one synthetic scene (with ground truth).

Decomposition Quality: On the synthetic scene, the method achieved higher quality albedo reconstruction (PSNR 29.4 vs. 13.7 for GI-GS) compared to state-of-the-art inverse rendering methods (GI-GS, R3GS), though it required more primitives. It successfully separated texture from shading, whereas competing methods often retained shading artifacts in the albedo layer.
Editing Capabilities: In comparative editing tasks against ReCoGS and SplatShop, the proposed method demonstrated superior ability to modify textures without altering shadows.
- ReCoGS preserved texture information when recoloring, failing to remove the original pattern.
- SplatShop overwrote both texture and shadows.
- The proposed method successfully removed the original texture and applied a new one while maintaining the original shadowing and shading effects.
Performance: The method is slower than vanilla 3DGS due to the need to render three separate fields during optimization and inference. However, it achieves real-time rendering speeds (approx. 96–128 FPS) on the final composited result.

5. Significance and Claims

The paper claims that by decoupling albedo and shading into independent Gaussian sets, it is possible to perform physically plausible editing of 3D scenes. The significance lies in:

Multi-View Consistency: Unlike single-view editing methods, the proposed workflow ensures edits remain consistent across different camera angles by leveraging depth-based reprojection and multi-view optimization.
Preservation of Lighting: The separation allows for texture modification without the "washing out" or distortion of shadows that typically occurs when editing entangled radiance fields.
Practicality: The approach avoids the prohibitive computational cost of full inverse rendering for complex scenes by relying on pre-trained diffusion models for the initial decomposition, making it feasible for full-scene editing rather than just isolated objects.

The authors acknowledge limitations, including the "fuzzy" nature of 3DGS making non-planar editing complex, dependence on the quality of diffusion priors (which can be blurry), and the current inability to perform full relighting (changing light sources), focusing instead on albedo modification.