💻 computer science

GOR-IS: 3D Gaussian Object Removal in the Intrinsic Space

This paper introduces GOR-IS, a novel framework for 3D Gaussian object removal that decomposes scenes into intrinsic components to explicitly model light transport and perform inpainting in the material and lighting domains, thereby achieving superior physical consistency and visual coherence compared to existing methods.

Original authors: Yonghao Zhao, Yupeng Gao, Jian Yang, Jin Xie, Beibei Wang

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yonghao Zhao, Yupeng Gao, Jian Yang, Jin Xie, Beibei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical 3D photograph of a room. In this room, there is a shiny, glossy table and a vase sitting on it. Now, imagine you want to use a digital eraser to remove the vase.

The Problem with Old Erasers
Previous methods of doing this were like using a blunt crayon. When they erased the vase, they often left behind a "ghost." Because the table was shiny, the vase was reflecting in it. Old erasers would remove the vase but leave the reflection of the vase on the table, making it look like a floating, invisible ghost.

Furthermore, these old methods assumed that the color of the background behind the vase was the same no matter which angle you looked at it from. But in the real world, shiny surfaces change color depending on your viewpoint (like how a car paint job looks different from the front vs. the side). Old erasers got confused by this, resulting in blurry or "ghostly" patches where the vase used to be.

The New Solution: GOR-IS
The paper introduces a new framework called GOR-IS (3D Gaussian Object Removal in the Intrinsic Space). You can think of this as a "smart, physics-aware eraser."

Here is how it works, using simple analogies:

1. The "Deconstructed Lego" Approach (Intrinsic Space)

Imagine the 3D scene isn't just a solid block of pixels, but a collection of Legos that have been taken apart into their raw ingredients: Material (what the object is made of) and Lighting (how light hits it).

Old way: Tries to erase the object by just painting over the pixels.
GOR-IS way: It separates the "vase material" from the "light hitting the vase." It understands that the vase is a physical object, but the reflection on the table is just light bouncing off. By separating these, it knows exactly what to delete (the vase and its light reflection) and what to keep (the table's material).

2. The "Mirror Detective" (Global Lighting)

When you remove the vase, you also need to remove its reflection in the shiny table.

GOR-IS acts like a detective that traces the path of light. It sees, "Ah, this shiny spot on the table is actually a reflection of the vase."
Because it understands the physics of light, it removes the reflection along with the vase. This prevents the "ghost" effect and makes the scene look physically real.

3. The "Universal Paint" (Intrinsic Inpainting)

When the vase is gone, there is a hole in the 3D scene. You need to fill that hole with the background that was hidden behind the vase.

The Challenge: If you just paint the hole based on one photo, the paint might look wrong when you walk around the 3D scene (because of the shiny, angle-changing surfaces).
The GOR-IS Fix: Instead of painting the "color" of the hole, it paints the "material properties" (like how rough or shiny the surface is). Since the material of a table doesn't change just because you move your head, this "Universal Paint" stays consistent from every angle. It fills the hole with the correct texture, ensuring the background looks sharp and real from any viewpoint.

4. The "Shadow Shield" (Lighting-Aware Masking)

Sometimes, even after removing the object, a faint blur or a leftover reflection might remain.

GOR-IS has a special "shield" that detects these leftover reflections. It says, "This blurry spot is still a reflection of the removed object; it doesn't belong here." It then wipes that away, ensuring the final image is clean and free of artifacts.

The Result

The authors tested this on both computer-generated scenes and real-world photos. They found that their method:

Removes objects and their reflections perfectly, without leaving "ghosts."
Fills in the background so it looks sharp and consistent from every angle, even on shiny surfaces.
Is significantly better at looking "real" (physically consistent) than previous methods.

In short, GOR-IS doesn't just delete an object; it understands the physics of the room, removes the object and its light effects, and seamlessly repaints the background using the room's true materials, resulting in a flawless 3D edit.

1. Problem Statement

The paper addresses the challenge of 3D object removal (inpainting) within scenes reconstructed using 3D Gaussian Splatting (3DGS). While existing methods have made progress in geometric completion and appearance consistency, they suffer from two critical limitations:

Neglect of Global Lighting Effects: Most methods fail to remove reflections cast by the target object on surrounding glossy surfaces. This leads to physically implausible results where an object is gone, but its reflection remains.
View-Dependence Assumption: Existing approaches often assume that inpainted colors are view-independent (Lambertian). This assumption breaks down in scenes with non-Lambertian surfaces (e.g., glossy, metallic), where appearance changes with the viewing angle, leading to blurring, ghosting, and visual artifacts.

2. Methodology: GOR-IS Framework

The authors propose GOR-IS (3D Gaussian Object Removal in the Intrinsic Space), a framework that decomposes the scene into intrinsic components (material and lighting) to perform physically consistent removal. The framework consists of three main stages:

A. Material and Lighting Decoupling

To handle global lighting consistency, the method extends standard 3DGS by incorporating Physically Based Rendering (PBR) material properties.

Representation: Each Gaussian primitive is extended with material properties: diffuse reflection ( $d$ ), Fresnel ( $f_0$ ), and roughness ( $r$ ), alongside a label property ( $l$ ) to identify the target object.
Explicit Light Transport: A global illumination model is introduced to separate direct and indirect radiance.
- Direct Radiance: Modeled via an optimizable environment map.
- Indirect Radiance: Computed using a 3DGS ray tracer.
Glossy Reflection Modeling: To balance realism and efficiency, the authors propose a screen-space filter. Instead of expensive multi-bounce ray tracing, they compute an ideal specular reflection and apply a mipmap-based filter guided by surface roughness to simulate realistic glossy reflections.

B. Intrinsic-Space Inpainting

To solve the non-Lambertian challenge, the inpainting process is shifted from the color domain to the intrinsic space (material and lighting domains).

Material Inpainting: Instead of inpainting RGB images directly (which are view-dependent), the method inpaints view-independent material maps (diffuse, Fresnel, roughness, normals). Since material properties do not change with the viewing angle, this ensures consistent completion of non-Lambertian surfaces across all viewpoints.
Lighting-Aware Masking: To prevent artifacts caused by retained reflections, the system uses the explicit light transport model to trace reflections originating from the target object. It generates a lighting-aware mask that identifies and suppresses reflection-affected regions during the loss computation, ensuring the inpainting model does not try to "fill in" a reflection that should be removed.

C. Training Strategy

The framework operates in two stages:

Scene Decomposition: Optimizes Gaussian primitives and the environment map using multi-view images to establish accurate material/lighting decomposition and light transport.
Inpainting: Fixes the environment map and optimizes the scene using 2D inpainting results (generated by a model like LaMa) as supervision. The loss function is split:
- Appearance Loss: Applied to Lambertian (rough) regions.
- Material Loss: Applied to non-Lambertian (glossy) regions using the inpainted material maps.
- Masking: Excludes target objects and their reflections from supervision to avoid contamination.

3. Key Contributions

Intrinsic Space Framework: A novel approach that decomposes 3D scenes into intrinsic material and lighting components, enabling physically consistent object removal.
Global Lighting Consistency: The first object removal method to explicitly model and remove reflections of the target object on surrounding glossy surfaces, ensuring physical plausibility.
Non-Lambertian Inpainting: An intrinsic-space inpainting module that operates on view-independent material properties, effectively solving the artifact issues (blurring/ghosting) associated with view-dependent surfaces.
Efficient Glossy Modeling: A screen-space filtering strategy that approximates complex glossy reflections with low computational overhead compared to full path tracing.

4. Experimental Results

The method was evaluated on synthetic (GOR-IS-Synthetic) and real-world (GOR-IS-Real) datasets containing strong global lighting effects, as well as the SPIn-NeRF dataset (Lambertian scenes).

Quantitative Performance: GOR-IS outperforms state-of-the-art methods (including 3DGIC, AuraFusion360, GS Grouping, and GScream) by:
- 13% improvement in perceptual similarity (LPIPS).
- 2dB improvement in Peak Signal-to-Noise Ratio (PSNR).
Visual Quality: Visual comparisons demonstrate that GOR-IS successfully removes both the object and its reflections, whereas baselines often leave behind "ghost" reflections or produce blurry artifacts on glossy surfaces.
Generalization: The method maintains competitive performance on Lambertian scenes (SPIn-NeRF), proving it does not sacrifice performance for standard scenes.
Ablation Studies: Confirm that both the explicit light transport modeling and the intrinsic-space inpainting module are essential for achieving high-quality results.

5. Significance

GOR-IS represents a significant leap forward in 3D scene editing. By moving beyond simple color inpainting to intrinsic decomposition, it addresses the fundamental physical constraints of light transport in 3D scenes.

Physical Realism: It bridges the gap between 3D editing and physical rendering, ensuring that edited scenes obey the laws of optics (e.g., reflections disappear when the object is removed).
Robustness: It provides a robust solution for editing complex, real-world scenes with shiny or reflective materials, which have previously been a major failure point for 3D inpainting algorithms.
Future Impact: This approach paves the way for more advanced 3D content creation, virtual reality, and embodied AI applications where physically consistent scene manipulation is critical.

Limitations: The current method does not explicitly model diffuse-related global illumination (soft shadows from indirect light) and struggles with multi-bounce reflections between multiple non-Lambertian surfaces due to computational constraints, leaving these as areas for future work.