💻 computer science

EAG-PT: Emission-Aware Gaussians and Path Tracing for Diffuse Indoor Scene Reconstruction and Editing

EAG-PT is a novel method that utilizes a unified 2D Gaussian representation with explicit emission separation and a decoupled optimization-rendering pipeline to achieve physically consistent, editable diffuse global illumination in indoor scenes, effectively bridging the gap between the geometric limitations of mesh-based inverse path tracing and the editability issues of traditional radiance fields.

Original authors: Xijie Yang, Mulin Yu, Changjian Jiang, Kerui Ren, Tao Lu, Jiangmiao Pang, Dahua Lin, Bo Dai, Linning Xu

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xijie Yang, Mulin Yu, Changjian Jiang, Kerui Ren, Tao Lu, Jiangmiao Pang, Dahua Lin, Bo Dai, Linning Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photograph of a cozy living room. If you try to move a lamp in the photo, or change the color of the walls, traditional photo-editing tools often fail. The shadows don't move, the light doesn't change color, and the room looks "painted on" rather than real. This is because most modern 3D reconstruction methods treat the entire room like a single, glowing painting. They capture how the light looked at that exact moment and bake it into the image. If you change the scene, the "paint" doesn't know how to react.

On the other hand, some older methods try to build a perfect 3D model (like a digital clay sculpture) and then simulate physics to calculate how light bounces. But these are incredibly fragile; if the digital clay has even a tiny crack or a wobbly edge, the light simulation breaks, and the shadows look wrong.

EAG-PT is a new method that tries to get the best of both worlds. Here is how it works, using simple analogies:

1. The Building Blocks: "Floating Post-it Notes"

Instead of building a solid clay sculpture (a mesh) or a blurry cloud of fog (a neural field), EAG-PT builds the room out of millions of tiny, flat, 2D ovals. Think of these as floating Post-it notes scattered in 3D space.

Why this helps: These notes are flexible. They can be thin, thick, or tilted. They are great at capturing fine details like the legs of a chair or the edge of a lamp without needing a rigid, perfect mesh.

2. The Big Idea: Separating the "Sun" from the "Wall"

The core problem with previous methods is that they treat the light source (the lamp) and the object (the wall) as the same thing.

The Old Way: Imagine a wall that is itself glowing. If you try to turn off the lamp, the wall stays glowing because the light is baked into the wall's texture.
The EAG-PT Way: It acts like a smart detective. It looks at the room and says, "Okay, that glowing bulb is a Light Source (an emitter), and that chair is just a Reflector."
It creates a special mask to separate the "glowing" parts from the "non-glowing" parts. This allows the system to know exactly what is emitting light and what is just reflecting it.

3. The Process: Two Stages of Learning

The method learns about the room in two steps:

Stage 1: The Snapshot (Radiant Reconstruction)
First, it takes all the photos and builds the "Post-it note" room. It figures out where every note is, what color it is, and which ones are the actual light bulbs. At this point, it's just capturing what the room looked like when the photos were taken.
Stage 2: The Physics Lesson (Material Recovery)
Now, the system asks: "If I turn off that light bulb, what color would the wall actually be?"
It uses a technique called Path Tracing. Imagine shooting a laser beam from the camera. The beam hits a wall, bounces off, hits a chair, bounces again, and finally hits a light bulb. By tracing these bounces, the system calculates the true color of the materials (albedo), stripping away the specific lighting of the original photo. It learns the "intrinsic" color of the objects.

4. The Magic: Editing the Room

Once the system knows which parts are lights and which parts are just objects, you can edit the scene:

Move a light: The system knows the light is a separate object. It moves the "Post-it note" representing the light.
Change a color: You can tell the system, "Make the chair red."
Add a new ball: You can drop a glowing ball into the scene.

Because the system understands the physics of light (how it bounces off surfaces), it instantly recalculates the shadows and reflections. The light from your new glowing ball will bounce off the floor and the walls, creating realistic shadows and "glow" that matches the new setup.

5. The Result: Realism Without the Cracks

The paper shows that this method produces results that look much more natural than previous techniques.

Compared to "Painted" methods: It doesn't just paste a new image; it simulates how light actually behaves, so shadows move correctly and reflections change.
Compared to "Clay" methods: Because it uses the flexible "Post-it notes" instead of a rigid 3D mesh, it doesn't suffer from jagged edges or broken geometry. It preserves fine details like thin chair legs that usually get lost in mesh-based models.

Summary

EAG-PT is like giving a 3D room a "physics brain." It separates the lights from the furniture, learns the true colors of the furniture, and then uses a light-bouncing simulation to let you rearrange the room, change the lights, or add new objects, all while keeping the shadows and reflections looking perfectly real. It bridges the gap between easy-to-edit 3D models and the complex physics of real-world light.

1. Problem Statement

Existing 3D reconstruction methods for indoor scenes, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), achieve high visual fidelity for novel view synthesis. However, they suffer from a critical limitation: they treat the entire scene as uniformly radiant.

Baked Illumination: In these models, lighting and shadows are implicitly encoded into the outgoing radiance. Consequently, editing the scene (e.g., changing light sources, moving objects, or modifying materials) fails to produce physically consistent changes in illumination, shadows, or inter-reflections.
Limitations of Current Solutions:
- Radiance Field Extensions: Methods adding limited reflection modeling (e.g., single-bounce) still rely on cached radiance from the original capture, failing to support complex global illumination (GI) changes after editing.
- Mesh-Based Inverse Rendering: Physically based methods using meshes and path tracing can model correct light transport but require highly accurate geometry. Real-world indoor scenes often contain clutter and fine details that are difficult to reconstruct as clean meshes, leading to artifacts and instability.

Goal: The authors aim to create a framework that enables physically based reconstruction and editing of diffuse indoor scenes, allowing for realistic relighting, object insertion, and material modification with coherent global illumination, without relying on explicit mesh reconstruction.

2. Methodology: EAG-PT

The proposed Emission-Aware Gaussians and Path Tracing (EAG-PT) framework utilizes a unified 2D Gaussian representation to bridge the gap between efficient radiance field reconstruction and physically accurate light transport. The pipeline consists of three main stages:

A. Scene Representation: 2D Gaussians

Instead of 3D volumetric Gaussians or meshes, EAG-PT uses 2D Gaussians as elliptical surface elements embedded in 3D space.

Ray Tracing: The method reformulates 2D Gaussians as traceable primitives. Rays intersect with these Gaussians to accumulate quantities like normal vectors, distance, radiance, emissive terms, and albedo via alpha blending.
Advantage: This representation offers a favorable trade-off between geometric accuracy (better than volumetric 3DGS) and appearance quality, avoiding the mesh triangulation artifacts common in inverse rendering.

B. Stage 0: Radiant Scene Reconstruction

The scene is first reconstructed as a "radiant scene" (capturing the appearance at the time of capture) using differentiable 2D Gaussian ray tracing.

Emission-Aware Decomposition: A key innovation is the explicit separation of emissive (light sources) and non-emissive (reflective surfaces) components using 2D emission masks.
Loss Functions: The optimization minimizes a weighted sum of:
- Radiance Loss: Reconstructs the color of the scene.
- Geometry Loss: Supervises surface normals and depth variations using monocular normal maps (e.g., StableNormal) to ensure accurate geometry for light bouncing.
- Emission Loss: Ensures the emissive component matches the provided emission masks.

C. Stage 1: Diffuse Material Recovery

Once the radiant scene and geometry are established, the method recovers the intrinsic material properties (diffuse albedo) of non-emissive surfaces.

Single-Bounce Approximation: It uses a radiance cache (the radiance recovered in Stage 0) to approximate incident light.
Differentiable Rendering: By performing a single bounce of light from the cache to the surface, the system optimizes the diffuse albedo ( $\rho$ ) of each Gaussian to match the observed multi-view images. This decouples the recovery of material properties from the complex multi-bounce light transport.

D. Path Tracing for Scene Editing

After editing the scene (e.g., moving a chair, changing a lamp color, or inserting a new object), the system switches to multi-bounce path tracing.

Dynamic Light Transport: Instead of relying on the obsolete radiance cache from the original capture, the system traces rays from the camera, bouncing them off the 2D Gaussians until they hit an emissive surface.
Global Illumination: This process naturally computes diffuse global illumination, including soft shadows, inter-reflections, and color bleeding, ensuring physical consistency.
Light Baking: To enable real-time visualization of edited scenes, the high-quality path-traced results can be "baked" back into the 2D Gaussian radiance parameters, allowing for fast, 0-bounce rendering that preserves the GI effects.

3. Key Contributions

Unified 2D Gaussian Representation: A novel approach that uses 2D Gaussians as a single representation for geometry, appearance, and light transport, eliminating the need for mesh conversion and preserving fine geometric details.
Emission-Aware Decomposition: Explicitly separating light sources from reflective surfaces during reconstruction, which is critical for enabling physically correct scene editing.
Decoupled Optimization Strategy: Separating the reconstruction of the radiant scene (Stage 0) and the recovery of diffuse materials (Stage 1) from the final path-traced rendering. This allows for efficient training while supporting high-fidelity multi-bounce rendering during editing.
Physically Consistent Editing: The ability to perform complex edits (relighting, object insertion, material changes) and generate photo-realistic renders with coherent global illumination, shadows, and inter-reflections.

4. Experimental Results

The authors evaluated EAG-PT on both synthetic and real-world indoor datasets (including f-classroom, lectureroom, and Eyeful Tower).

Quantitative Performance:
- On synthetic scenes with ground-truth relighting, EAG-PT (Path Tracing) achieved significantly lower FLIP and LPIPS errors compared to 0-bounce and 1-bounce baselines after editing.
- On real scenes, EAG-PT outperformed the state-of-the-art mesh-based inverse path tracing method (FIPT) in novel-view synthesis quality (higher PSNR, lower LPIPS/FLIP).
Qualitative Improvements:
- Geometry: EAG-PT preserved fine structures (e.g., chair legs, lamp arms) that mesh-based methods often fail to capture or introduce triangulation artifacts.
- Editing: The method successfully generated realistic shadows and inter-reflections when inserting glowing balls, duplicating objects with modified materials, or changing lighting conditions.
- Efficiency: While path tracing is computationally expensive, the "Light Baking" strategy reduced render times by orders of magnitude, enabling interactive navigation of edited scenes.
Storage Efficiency: The unified Gaussian representation required significantly less storage (approx. 33 MB) compared to mesh-based baselines which required hundreds of MBs for geometry and materials.

5. Significance and Impact

EAG-PT represents a significant step forward in inverse rendering and 3D scene editing:

Practical Applications: It enables realistic workflows for interior design (visualizing furniture and lighting changes), XR content creation (generating editable 3D assets), and Embodied AI (training robots in physically consistent, relightable environments).
Bridging the Gap: It successfully bridges the gap between the efficiency of Gaussian Splatting and the physical correctness of path tracing, solving the "baked illumination" problem that has plagued radiance field methods.
Future Direction: The work highlights the potential of 2D Gaussians as a robust primitive for complex light transport tasks, moving beyond simple view synthesis toward fully editable, physically grounded digital twins.

Limitations: The current method assumes diffuse reflectance (handling specular materials is future work) and relies on manual or semi-automatic emission masks for light source separation. Additionally, real-time path tracing remains computationally heavy, though light baking mitigates this for visualization.