LumiMotion: Improving Gaussian Relighting with Scene Dynamics

1. Problem Statement

The paper addresses the challenge of inverse rendering in 3D reconstruction, specifically the task of disentangling material properties (albedo, roughness) from illumination (shadows, direct/indirect light) to enable realistic relighting of a scene under novel lighting conditions.

• Limitations of Existing Methods: Current Gaussian Splatting-based inverse rendering methods (e.g., IRGS, R-3DGS) primarily target static scenes. They often struggle in real-world conditions because they cannot distinguish between a dark surface caused by a shadow (illumination) and a dark surface caused by the material itself (albedo). Without motion, these factors are entangled, leading to "baked-in" shadows in the albedo and poor relighting capabilities.
• The Gap: Existing dynamic methods often rely on human pose priors, known lighting conditions, or unnatural setups (OLAT). There is a lack of general-purpose methods that can perform inverse rendering on arbitrary dynamic scenes under unknown, natural lighting.

2. Methodology: LumiMotion

LumiMotion is a two-stage framework that leverages scene dynamics (motion) as a supervisory signal to separate lighting from material. It operates on 2D Gaussian Splatting (2DGS), which provides accurate surface normals crucial for rendering equations.

Stage 1: Dynamic Geometry Learning

The goal is to reconstruct the scene geometry and model its dynamics while separating static and dynamic elements.

• Representation: The scene is represented by a collection of 2D Gaussians (flat disks).
• Deformation Network: A Multi-Layer Perceptron (MLP) predicts transformations (position $\Delta\mu$, rotation $\Delta r$, and color $\Delta c$) for each Gaussian based on a timestep $t$.
• Static-Dynamic Separation (Key Innovation):
  • To prevent the model from confusing moving shadows with moving objects, the authors introduce a per-Gaussian binary variable $P$ indicating whether a Gaussian is static or dynamic.
  • $P$ is sampled using a Binary Concrete distribution (a continuous relaxation of a Bernoulli distribution) to allow for differentiable training.
  • The final position and rotation are computed as $\mu' = \mu + \tilde{P}\Delta\mu$. This ensures that only Gaussians classified as "dynamic" actually move.
  • Color Variation: Color changes are modeled multiplicatively ($c' = c(1 - \Delta c)$) to simulate lighting effects (shadows) on both static and dynamic elements, rather than changing the intrinsic material color.
• Loss Functions: Includes standard reconstruction losses (color, normal consistency, depth) plus specific regularization terms:
  • Separation Loss ($L_P$): Encourages Gaussians to remain static unless necessary (minimizing $\sum |P_i|$).
  • Change Regularization: Penalizes excessive position and color changes to prevent overfitting to noise.

Stage 2: Inverse Rendering

Once the geometry and dynamics are learned and frozen, the method performs inverse rendering to optimize material and lighting.

• Decomposition: Each Gaussian is assigned a diffuse albedo ($\rho$) and roughness ($\alpha$). The environment map ($L_{env}$) is modeled as an image representing incoming light.
• Rendering Equation: Instead of rendering Gaussian colors directly, the method computes pixel colors using the full rendering equation (Disney BRDF model).
  • It employs ray tracing to compute visibility ($V$) and indirect illumination ($L_{ind}$).
  • The incident light is calculated as $L_i = V \cdot L_{env} + L_{ind}$.
• Optimization: The albedo, roughness, and environment map are jointly optimized to minimize the difference between rendered pixels and ground truth images. Crucially, the MLP's color output ($\Delta c$) from Stage 1 is not used; lighting effects are now generated purely by the rendering equation.

3. Key Contributions

1. First Dynamic Gaussian Inverse Rendering: LumiMotion is the first Gaussian-based approach to perform inverse rendering on arbitrary dynamic scenes under unknown natural lighting, without requiring human pose priors or known light conditions.
2. Motion as Supervision: It introduces a novel mechanism to use scene motion as a supervisory signal. By observing surfaces under varying lighting conditions (as objects move), the model can better disentangle albedo from shadows.
3. Novel Constraints: The paper proposes a Static-Dynamic Separation Loss and a specific deformation network architecture that prevents the model from "moving" Gaussians to simulate shadows, ensuring shadows are handled by the lighting model rather than geometry.
4. New Benchmark: The authors release a new synthetic dataset comprising 5 scenes under 4 different lighting conditions, with both static and dynamic variants. This is the first dataset to enable systematic evaluation of inverse rendering in dynamic environments with ground-truth lighting and materials.

4. Results

The method was evaluated on both the new synthetic benchmark and real-world data (ENeRF dataset).

• Quantitative Performance:
  • Albedo Estimation: LumiMotion outperforms state-of-the-art static baselines (IRGS, R-3DGS, GI-GS) significantly. It improves LPIPS by 23% for albedo estimation.
  • Relighting: Achieves a 15% improvement in LPIPS for relighting tasks compared to the next-best baseline.
  • Robustness: The method maintains high performance even when training on dynamic scenes (which are inherently harder) and testing on single timesteps.
• Qualitative Performance:
  • Shadow Removal: The method successfully removes shadows from the estimated albedo, whereas baselines often bake shadows into the texture.
  • Relighting Fidelity: When re-rendered under new lighting, objects cast correct shadows and exhibit realistic specular highlights.
  • Environment Maps: The estimated environment maps capture the dominant light direction more accurately than static baselines.

5. Significance and Impact

• Advancing 3D Content Creation: By enabling accurate relighting of dynamic scenes, LumiMotion bridges the gap between current 3D reconstruction techniques and the requirements of film, gaming, and VR, where control over illumination is essential.
• Solving the "Shadow-Albedo" Ambiguity: The paper demonstrates that motion is a critical cue for solving the inherent ambiguity in inverse rendering. This insight could influence future research in static scene reconstruction by encouraging the use of multi-view temporal data even for seemingly static objects.
• Generalizability: Unlike avatar-specific methods, LumiMotion is object-agnostic, making it applicable to a wide range of real-world scenarios involving moving people, vehicles, or objects.

Limitations: The authors note that performance relies heavily on the accuracy of normal estimation and the learned deformations. Complex, intricate motions (e.g., near surfaces) can still cause artifacts, suggesting a need for more specialized supervision (like optical flow) in future work. The method is also sensitive to inaccurate camera pose estimation.