Technical Summary: Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition

Problem Statement

Inverse rendering aims to decompose outgoing surface radiance into intrinsic material properties (albedo, roughness) and illumination. This task is severely ill-posed because multiple combinations of materials and lighting can produce the same observed color. Traditionally, this ambiguity is resolved by capturing static objects under multiple lighting conditions or by relying on hand-crafted or data-driven priors. However, these approaches often require sophisticated lighting setups, extended training procedures on limited 3D data, or specific assumptions about the scene.

The authors posit that the ambiguity inherent in static multiview captures—where albedo and lighting often become entangled—can be resolved more effectively by observing an object undergoing significant rigid motion. While static captures offer diverse viewpoints, they often lack diverse light-surface interactions if the illumination is fixed. Conversely, a hand-held capture setting, where a user rotates an object in front of a static camera, provides a single RGB video with diverse surface-light interactions under a single far-field light representation. This motion-induced diversity offers stronger constraints for disentangling material and lighting without requiring complex multi-light setups or massive datasets.

Methodology

The paper proposes a three-stage pipeline that integrates object tracking, 3D reconstruction, and inverse rendering for general rigidly moving objects. The approach avoids joint optimization of all parameters from the start to prevent convergence to suboptimal local minima.

Stage 1: Progressive Pose and Geometry Estimation

The first stage focuses on obtaining coarse geometry and initial object poses.

• Representation: The authors utilize the Neural SDF (NeuS) representation [53], leveraging its regularization properties and unit sphere initialization to maintain surface integrity under sparse observations.
• Optimization: A progressive sequential optimization scheme is employed. The system learns geometry on a coarse level to prevent pose overfitting to high-frequency noise.
• Tracking: Object poses are estimated frame-by-frame using a per-frame pose MLP. To handle the unknown camera/object pose, the system uses 2D feature correspondences from state-of-the-art matchers (e.g., [10, 31]) to impose cross-frame consistency, which is crucial for textureless or specular hand-held objects.
• Virtual Camera: A virtual camera system centers the object in each image using 2D masks, reducing degrees of freedom to ease optimization for noisy real-world sequences.

Stage 2: Global Pose and Geometry Refinement

Once initial estimates are obtained, the pipeline switches to a finer representation to capture detailed geometry.

• Representation: The method introduces 3D Gaussians [29] integrated with an SDF (DiscretizedSDF [65]). This hybrid representation captures fine details while stabilizing the geometry, particularly for shiny objects.
• Refinement: A global MLP with per-pose embeddings refines the poses estimated in Stage 1. The Gaussians are transformed according to the object pose, and their normals are derived from the rotation axis corresponding to the smallest scale.
• Optimization: The system globally optimizes geometry and all object poses using a loss function that includes photometric loss, normal regularization, variance constraints, and rank constraints to ensure disc-like Gaussians. Adaptive density control (from FastGS [43]) is used to maintain efficiency.

Stage 3: Dynamic Inverse Rendering

In the final stage, material properties are added to the 3D Gaussians to perform physically-based inverse rendering.

• Material Model: The Bidirectional Reflectance Distribution Function (BRDF) is parameterized with a diffuse term (Lambertian) and a specular term (GGX microfacet model). Spatially varying albedo and roughness are stored at the corners of the Gaussian planes and interpolated at intersection points.
• Illumination & Visibility: The environment is modeled as a distant light source (environment map). To handle self-occlusions and global illumination, the pipeline employs efficient Gaussian ray tracing [38]. Secondary rays are traced to determine visibility and approximate indirect illumination.
• Optimization: The system jointly optimizes material parameters, environment illumination, geometry, and poses using a deferred shading approach. The loss includes smoothness constraints on the environment and material maps.
• Relighting: For relighting, the method uses BRDF- and lighting-based importance sampling and the split-sum approximation [28] to compute one-bounce indirect radiance, ensuring high-fidelity relighting.

Key Contributions

1. Motion as a Constraint: The paper empirically validates that observing an object under significant motion provides stronger constraints for material-lighting decomposition than static capture, even when using a simple far-field light representation.
2. 4D Inverse Rendering Framework: A novel pipeline is proposed that unifies object pose tracking and surface material estimation for general rigidly moving objects, utilizing a progression from Neural SDFs to 3D Gaussians.
3. Synthetic Dataset: The authors introduce a synthetic dataset of common household objects with ground-truth materials and relit views. This dataset enables controlled evaluation across three capture settings: static multiview, turntable, and hand-held capture.

Experimental Results

The method was evaluated on both synthetic data and real-world hand-held object sequences.

• Synthetic Evaluation: Quantitative results (PSNR, SSIM, LPIPS, MAE) demonstrate that the hand-held setting significantly outperforms static and turntable settings. Notably, the hand-held setting with estimated poses outperformed the static setting with ground-truth poses, highlighting the strength of the motion-induced constraints. The improvement was particularly pronounced for diffuse objects where material-lighting ambiguity is highest.
• Real-World Performance: Qualitative results on real hand-held videos show that the method produces cleaner albedo maps and more plausible relighting compared to static multiview captures, which often bake shadows into the albedo.
• Ablation Studies: Analysis confirms that the gains are due to the diversity of surface-light interactions rather than the number of views; a small number of hand-held views outperformed a large number of static views. Furthermore, the method maintains albedo quality under complex lighting conditions where static methods degrade.
• Benchmarking: On the TensoIR benchmark (static setting), the method is competitive with existing baselines. In the dynamic setting, it outperforms adapted baselines like ReCap, SVG-IR, and IRGS.

Significance and Claims

The paper claims that leveraging the natural diversity of surface-light interactions induced by rigid object motion is a viable and effective strategy for resolving the long-standing material-lighting ambiguity in inverse rendering. By avoiding the need for additional light sources or complex learned priors, the proposed method achieves robust decomposition in the challenging hand-held capture setting. The authors suggest that this approach enables high-quality relighting and augmented reality applications using standard monocular video of hand-held objects.

Future work directions identified by the authors include modeling hand occlusions, handling complex near-field light interactions, and extending the pipeline to non-rigid objects.