LTGS: Long-Term Gaussian Scene Chronology From Sparse View Updates

1. Problem Statement

Existing 3D reconstruction methods, such as Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS), excel at creating photo-realistic static scenes from dense camera captures. However, they struggle with everyday environments that undergo frequent, discrete changes (e.g., objects being moved, added, or removed).

• Limitations of Current Approaches:
  • Static Reconstruction: Requires re-scanning the entire scene from scratch when changes occur, discarding previous data.
  • Dynamic/4D Reconstruction: Typically requires dense, continuous temporal observations to model smooth motion, failing to handle abrupt, sparse changes.
  • Continual Learning: Existing methods often lack structural priors, requiring many new images for updates and suffering from catastrophic forgetting or severe artifacts in sparse-view settings.
• The Challenge: How to efficiently update a 3D scene representation using a spatio-temporally sparse set of images (few-shot, casual captures) while maintaining photorealistic rendering and avoiding artifacts, without retraining the entire scene.

2. Methodology: LTGS Framework

The authors propose LTGS (Long-Term Gaussian Scene Chronology), an integrated pipeline that treats the scene as a collection of reusable object-level Gaussian templates and a static background. The framework operates in four main stages:

A. Change Detection

To identify what has changed between the initial reconstruction ($G_0$) and new sparse inputs ($I_t$):

• Camera Localization: Uses a robust hierarchical localization pipeline to align new camera poses with the initial scene.
• Hybrid Difference Detection: Compares rendered images of the initial scene against captured images using:
  • Semantic Differences: Cosine similarity of features from the Segment Anything Model (SAM) to detect object-level changes regardless of lighting.
  • Photometric Differences: Structural Similarity Index (SSIM) to catch subtle geometric deviations SAM might miss.
• Mask Generation: Combines these cues to generate binary change masks, which are then refined using SAM to isolate specific object instances.

B. Object Tracking and Template Reconstruction

Instead of re-scanning the whole scene, LTGS focuses on individual objects:

• 2D Instance Matching:
  • Intra-timestep: Uses MASt3R (a geometric foundation model) to match object masks across different views within the same time step, building a graph to assign unique instance IDs.
  • Cross-timestep: Aggregates SAM features within object regions and uses Hungarian matching to associate objects across different time steps.
• Template Initialization:
  • Existing Objects: Decomposes the initial 3DGS into object-specific Gaussians and a background.
  • New Objects: Initializes 3D point clouds for new objects using MASt3R depth maps and converts them into Gaussian primitives.
• Template Tracking: Uses robust point cloud registration (augmented with DINO features) to establish 6DoF poses between object instances across time, ensuring geometric consistency.

C. Long-Term Gaussian Optimization

The core innovation is the creation of reusable Gaussian templates ($T$) that serve as structural priors.

• Aggregation: Observations from multiple time steps are aggregated to refine a shared template for each object.
• Transformation: Templates are transformed to specific time steps using the estimated 6DoF poses.
• Optimization: The system optimizes Gaussian parameters (position, rotation, scale, opacity, color) to minimize the photometric loss ($L_1$ + D-SSIM) between rendered and captured images.
  • Key Constraint: It enforces consistency using the initial camera poses and applies a temporal opacity filter to hide transient objects.
  • Efficiency: The process avoids densification/cloning (common in standard 3DGS) and skips opacity resetting, relying on the strong priors from the templates.

D. Scene Compositing

The final scene at any time step $t$ is rendered by compositing the transformed object templates with the static background Gaussians ($B_0$).

3. Key Contributions

1. LTGS Framework: A novel strategy for updating 3DGS scenes from sparse, casual captures by modeling object-level chronology rather than global scene dynamics.
2. Reusable Geometric Priors: The introduction of Gaussian templates that act as structural priors, allowing the system to relocalize and refine objects across time without re-scanning the entire environment.
3. Robust Sparse-View Handling: A pipeline combining foundation models (SAM, MASt3R, DINO) with geometric aggregation to handle abrupt geometric changes (additions, removals, relocations) where previous methods fail.
4. New Real-World Dataset: The authors collected a new dataset containing five indoor scenes with dynamic object-level changes across multiple time steps, specifically designed to evaluate long-term sparse updates.

4. Experimental Results

The framework was evaluated on both a synthetic dataset (CL-NeRF) and the authors' new real-world dataset.

• Quantitative Performance:
  • LTGS achieved superior PSNR, SSIM, and LPIPS scores compared to baselines like 3DGS, InstantSplat, 4DGS, NSC, and Continual Learning methods (CL-NeRF, CL-Splats).
  • On the real-world dataset, LTGS achieved 23.46 PSNR and 0.889 SSIM, significantly outperforming the next best method (CL-Splats at 21.12 PSNR).
• Qualitative Performance:
  • LTGS successfully reconstructed discrete changes (e.g., a laptop appearing or a chair moving) without the "floating artifacts" or over-smoothing seen in 4DGS or NSC.
  • It maintained sharp object boundaries and consistent geometry even with only 3 images per time step.
• Efficiency:
  • Speed: The entire process (detection, matching, optimization) takes approximately 6.5 minutes on an NVIDIA RTX 4090.
  • Memory: The memory footprint remains comparable to the initial static 3DGS, unlike continual learning methods that accumulate significant memory overhead.

5. Significance and Impact

• Practicality for Digital Twins: LTGS bridges the gap between high-fidelity 3D reconstruction and the reality of everyday environments where objects move frequently. It enables the maintenance of "living" digital twins without requiring constant, dense scanning.
• Scalability: By decoupling the scene into reusable object templates, the method scales efficiently to long time horizons, avoiding the redundancy of retraining entire scenes.
• Foundation Model Integration: The paper demonstrates a successful application of foundation models (SAM, MASt3R) not just for perception, but as structural components for 3D reconstruction and tracking in sparse settings.
• Future Directions: The authors note that while the method handles geometric changes well, significant lighting changes or non-rigid articulations (e.g., a person walking) remain challenges, suggesting future work in modeling complex appearance variations and articulated objects.

In summary, LTGS provides a robust, efficient, and scalable solution for maintaining 3D scene representations in dynamic, real-world environments using only sparse, casual observations.