Bringing a Personal Point of View: Evaluating Dynamic 3D Gaussian Splatting for Egocentric Scene Reconstruction

1. Problem Statement

Egocentric video (first-person perspective from head-mounted cameras) is crucial for applications like augmented reality (AR), robotics, and assistive technology. However, reconstructing 3D scenes from this perspective is significantly more challenging than from third-person (exocentric) views due to:

• Rapid and unpredictable camera motion driven by head and body movement.
• Complex scene dynamics, often involving intricate hand-object interactions.
• Lack of evaluation: While 3D Gaussian Splatting (3DGS) has become the state-of-the-art for static and dynamic scene reconstruction, existing dynamic monocular 3DGS models are primarily evaluated on third-person datasets. It remains unclear whether these models generalize to the unique challenges of egocentric video or if specialized approaches are required.

2. Methodology

The authors conducted a comprehensive empirical evaluation comparing existing dynamic monocular 3DGS models on paired ego-exo (first-person and third-person) recordings from the EgoExo4D dataset.

• Dataset & Setup:

  • Used 25 scenes from EgoExo4D: 20 random clips (300 frames each) and 5 clips specifically selected to meet the rigid-motion constraints of the EgoGaussian model.
  • Preprocessing: EgoExo4D uses fisheye cameras. The authors undistorted frames using ground-truth intrinsics and applied binary masks to filter invalid pixels (black borders).
  • Training Strategy:
    • Egocentric: Standard temporal split (even frames for training, specific indices for validation/test).
    • Exocentric: Since exo cameras are static, the authors simulated monocular input by randomly selecting a viewpoint at each frame index to preserve 3D cues while maintaining a single-camera training setup.

• Models Evaluated:

  1. Deformable-3DGS: Uses a global deformation field.
  2. 4DGS: Uses a global motion field.
  3. RTGS: Does not rely on an explicit motion field.
  4. EgoGaussian: A model specifically designed for egocentric vision, requiring manual splitting of clips into "passive" (static background) and "active" (object manipulation) segments.

• Evaluation Protocol:

  • Metrics: Masked PSNR (mPSNR), masked SSIM (mSSIM), and masked LPIPS (mLPIPS).
  • Granular Analysis: The authors introduced a protocol to evaluate static and dynamic regions separately using masks generated by Segment Anything 2 (SAM 2).
  • Motion Correlation: Investigated the correlation between camera motion (linear velocity, angular velocity, and baseline distance) and reconstruction quality.

3. Key Contributions

1. First Large-Scale Egocentric Evaluation: Provided the first quantitative comparison of dynamic monocular 3DGS models on paired ego-exo data, revealing a consistent performance gap.
2. Static vs. Dynamic Decomposition: Proposed and validated an evaluation protocol that separates static and dynamic regions, showing that current models struggle significantly more with static backgrounds in egocentric views than previously assumed.
3. Motion-Quality Correlation: Demonstrated that increased camera motion in egocentric videos correlates with worse reconstruction quality, challenging prior assumptions that more motion (providing multi-view cues) aids reconstruction.
4. Reproduction & Benchmarking: Re-evaluated EgoGaussian and found it underperforms compared to general baselines (Deformable-3DGS, 4DGS) when using corrected metric calculations, contradicting its original claims.

4. Key Results

A. Egocentric vs. Exocentric Performance

• Overall Gap: All models performed consistently worse on egocentric views compared to exocentric views across all metrics (mPSNR, mSSIM, mLPIPS).
• Static vs. Dynamic Source of Error:
  • Contrary to intuition, the performance gap was primarily driven by static region reconstruction, not dynamic objects.
  • In random scenes, dynamic mPSNR was sometimes comparable or even slightly better in ego views, but static mPSNR was significantly lower in ego views.
  • This suggests that rapid camera motion disrupts the modeling of the static background more severely than the dynamic objects in these models.

B. EgoGaussian Performance

• Underperformance: In the authors' experiments, EgoGaussian performed worse than general baselines (Deformable-3DGS, 4DGS) on both static and dynamic regions.
• Metric Discrepancy: The authors identified that the original EgoGaussian paper used uncorrected metrics (e.g., zeroing out masked areas rather than ignoring them, and failing to normalize inputs for LPIPS). When corrected, the baselines outperformed EgoGaussian.

C. Camera Motion Analysis

• Velocity Correlation: There is a strong negative correlation between camera linear/angular velocity and reconstruction quality (higher velocity = lower mPSNR/mSSIM, higher mLPIPS).
• Baseline Correlation: Unlike some prior works suggesting that larger camera baselines (more multi-view coverage) improve reconstruction, this study found no positive correlation. Increased baseline did not lead to better quality and often correlated with degradation.

5. Significance and Implications

• Need for Specialized Models: The results strongly indicate that existing dynamic 3DGS models, trained on third-person data, do not generalize well to egocentric settings. The unique challenges of head-mounted camera motion require egocentric-specific architectures.
• Re-evaluation of Static/Dynamic Modeling: The finding that static regions are the bottleneck in egocentric reconstruction suggests that future methods should not treat static and dynamic modeling as a monolithic task. Separating these regions (as done in EgoGaussian or Gaufre) may be beneficial, but the current implementation of EgoGaussian is insufficient.
• Dataset & Benchmarking: The paper highlights the need for denser exocentric camera coverage or moving exocentric cameras in future datasets to ensure fair comparisons between ego and exo modalities.
• Practical Impact: For AR and robotics, relying on current off-the-shelf 3DGS models for first-person scene understanding may lead to poor spatial perception, particularly regarding the static environment.

Conclusion

The paper concludes that egocentric 3D reconstruction is fundamentally more difficult than exocentric reconstruction for current dynamic 3DGS models. The degradation is not solely due to dynamic objects but is heavily influenced by the difficulty of modeling static backgrounds under rapid camera motion. Future work must focus on developing models specifically tailored to the physics and motion patterns of first-person vision.