Scalable Adaptation of 3D Geometric Foundation Models via Weak Supervision from Internet Video

Technical Summary: Scalable Adaptation of 3D Geometric Foundation Models via Weak Supervision from Internet Video (SAGE)

1. Problem Statement

The development of 3D Geometric Foundation Models (GFMs)—such as DUSt3R and MV-DUSt3R—has revolutionized pose-free 3D reconstruction. However, these models face a critical data scarcity bottleneck. Unlike 2D vision models trained on billions of web images, 3D models rely on expensive, sensor-captured datasets (e.g., ScanNet) that are limited in scale and often restricted to specific indoor environments. This leads to a significant distribution shift and poor zero-shot generalization when models are applied to diverse, "in-the-wild" scenes.

While Internet videos (e.g., YouTube) offer virtually unlimited data, they lack ground-truth 3D geometry and camera poses. Simply fine-tuning models using pseudo-depth from video depth estimators (a method the authors call VDS) often fails because these estimates are noisy and lack multi-view consistency, which can actually degrade reconstruction quality.

2. Methodology: The SAGE Framework

To overcome these challenges, the authors propose SAGE (Scalable Adaptation of GEometric foundation models), a framework that transforms raw, unlabelled video streams into effective training signals through a hierarchical mining and hybrid supervision pipeline.

A. Spatio-Temporal Mining Pipeline:
Instead of uniform sampling, SAGE uses stochastic temporal perturbations to sample video frames. By adding random offsets to temporal intervals, the framework maximizes viewpoint diversity while maintaining enough geometric overlap to ensure stable supervision.

B. Hybrid Supervision Strategy:
SAGE employs two complementary levels of weak supervision to balance accuracy and density:

1. Sparse Geometric Anchoring (Global Guidance): The framework uses Structure-from-Motion (SfM) via COLMAP to extract sparse 3D point clouds from the videos. These points act as "anchors" that provide reliable, globally consistent structural guidance, preventing the model from drifting into degenerate geometric solutions.
2. Dense Differentiable Consistency (Local/Dense Guidance): To fill the gaps between sparse points, SAGE leverages 3D Gaussian Splatting (3DGS) as a differentiable rendering module. By enforcing photometric consistency between "context frames" (inputs) and "target frames" (unseen views within the same video), the model propagates dense gradients to refine the underlying geometry.

C. Regularization against Catastrophic Forgetting:
To ensure the model retains the high-quality geometric priors learned during its initial supervised pre-training, SAGE introduces a regularization strategy. It mixes a small fraction (approximately 3%) of the original supervised "anchor data" into the training loop, keeping the model within the manifold of plausible 3D geometries.

3. Key Contributions

• First Scalable Adaptation Framework: It is the first work to enable the scaling of feed-forward 3D geometric foundation models using unlabelled Internet video.
• Robust Hybrid Supervision: The integration of sparse SfM anchors with dense differentiable Gaussian rendering provides a unique way to extract signal from noisy video.
• Demonstrated Scalability: The authors prove that the model follows a log-linear scaling law, where performance improves consistently as the volume of video data increases from 100 to 10,000 scenes.

4. Experimental Results

The framework was evaluated by adapting the pretrained MV-DUSt3R model using over 10,000 video clips from RealEstate10K and DL3DV.

• Zero-Shot Generalization: On unseen benchmarks (7Scenes, TUM-RGBD, and Matterport3D), SAGE significantly outperformed the baseline, reducing the Chamfer Distance (reconstruction error) by 20% to 42%.
• Pose Estimation: Beyond geometry, SAGE improved the accuracy of camera pose estimation (rotation and translation errors) compared to the baseline.
• Robustness: The model showed surprising generalization to out-of-distribution outdoor datasets (UASOL), despite being trained primarily on indoor videos.
• Ablation Insights: Experiments confirmed that the combination of sparse anchors and dense consistency is essential, and that "interpolated" target views (views between input frames) provide more stable supervision than "extrapolated" views.

5. Significance

SAGE represents a paradigm shift in 3D computer vision. It moves the field away from a reliance on expensive, manually annotated 3D datasets toward a data-abundant regime powered by the vastness of the Internet. By proving that unlabelled video can be effectively "distilled" into geometric knowledge, SAGE establishes a scalable path toward creating truly general-purpose, world-scale 3D foundation models.