3D sans 3D Scans: Scalable Pre-training from Video-Generated Point Clouds

1. Problem Statement

Despite significant advancements in 2D vision foundation models (e.g., DINOv3, SAM) trained on billions of images, 3D self-supervised learning (3D-SSL) for indoor scenes remains severely constrained by data scarcity.

• The Bottleneck: High-quality 3D data requires expensive LiDAR/RGB-D scanning and labor-intensive manual annotation. The largest widely used indoor datasets (e.g., ScanNet) contain only ~5k unique scenes.
• Current Limitations: Existing 3D-SSL methods (e.g., Sonata) rely on mixing real scans with synthetic data, but the total scale remains small (~140k samples, with only ~18k real scans). This data limitation prevents 3D-SSL from reaching the generalization levels seen in 2D vision.
• The Hypothesis: The authors hypothesize that unlabeled videos (e.g., real estate walkthroughs) contain sufficient geometric cues to learn robust 3D representations, provided they can be reconstructed into point clouds without real 3D sensors.

2. Methodology

The proposed framework consists of two main components: a large-scale dataset construction pipeline and a novel self-supervised learning algorithm.

A. Dataset: RoomTours

The authors constructed RoomTours, a dataset of 49,219 video-generated point cloud (VGPC) scenes.

• Source: Collected 3,462 indoor walkthrough videos from the web (YouTube, real estate tours) across 19 countries.
• Reconstruction: Videos are segmented into scene sequences (living rooms, bedrooms, bathrooms) using CLIP. These sequences are processed by $\pi^3$, a state-of-the-art feed-forward 3D reconstruction model, to generate point clouds.
• Preprocessing: Since $\pi^3$ outputs are noisy and lack consistent coordinate systems, the authors apply a lightweight alignment pipeline:
  • Statistical outlier removal (SOR).
  • Z-axis alignment (ensuring "up" is consistent).
  • Scale normalization (aligning scene diagonal lengths to a distribution estimated from ScanNet).
  • Normal estimation via local PCA.

B. Algorithm: LAM3C (Laplacian-Aware Multi-level 3D Clustering)

The core innovation is LAM3C, a self-supervised framework designed to learn stable representations from noisy VGPC. It uses a Teacher-Student architecture (with the teacher updated via Exponential Moving Average) and Sinkhorn-Knopp clustering.

To address the noise and missing regions inherent in VGPC, LAM3C introduces two novel noise-regularized losses alongside the standard clustering loss:

1. Laplacian Smoothing Loss ($R_{Lap}$):

   • Goal: Enforce local geometric smoothness.
   • Mechanism: Constructs a k-Nearest Neighbor (kNN) graph on the point cloud. It penalizes large differences in embeddings between spatially adjacent points.
   • Robustness: Uses distance-based weights (closer points have higher weight) and a Huber penalty to reduce the influence of outliers. This ensures that even if a point is noisy, its neighbors help stabilize its representation.

2. Noise Consistency Loss ($R_{cons}$):

   • Goal: Ensure feature stability across different views of the same scene.
   • Mechanism: Takes two augmented views of the same VGPC (one with noise augmentation, one masked) and minimizes the discrepancy between the student's and teacher's embeddings for corresponding points.
   • Effect: Forces the model to learn features that are invariant to the specific noise patterns introduced during reconstruction.

Total Objective:
$$\mathcal{L}_{total} = \mathcal{L}_{clustering} + \lambda R_{Lap} + \mu R_{cons}$$
Where $\lambda$ is linearly scheduled during training, and $\mu$ is fixed.

3. Key Contributions

1. RoomTours Dataset: A scalable dataset of ~49k indoor point clouds generated entirely from unlabeled web videos, bypassing the need for physical 3D scanning.
2. LAM3C Framework: A novel pre-training method that successfully learns from imperfect, noisy point clouds by introducing Laplacian smoothing and noise consistency regularization.
3. Scan-Free Pre-training: Demonstrates that 3D representations can be learned without any real 3D scan data, challenging the necessity of expensive LiDAR datasets for pre-training.

4. Experimental Results

The authors evaluated LAM3C on semantic and instance segmentation tasks across four benchmarks: ScanNet, ScanNet200, ScanNet++, and S3DIS.

• Performance vs. State-of-the-Art:
  • LAM3C trained solely on VGPC outperforms Sonata (trained on 18k real scans + 121k synthetic) on multiple benchmarks.
  • Semantic Segmentation: On ScanNet (Full Fine-Tuning), LAM3C (49k VGPC) achieves 77.7 mIoU, surpassing Sonata's 76.0 mIoU (trained on real+synthetic).
  • Instance Segmentation: LAM3C achieves 45.7 mAP on S3DIS, outperforming Sonata's 45.5 mAP.
• Data Scaling: Performance scales with the size of the VGPC dataset. Moving from 16k to 49k scenes yields significant gains, suggesting the approach is highly scalable.
• Linear Probing: LAM3C shows particularly strong results in linear probing (frozen backbone), indicating the learned features are highly transferable and robust.
• Ablation Studies:
  • Removing the noise regularizers causes a significant drop in performance.
  • Using the $\pi^3$ reconstruction model yields better results than VGGT or MapAnything.
  • Z-axis and scale alignment are critical for performance, especially in linear probing.

5. Significance and Conclusion

• Paradigm Shift: The paper proves that unlabeled videos are a viable, abundant, and scalable source for 3D self-supervised learning, potentially democratizing 3D AI development by removing the dependency on expensive 3D sensors.
• Handling Imperfection: The work highlights that modern feed-forward reconstruction models, combined with specific regularization techniques (Laplacian smoothing), can overcome the noise and incompleteness of video-derived 3D data.
• Future Impact: This approach offers a path toward "Scan-Free" 3D understanding, allowing for the pre-training of massive 3D foundation models using the vast amount of video data available on the internet, similar to how 2D models utilize image datasets.

In summary, LAM3C successfully bridges the gap between 2D video data and 3D understanding, achieving state-of-the-art performance without a single real 3D scan.