Lifting Unlabeled Internet-level Data for 3D Scene Understanding

1. Problem Statement

The field of 3D scene understanding is currently bottlenecked by the scarcity and high cost of annotated 3D data. While 2D image datasets can be easily scraped and annotated from the web, acquiring high-quality 3D data (e.g., RGB-D scans, LiDAR point clouds with semantic labels) requires specialized hardware, manual reconstruction, and labor-intensive labeling. Existing datasets (like ScanNet, ARKitScenes) have seen limited quantitative scaling since their inception. Conversely, the internet contains orders of magnitude more unlabeled video data that captures 3D environments, but this resource remains largely untapped for training comprehensive 3D models due to the difficulty of automatically converting 2D video into structured 3D representations.

2. Methodology: SceneVerse++ Pipeline

The authors propose SceneVerse++, a framework that leverages carefully designed automated "data engines" to transform unlabeled internet videos into high-quality training data for 3D scene understanding. The pipeline consists of three main stages:

A. Data Curation & Sparse Reconstruction (SfM)

• Source: 8,217 indoor room-tour videos from YouTube and Bilibili.
• Preprocessing: Videos are split into shots using TransNetV2. Redundant frames are filtered, and keyframes are selected based on parallax (rather than uniform sampling) to ensure robust triangulation.
• Filtering: Content is filtered to remove outdoor scenes, humans, and low-quality clips.
• Sparse Reconstruction: The pipeline uses a dense pixel matching and bundle adjustment approach (inspired by Mast3R-SfM) to generate camera poses and sparse 3D point clouds.
• Quality Control: Scenes with small spatial coverage or incorrect SfM results are filtered out using human annotation (<10 seconds/scene) to ensure downstream data quality.
• Output: 6,687 reconstructed scenes with camera poses and sparse geometry.

B. Dense Reconstruction & Instance Segmentation

To convert sparse outputs into dense, semantically labeled 3D scenes:

• Dense Reconstruction: The pipeline projects sparse 3D points to create sparse depth maps, which serve as priors for PriorDA to predict dense metric depth. These depths are fused using a Truncated Signed Distance Function (TSDF) to generate watertight 3D meshes. This balances efficiency and quality, avoiding the computational cost of per-scene neural rendering optimization.
• Instance Segmentation:
  1. 2D Masking: Per-frame segmentation masks are generated using CropFormer.
  2. 3D Lifting: Masks are aggregated in 3D space using neighboring-frame view consensus and spatial agreement to lift 2D segments to 3D instances.
  3. Semantic Alignment: Describe Anything and Qwen2-VL generate textual descriptions for each 3D instance, aligning them to the ScanNet category set.
• Efficiency: The average runtime is ~71s for reconstruction and ~96s for segmentation per scene.

C. Automated Annotation Generation

The framework generates annotations for three distinct tasks:

1. 3D Object Detection & Segmentation: Direct output of 3D meshes with instance labels.
2. 3D Spatial VQA: 3D scene graphs are constructed from the point clouds. Task-specific templates generate questions regarding object counts, relative distances, directions, sizes, and room dimensions.
3. 3D Vision-Language Navigation (VLN):
   • Trajectory Refinement: Raw camera trajectories from room-tour videos are processed to remove redundancy (backtracking, erratic rotations) and split into coherent sub-paths.
   • Action Encoding: Camera poses are converted into discrete navigation actions (Move Forward, Turn Left/Right) compatible with the R2R benchmark.
   • Instruction Generation: VLMs generate natural language instructions (Formal, Conversational, Narrative) for each trajectory using Chain-of-Thought reasoning.

3. Key Contributions

• SceneVerse++ Dataset: A large-scale dataset of 6,687 real-world indoor scenes derived from internet videos, containing images, camera poses, dense reconstructions, instance segmentations, and high-level reasoning annotations. It exceeds existing datasets in scene size, object diversity, and scale.
• Automated Data Engine: A modular pipeline demonstrating that high-quality 3D training data can be generated automatically from unlabeled web videos without specialized hardware.
• Systematic Analysis: The paper provides a critical analysis of bottlenecks in automated data generation, highlighting the trade-offs between efficiency and quality, and the impact of domain shifts on model scaling.
• Task-Agnostic Validation: The approach is validated across three distinct perception granularities: low-level perception (detection/segmentation), high-level reasoning (VQA), and embodied navigation (VLN).

4. Experimental Results

The authors evaluated models trained on SceneVerse++ on standard benchmarks (ScanNet, ARKitScenes, VSI-Bench, R2R):

• 3D Object Detection (SpatialLM):
  • Zero-shot: Models pretrained on SceneVerse++ outperformed those trained only on synthetic data on real-world benchmarks.
  • Fine-tuning: Pretraining on SceneVerse++ followed by fine-tuning on ScanNet achieved a significant boost, reaching 58.6 F1@0.25 (vs. 38.0 for ScanNet-only training).
• 3D Instance Segmentation (Mask3D):
  • Zero-shot transfer was limited due to sensitivity to domain-specific biases (e.g., sensor noise, segmentation thresholds).
  • However, fine-tuning on ScanNet after SceneVerse++ pretraining yielded consistent improvements across all metrics compared to training from scratch.
• 3D Spatial VQA (VSI-Bench):
  • Training on SceneVerse++ significantly improved spatial reasoning capabilities of VLMs (Qwen2.5-VL).
  • Zero-shot performance on the full VSI-Bench improved by +14.9% (3B model) and +9.8% (7B model).
  • The model showed strong generalization on "general spatial knowledge" (e.g., relative direction) but struggled with "domain-specific" tasks (e.g., object counting) where data distribution gaps were larger.
• 3D Vision-Language Navigation (R2R):
  • Zero-shot transfer from internet videos to simulation showed a modest improvement in Success Rate (SR).
  • Fine-tuning: Pretraining on SceneVerse++ followed by fine-tuning on R2R increased SR to 0.228 (vs. 0.088 for R2R-only), demonstrating that real-world trajectory priors are crucial for navigation.
  • Ablation studies confirmed that trajectory refinement and instruction enrichment are critical; raw video data alone is insufficient.

5. Significance and Future Directions

• Scalability: The work proves that web-scale unlabeled video is a viable path to scaling 3D scene understanding, potentially overcoming the stagnation in 3D data acquisition.
• Generalization vs. Overfitting: The study highlights that while automated data improves general spatial reasoning, models remain sensitive to domain shifts (e.g., specific object distributions or sensor biases).
• Future Work: The authors advocate for:
  • Developing sub-modules (SfM, segmentation) specifically optimized for "in-the-wild" automated pipelines rather than just task-specific benchmarks.
  • Creating benchmarks that better measure generalization and reduce data contamination.
  • Addressing ethical concerns regarding privacy in public internet videos.

In conclusion, SceneVerse++ demonstrates that with robust automated data engineering, the vast reservoir of unlabeled internet videos can be effectively "lifted" to train state-of-the-art 3D models, bridging the gap between 2D web data and 3D spatial intelligence.