Technical Summary: CasaMaestro

Problem Statement

The rapid development of embodied AI for home environments necessitates fast, metrically accurate 3D reconstruction of residential spaces to support navigation and long-horizon task execution. Current state-of-the-art 3D reconstruction pipelines primarily rely on pinhole camera vision foundation models (VFMs). These methods face significant limitations when applied to house-scale indoor scenes:

1. Limited Field of View (FoV): Pinhole cameras require dense image sequences (thousands of images) to cover multiple rooms, leading to high computational costs and human effort.
2. Drift and Accumulation Errors: Long chains of incremental alignment in dense sequences exacerbate drift, causing severe reconstruction errors.
3. Sparse Capture Failure: Existing pinhole models struggle with sparse viewpoints (fewer images), often failing to recover scene geometry due to insufficient overlap.
4. Hardware Dependency: While LiDAR offers high-quality assets, it is costly and hardware-dependent, whereas vision-based solutions lack robustness for large-scale, sparse panoramic inputs.

Existing panoramic approaches are largely restricted to single-view depth prediction or small-scale, short-range multi-view settings (e.g., 2–5 views), failing to address the challenge of house-scale reconstruction from sparse multi-view panoramas.

Methodology

CasaMaestro is introduced as the first feedforward model capable of extrinsic-free, multi-view panoramic 3D reconstruction using sparse house-scale captures (20–50 panoramas). The architecture follows a minimalist design philosophy, built upon a DINOv2 backbone with specialized heads for depth and pose.

1. Unified Multi-View Backbone

The model utilizes a vanilla DINOv2 transformer backbone to process an arbitrary number of input panoramic views ($V$).

• Tokenization: Input panoramas are treated as distinct views. Patch embeddings are generated for each view.
• Attention Mechanism: The backbone alternates between intra-view self-attention (processing individual views independently) and cross-view self-attention (flattening the view dimension to allow tokens from different views to exchange information). This global interaction injects comprehensive scene context, improving robustness against large viewpoint displacements.
• Output: The backbone produces shared representations used for both depth and pose estimation.

2. Panoramic Camera Pose Decoder

To address the specific geometric challenges of panoramic inputs, a dedicated lightweight camera pose decoder is introduced, decoupling pose estimation from the main backbone.

• Motivation: While the backbone extracts dense features, pose estimation is a global geometric task requiring alignment across views. Independent regression (MLP per view) lacks explicit information exchange.
• Architecture:
  • Feature Projection: Inputs are projected to the model dimension.
  • Per-view Refinement: A pre-attention MLP with residual connections conditions the tokens.
  • Cross-view Attention: A transformer decoder layer aggregates information across the view dimension, utilizing an optional attention mask to handle invalid regions (e.g., camera poles).
  • Pose Heads: Separate linear heads regress translation ($\mathbf{t} \in \mathbb{R}^3$) and rotation (unit quaternion $\mathbf{q} \in \mathbb{R}^4$). The rotation is explicitly normalized to ensure geometric validity.
• Advantage: This design allows the backbone to focus on feature extraction while the decoder explicitly enforces global consistency for pose prediction.

3. ERP Data Augmentation

To overcome the scarcity of diverse viewpoints in existing datasets (where views often have minimal relative rotation), the authors propose a novel Equirectangular Projection (ERP) data augmentation strategy.

• Process: Random rotations (yaw, pitch, roll) are sampled and applied to the source panorama via ray rotation and remapping.
• Consistency: The augmentation strictly preserves 3D geometry by simultaneously warping the RGB images, depth maps, and updating the camera extrinsic matrices (rotation block) via right-multiplication.
• Impact: This generates abundant pose-view pairs, significantly boosting the model's robustness to orientation changes and encouraging cross-view reasoning.

Key Contributions

1. CasaMaestro Model: The first feedforward model for metric, house-scale panoramic reconstruction. It accepts 20–50 sparse indoor panoramas and directly predicts metric depth and camera poses, enabling immediate point-cloud reconstruction of entire homes.
2. Lightweight Pose Decoder: A specialized decoder designed to handle large displacements in panoramic inputs, achieving superior pose estimation accuracy through cross-view attention mechanisms.
3. ERP Data Augmentation: A novel strategy to generate diverse pose-view pairs from existing data, addressing the limitation of limited viewpoint rotation in current training datasets.

Experimental Results

The model was trained on the Realsee3D dataset and evaluated on both real-world and synthetic scenes, including zero-shot tests on unseen datasets (PanoSUNCG, HM3D, Replica).

• Pose Estimation: CasaMaestro significantly outperforms existing methods (including PanoPose, SPR, VGGT, and Pi3).
  • On Realsee-Real, it achieves an AUC@30 of 0.903, representing an 84% improvement over the best previous result.
  • On Realsee-Syn, it achieves an AUC@30 of 0.927, representing a 119% improvement.
  • It demonstrates balanced low rotation and translation errors, whereas other methods often suffer from high rotation errors or fail to reconstruct house structures under sparse views.
• Depth Estimation:
  • CasaMaestro achieves an AbsRel of 0.078 on Realsee-Real and 0.079 on Realsee-Syn.
  • In zero-shot evaluations on unseen datasets, it shows an average 21.98% decrease in AbsRel compared to the best previous panoramic depth estimation methods, despite not being explicitly designed solely for depth.
• Qualitative Performance: Visual comparisons show that while other methods produce messy or corrupted reconstructions with sparse inputs, CasaMaestro accurately restores house structures with full coverage and geometric consistency.

Significance and Claims

The paper claims that CasaMaestro establishes a new paradigm for 3D reconstruction by demonstrating that feedforward "any-view" reconstruction is not restricted to pinhole cameras. By leveraging multi-view panoramas, the model achieves:

• Efficiency: Reducing reconstruction time from ~25s (dense pinhole) to 0.56s for house-scale scenes.
• Robustness: Successfully handling sparse inputs (20–50 views) where traditional methods fail.
• Foundation for Simulation: Providing a strong foundation for acquiring house-scale 3D indoor assets for closed-loop simulation in embodied AI.

The authors conclude that the rise of Vision Foundation Models built upon panoramas and fisheye cameras could provide a new revolution in 3D tasks, moving beyond the limitations of narrow-field-of-view pinhole models.