Technical Summary: MapAnything

Problem Statement

Traditional image-based 3D reconstruction relies on decomposing the problem into distinct, sequential tasks such as feature detection, matching, pose estimation, camera calibration, bundle adjustment, and multi-view stereo (MVS). While recent feed-forward approaches have unified some of these tasks, they often remain limited to specific input configurations (e.g., fixed number of views, image-only inputs) or require expensive post-processing to recover metric scale and camera poses. There is a lack of a unified, end-to-end model capable of handling diverse 3D reconstruction tasks—from uncalibrated Structure-from-Motion (SfM) to calibrated MVS and depth completion—while effectively leveraging heterogeneous inputs like camera intrinsics, poses, and depth maps.

Methodology: MapAnything

MapAnything is a unified, transformer-based feed-forward model designed to regress metric 3D scene geometry and camera poses directly from $N$ input images and optional geometric inputs.

1. Factored Scene Representation

The core insight of MapAnything is the use of a factored representation of multi-view scene geometry. Instead of predicting a coupled pointmap (where geometry and camera parameters are entangled), the model decomposes the output into:

• Local Ray Directions ($R_i$): Normalized vectors representing the direction of rays for each pixel.
• Ray Depths ($\tilde{D}_i$): Depth values along the rays in an "up-to-scale" space.
• Camera Poses ($\tilde{P}_i$): Represented as unit quaternions ($Q_i$) and up-to-scale translations ($\tilde{T}_i$) relative to the first view.
• Metric Scale Factor ($m$): A single global scalar that upgrades the local, up-to-scale reconstruction into a globally consistent metric frame.

The final metric 3D pointmap $X^{metric}_i$ is computed as $m \cdot (O_i \cdot (R_i \cdot \tilde{D}_i) + \tilde{T}_i)$, where $O_i$ is the rotation matrix derived from the quaternion.

2. Architecture

• Input Encoding: The model accepts $N$ RGB images and optional geometric inputs (ray directions, poses, depth maps). Images are encoded using DINOv2 (ViT-G). Geometric inputs are factorized and encoded via shallow convolutional encoders (for dense inputs like rays/depth) or MLPs (for global quantities like rotation/translation scales).
• Multi-View Transformer: The encoded features are flattened into tokens. A single learnable scale token is appended to the set of $N$ view tokens. A fixed reference view embedding is added to the first view to distinguish it. These tokens are processed by a 16-layer alternating-attention transformer (initialized from DINOv2 layers) to fuse information across views.
• Decoding Heads:
  • DPT Head: Decodes the $N$ view tokens into dense per-view outputs: ray directions, up-to-scale ray depths, confidence masks, and pointmap confidence maps.
  • Pose Head: An average-pooling-based convolutional head predicts unit quaternions and up-to-scale translations for all views.
  • Scale Head: An MLP processes the scale token to predict the global metric scaling factor $m$.

3. Training Strategy

MapAnything is trained end-to-end on a diverse aggregation of 13 datasets (indoor, outdoor, synthetic) covering over 12 different problem configurations.

• Flexible Input Augmentation: During training, geometric inputs (intrinsics, poses, depth) are provided with varying probabilities (e.g., 0.5 for each factor) to simulate different real-world scenarios where data may be partial or missing.
• Loss Functions: The model utilizes a composite loss function including:
  • Regression losses for rays, rotations, and translations.
  • Log-space losses for ray depths, pointmaps, and the metric scale factor to handle large dynamic ranges.
  • Confidence-weighted pointmap loss and binary cross-entropy for masks.
  • Normal and multi-scale gradient matching losses (applied to synthetic data).
• Scale Handling: To prevent gradients from the scale loss from corrupting the geometry, the metric scale factor is computed using a detached up-to-scale norm scaling factor ($z_{metric} = m \cdot \text{sg}(\tilde{z})$).

Key Contributions

1. Unified Feed-Forward Model: MapAnything supports over 12 different 3D reconstruction problem configurations (e.g., uncalibrated SfM, calibrated MVS, monocular depth, camera localization) in a single model, outperforming or matching specialist models.
2. Factored Scene Representation: By decoupling local geometry (rays/depth) from global pose and metric scale, the model enables flexible inputs and efficient prediction without costly post-processing or redundant predictions.
3. State-of-the-Art Performance: The model achieves superior results compared to other feed-forward baselines (e.g., DUSt3R, VGGT, MASt3R) and matches or exceeds expert models tailored for specific tasks.
4. Open Source Release: The authors release code for data processing, inference, and training, along with pre-trained models under permissive licenses (Apache 2.0 and CC BY-NC 4.0).

Results

The paper benchmarks MapAnything across multiple tasks and input configurations:

• Multi-View Dense Reconstruction: On ETH3D, ScanNet++ v2, and TartanAirV2-WB, MapAnything achieves state-of-the-art performance in pointmap, pose, depth, and ray direction estimation using only image inputs. It significantly outperforms baselines when provided with auxiliary geometric inputs (intrinsics, poses, depth).
• Two-View Reconstruction: MapAnything surpasses Pow3R (the only other two-view feed-forward method using priors) and image-only baselines across all input configurations.
• Single-View Calibration: Despite not being trained specifically on single images, MapAnything achieves state-of-the-art perspective calibration accuracy on ETH3D, ScanNet++ v2, and TartanAir.
• Depth Estimation: In the Robust-MVD benchmark, MapAnything outperforms specialized models like MASt3R and MUSt3R in multi-view metric depth estimation. With auxiliary inputs, it delivers competitive results against task-specific models.
• Efficiency: Ablation studies (Table 5) demonstrate that a single universally trained model performs equivalently to or better than multiple bespoke models trained for specific input configurations, despite having comparable computational costs to just two bespoke models.

Significance and Claims

The authors position MapAnything as a step toward a universal 3D reconstruction backbone. By unifying diverse tasks and input modalities into a single feed-forward pass, the model eliminates the need for task-specific tuning and complex optimization pipelines (like bundle adjustment) for many scenarios. The factored representation allows the model to effectively leverage partial annotations and heterogeneous inputs, making it robust for "in-the-wild" applications. The paper claims that this approach paves the way for future research into 3D/4D foundation models, though it acknowledges limitations in handling dynamic motion, explicit uncertainty quantification, and scalability to extremely large scenes.