🤖 AI

MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources

MetricAnything introduces a scalable pretraining framework that leverages a novel "Sparse Metric Prompt" to learn metric depth from 20 million noisy, heterogeneous 3D sources, establishing the first clear scaling trend in metric depth and achieving state-of-the-art performance across diverse tasks including monocular estimation, 3D reconstruction, and spatial intelligence in multimodal models.

Original authors: Baorui Ma, Jiahui Yang, Donglin Di, Xuancheng Zhang, Jianxun Cui, Hao Li, Yan Xie, Wei Chen

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Baorui Ma, Jiahui Yang, Donglin Di, Xuancheng Zhang, Jianxun Cui, Hao Li, Yan Xie, Wei Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Noisy Kitchen"

Imagine you want to teach a robot how to understand the real world in 3D—specifically, how far away things are (metric depth). To do this, you usually need a massive library of "textbooks" (datasets) showing pictures and their exact 3D distances.

The problem is that these textbooks are a mess.

Some are written by LiDAR (lasers), which are great but noisy and sparse (like a sketch with missing dots).
Some are reconstructed by computers from videos, which can have weird errors in shiny or blurry areas.
Some are rendered (computer-generated), which are perfect but look fake.
They all come from thousands of different cameras, each with its own quirks and biases.

Previous attempts to teach robots with this "noisy kitchen" of data failed because the noise confused the AI. They tried to build complex, custom rules for every type of noise, which didn't scale well. It was like trying to clean a messy kitchen by hand-scrubbing every single dish individually instead of using a dishwasher.

The Solution: "Metric Anything"

The authors created a new system called Metric Anything. Think of it as a universal "dishwasher" that can handle any kind of dirty dish (data source) without needing a special setting for each one.

Here is how it works, broken down into three simple steps:

1. The "Sparse Prompt" (The Magic Hint)

Instead of showing the AI the whole picture of the 3D world at once (which is too noisy), they give it a Sparse Metric Prompt.

The Analogy: Imagine you are trying to guess the shape of a hidden object in a box. Instead of letting you see the whole box, someone gives you a few random "hints" (dots) that tell you the exact distance of a few specific points.
How they do it: They take a 3D map, randomly hide most of it, and only show the AI a few scattered dots with their exact distances.
Why it works: This forces the AI to learn the logic of space (how objects relate to each other) rather than memorizing the specific noise patterns of a camera. It decouples "spatial reasoning" from "sensor bias."

2. The "Teacher" (The Pre-trained Model)

They fed this system a massive amount of data: 20 million image-depth pairs from over 10,000 different camera models.

The Analogy: This is like hiring a genius tutor who has read every book in the library, regardless of whether the books were written in English, French, or a made-up language.
The Result: This "Teacher" model learns to look at a picture and a few random distance hints, then fill in the rest of the 3D world perfectly. It gets better the more data you give it (a "scaling trend"), which was previously thought impossible for this type of task.

3. The "Student" (The Distilled Model)

The "Teacher" is great but requires those "hints" (prompts) to work. For many real-world jobs (like a self-driving car or a robot), you don't have those hints available.

The Analogy: The Teacher is a master chef who needs a recipe card. The Student is the chef's apprentice. The Teacher cooks thousands of meals using the recipe cards, and the apprentice watches closely, learning the technique without needing the cards.
The Result: They created a "Student" model that learned from the Teacher. Now, the Student can look at a plain photo and instantly know the exact 3D distances, without needing any hints or prompts.

What Can This Do? (The "Superpowers")

Because this system learned from such diverse and noisy data, it is incredibly robust. The paper shows it excels at:

Filling in the Blanks: If you give it a blurry or low-resolution depth map, it can "super-resolve" it to make it sharp and detailed.
Mixing Sensors: It can take a very sparse, noisy signal from a Radar (which is much sparser than LiDAR) and fuse it with a camera to create a perfect 3D map.
Fixing Cameras: It can look at a photo and guess the camera's settings (focal length) just by understanding the geometry of the scene.
Robotics & AI: When they gave this "spatial brain" to a robot (VLA) or a large language model (MLLM), the robot became much better at navigating rooms, estimating how far away a cup is, and planning paths. It stopped guessing and started measuring.

The Main Takeaway

The paper proves that more data + a simple, smart way to handle noise = a better 3D brain.

They didn't need to build complex, custom hardware or write special rules for every camera. They just needed a simple "hint-based" training method and a massive, messy dataset. The result is a model that understands the 3D world as well as (or better than) any previous system, and it works everywhere—from rainy streets to cartoon worlds to robot kitchens.

In short: They taught an AI to understand distance by showing it a million messy pictures with a few random clues, and now it can see the world in 3D better than ever before.

Technical Summary: Metric Anything

Problem Statement

While scaling has driven significant advances in vision foundation models, extending this paradigm to metric depth estimation has remained challenging. Unlike relative depth, which learns ordinal relationships and benefits from synthetic data and pseudo-label distillation, metric depth requires learning absolute, physically meaningful distances. The field has been bottlenecked by the heterogeneity of data sources. Real-world 3D annotations originate from diverse sensors (LiDAR, RGB-D, stereo) and reconstruction algorithms (SfM, MVS, SLAM), each introducing distinct noise patterns, hardware artifacts, and camera-dependent biases. This heterogeneity creates significant domain gaps, noisy supervision, and metric ambiguity, preventing the "data soup" scaling strategy from succeeding in metric depth. Previous attempts to address this via prompt-based methods have been limited to specific tasks, small datasets, and complex, hand-crafted prompt pipelines that introduce strong human priors.

Methodology

The authors introduce Metric Anything, a minimalist pretraining framework designed to learn metric depth from noisy, diverse 3D sources without task-specific architectures or manually engineered prompts.

1. Multi-Source Data Collection

The authors aggregated a dataset of approximately 20 million image–depth pairs spanning over 10,000 camera models. The data is categorized into three sources:

Reconstructed 3D Data: Depth maps from SfM, SLAM, MVS, and stereo techniques, often containing artifacts and missing regions.
Captured 3D Data: Point clouds from LiDAR, ToF, and RGB-D sensors, which are often sparse and noisy due to sensor limitations and environmental factors.
Rendered 3D Data: Synthetic depth maps from virtual engines, providing noise-free geometric supervision.
All samples are standardized into per-pixel metric depth maps with validity masks.

2. Pretraining via Sparse Metric Prompt

The core innovation is the Sparse Metric Prompt, a universal interface that decouples spatial reasoning from sensor biases.

Prompt Generation: Instead of complex simulation, prompts are created by randomly masking valid pixels (sampling 2,000–40,000 pixels) from the ground truth depth maps.
Prompt Preparation: To handle the heterogeneity of the raw prompts, the authors apply a Pixel-wise Depth Scale Alignment (PDSA) and Global Metric Depth Recovery (GMDR) using a pre-trained depth model to fill missing regions and align scales, creating a unified intermediate representation.
Architecture: The model uses a standard ViT backbone with a DPT decoder. Prompts are injected via a conditioned DPT head (adding only ~5% parameters) rather than modifying the backbone. This design avoids bespoke architectural changes, allowing the model to learn structural and metric understanding directly from data.
Loss Functions: The training objective combines Mean Absolute Error (MAE) and Scale-and-Shift-Invariant Mean Absolute Gradient Error (SSI-MAGE). For real-world data, a robust MAE is used, discarding the top 20% of largest errors to mitigate noise.

3. Prompt-Free Model Distillation

To enable deployment in scenarios without prompts, the authors distill the pre-trained "teacher" model into a prompt-free student.

Pseudo-Label Generation: The teacher model generates high-fidelity, dense metric depth maps for the entire dataset conditioned on sparse prompts. These pseudo-labels cover a wide depth range (near to far), unlike many real-world datasets biased toward short ranges.
Student Architecture: The student removes the prompt injection layers. Crucially, the authors invert the standard skip-connection scheme: deep, high-level ViT features are injected into deeper decoding layers, while shallow features feed into shallower layers. This leverages the high semantic consistency of the teacher's pseudo-labels.
Loss Function: A Distance-Balanced Inverse-Depth Loss is introduced. By operating in a log-space ( $D_{log} = 1 - \ln(D)/\ln(C)$ ), this loss preserves fine-grained sensitivity in near regions while extending effective supervision to long-distance areas, addressing the limitations of standard inverse-depth losses.

Key Contributions

Metric Anything Framework: A minimalist, scalable pretraining paradigm that uses Sparse Metric Prompts to learn metric depth from heterogeneous, noisy 3D sources without hand-crafted prompts or task-specific architectures.
Demonstrated Scaling: The aggregation of 20M diverse image-depth pairs reveals a clear scaling trend in metric depth estimation, a phenomenon previously unseen in this domain. Performance consistently improves as training data size increases.
Universal Generalization: Both the pre-trained model and the distilled prompt-free student achieve state-of-the-art (SOTA) performance across ten downstream tasks without task-specific engineering. These tasks include:
- Prompt-based: Depth completion, super-resolution, and Radar-camera fusion.
- Prompt-free: Monocular depth estimation, camera intrinsics recovery, multi-view 3D reconstruction, and VLA planning.
- MLLM Enhancement: Significantly boosting the spatial reasoning capabilities of Multimodal Large Language Models.

Results

Scaling Trend: Experiments show a consistent increase in zero-shot $\delta_1$ accuracy on Middlebury as training data scales from 2M to 20M pairs.
Prompt-Based Tasks: The model achieves SOTA zero-shot performance in depth super-resolution and completion across NYUv2, ETH3D, and KITTI, outperforming prior methods that rely on synthetic prompt simulation. In Radar-Camera depth estimation on nuScenes, the fine-tuned model nearly doubles the accuracy of its from-scratch counterpart and surpasses all prior fusion methods.
Prompt-Free Tasks: The distilled student model ranks 1st or 2nd across six diverse monocular depth estimation benchmarks (ETH3D, NuScenes, Sun-RGBD, etc.) in a strict zero-shot setting. It also achieves SOTA results in camera intrinsics recovery and multi-view 3D reconstruction.
Spatial Intelligence: When used as a visual encoder for MLLMs, the model significantly improves performance on the VSI-Bench, particularly in tasks requiring object size estimation, absolute distance measurement, and route planning.
Robustness: The model demonstrates strong generalization to unseen sensor configurations (e.g., different LiDAR-camera setups), extreme environmental conditions (night, rain, fog), and unseen visual domains (fisheye, panoramas, cartoons).

Significance and Claims

The paper posits that metric depth estimation can benefit from the same scaling laws that drive modern foundation models in NLP and 2D vision. By minimizing human prior design in prompts and relying on data-driven learning from heterogeneous sources, Metric Anything establishes a new path toward scalable and efficient real-world metric perception.

The authors align their findings with "The Bitter Lesson," arguing that general-purpose, data-driven methods systematically outperform hand-crafted designs. They claim to have solved the "data scarcity" issue in metric depth by effectively utilizing noisy, heterogeneous sources, thereby unlocking emergent capabilities and robust generalization. The work suggests that future research should shift focus from task-specific engineering to general pretraining paradigms that can adapt to diverse sensors and scenarios. The authors open-source the framework to support community research.