💻 computer science

Unlocking Dense Metric Depth Estimation in VLMs

The paper introduces DepthVLM, a framework that transforms a single Vision-Language Model into a native, efficient dense metric depth predictor through a lightweight depth head and unified training, significantly outperforming existing models in both 3D geometry recovery and spatial reasoning while preserving multimodal capabilities.

Original authors: Hanxun Yu, Xuan Qu, Yuxin Wang, Jianke Zhu, Lei ke

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanxun Yu, Xuan Qu, Yuxin Wang, Jianke Zhu, Lei ke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant (a Vision-Language Model, or VLM) that is great at looking at a picture and telling you a story about it, or answering questions like "What color is the car?" or "Is the dog happy?"

However, this robot has a blind spot: it doesn't really understand depth. It sees a flat picture, not a 3D world. It can't tell you exactly how many meters away a chair is, or if a tree is in front of a building.

The paper introduces a new system called DepthVLM that fixes this blind spot without breaking the robot's ability to chat and understand images. Here is how they did it, using simple analogies:

1. The Problem: The "Text-Only" Robot

Most current smart robots are trained like students who only learn from reading books. They see an image, but they only "speak" in text.

The Issue: If you ask them, "How far is that cup?", they have to guess and write a sentence. They don't actually calculate the distance for every single pixel in the image.
The Old Fix: Previous attempts tried to glue a separate "depth calculator" onto the robot. But this was like hiring a second, clumsy intern to do the math. The main robot would pass the image to the intern, who would make mistakes, and then pass the answer back. The errors piled up, and it was very slow.

2. The Solution: Giving the Robot a "Depth Sense"

The authors, Hanxun Yu and his team, asked a simple question: Can we teach the robot to see depth directly, using the same brain it uses for talking?

They built DepthVLM by adding a tiny, lightweight "depth head" (a special tool) to the robot's existing brain.

The Analogy: Think of the robot's brain as a master chef who is already great at cooking (understanding images) and talking (generating text). Instead of hiring a separate sous-chef to measure ingredients, they just gave the master chef a new, high-tech measuring tape. Now, the chef can chop vegetables and measure them at the exact same time, without slowing down.

3. How It Works: The Two-Step Training

You can't just hand a new tool to a master chef and expect them to use it perfectly immediately. The paper uses a two-stage training strategy:

Stage 1 (The Drill): They froze the chef's brain (so they wouldn't forget how to cook) and only trained the new measuring tape. This taught the robot how to guess distances without messing up its existing knowledge.
Stage 2 (The Practice): They unfroze the brain and let the robot practice using the measuring tape while talking. This helped the robot learn how to combine "seeing depth" with "understanding the scene" seamlessly.

4. The Magic Trick: One Pass, Instant Results

Previous methods were incredibly slow.

The Old Way (DepthLM): To measure the distance of a whole room, the old robot had to ask, "How far is pixel 1?" then "How far is pixel 2?" all the way to pixel 100,000. It took hours (13 hours in some tests!).
The New Way (DepthVLM): The new robot looks at the whole picture once and instantly spits out a full, detailed 3D map of the room in less than half a second (0.42s). It's like going from counting every grain of sand on a beach one by one to taking a single photo of the whole beach.

5. The Results: Better Than Specialists

The team tested DepthVLM on a huge variety of scenes (indoor rooms, outdoor streets, driving scenes).

Beating the Chatbots: It crushed other smart robots (like GPT-5.5) at guessing distances. While other robots were guessing wildly, DepthVLM was accurate.
Beating the Experts: Surprisingly, this "all-in-one" robot was even better at measuring depth than robots built only for measuring depth (specialized vision models).
Keeping the Personality: Crucially, adding this depth sense didn't make the robot "dumb" at other tasks. It could still write poems, answer questions, and understand complex scenes just as well as before.

6. Why This Matters (According to the Paper)

The paper claims this is a step toward a Unified Foundation Model.

The Analogy: Imagine a Swiss Army Knife. Before, you had a separate tool for cutting, a separate tool for screwing, and a separate tool for measuring. They were all separate. DepthVLM is like a single tool that can do all three perfectly at once.
The Benefit: It allows robots to not just "see" a 3D world but to "reason" about it. For example, it can look at a photo and say, "The trash can is closer than the sink," or "The washing machine is about 1 meter tall," all in one go.

In Summary:
The paper presents a way to turn a standard "chatting robot" into a "3D-aware robot" by adding a tiny, efficient depth sensor to its brain. It learns to measure the world in 3D instantly, without needing a second robot to help, and without forgetting how to talk. It's faster, more accurate, and more versatile than anything that came before it.

Technical Summary: DepthVLM

Problem Statement

Vision-Language Models (VLMs) have achieved remarkable success in 2D tasks such as visual grounding and image captioning but remain fundamentally limited in 3D understanding. A primary bottleneck is the prevailing text-only supervision paradigm, where visual inputs are consumed to generate autoregressive text outputs. This design inherently under-constrains fine-grained visual perception and prevents the explicit modeling of dense scene geometry.

Existing approaches to bridge this gap suffer from significant drawbacks:

Knowledge Distillation Pipelines: Methods that inject geometric signals (e.g., depth maps) from external pretrained 3D models rely on error accumulation and lack native integration.
Inefficient Direct Prediction: Recent attempts to enable VLMs to predict geometry directly face efficiency or quality trade-offs. For instance, DepthLM achieves high accuracy but requires per-pixel queries, resulting in prohibitively slow inference ($O(HW)$). Youtu-VL enables full-image prediction in one pass but produces coarse token-level outputs requiring post-hoc interpolation. Furthermore, many of these methods degrade the VLM's general multimodal capabilities due to text-heavy supervision or from-scratch training requirements.

The core challenge addressed by this paper is: Can a VLM serve as a native dense geometry predictor with minimal architectural changes while preserving its general multimodal capability?

Methodology: DepthVLM

The authors propose DepthVLM, a framework that transforms a single VLM backbone into a unified foundation model capable of generating both dense metric depth maps and language responses in a single forward pass.

1. Unified Architecture

DepthVLM extends the standard VLM architecture by attaching a lightweight depth head to the Large Language Model (LLM) backbone.

Feature Extraction: Instead of relying solely on the final LLM hidden states, the model extracts a multi-scale feature pyramid. This includes three intermediate layers from the Vision Transformer (ViT) encoder and the final hidden states of the LLM at image-token positions.
Depth Head Design: Inspired by DPT (Vision Transformers for Dense Prediction), the head processes these multi-scale features. Crucially, because VLM visual tokens are already downsampled by patch mergers, the authors avoid further downsampling. Instead, they construct a bottom-up pyramid via upsampling, assigning higher spatial resolution to earlier ViT layers.
Output: The fused features are decoded into a dense, full-resolution metric depth map ( $\hat{D} \in \mathbb{R}^{H \times W}$ ) alongside the text response ( $\hat{T}$ ). A Softplus activation ensures strictly positive depth values.

2. Two-Stage Training Strategy

To introduce dense geometry prediction without compromising the VLM's pre-trained multimodal understanding, the authors employ a two-stage training schedule:

Stage 1 (Depth Head Initialization): The VLM backbone is frozen. Only the newly added depth head is trained using a scale-invariant logarithmic (SILog) loss on metric depth data. This establishes initial depth prediction capabilities without disrupting the pretrained knowledge.
Stage 2 (End-to-End Fine-Tuning): The LLM backbone is unfrozen, and the entire model is fine-tuned end-to-end on a mixture of instruction-following data. The objective function combines the standard autoregressive language modeling loss ( $L_{text}$ ) and the depth loss ( $L_{depth}$ ), balanced by a factor $\alpha$ .

3. Data Curation and Normalization

Focal-Length Normalization: To mitigate camera-induced scale ambiguity across heterogeneous datasets (indoor vs. outdoor, different sensors), all images are rescaled to a unified canonical focal length ( $f_c$ ). This aligns projective geometry and enforces a consistent pixel-to-metric mapping.
DepthVLM-Bench: The authors introduce a unified benchmark aggregating 8 training datasets (covering indoor and outdoor scenes) and 9 evaluation datasets. This allows for fair comparison between VLMs and specialized pure vision models under a consistent protocol.

Key Contributions

Native Dense Prediction: The paper demonstrates that a VLM can serve as a native dense geometry predictor. By attaching a lightweight head, DepthVLM achieves dense pixel-level depth estimation without altering the text generation pathway.
Efficiency and Capability Preservation: Unlike prior methods that degrade general capabilities or suffer from slow inference, DepthVLM achieves $O(1)$ inference cost (single forward pass for full resolution) and preserves the VLM's original multimodal performance through a two-stage training strategy.
Unified Benchmark: The introduction of DepthVLM-Bench enables effective training and direct, fair comparison between VLMs and specialized pure vision models on metric depth estimation.

Experimental Results

Extensive experiments across diverse datasets (Waymo, NuScenes, ETH3D, sunRGBD, etc.) highlight the following:

Superiority over VLMs: DepthVLM significantly outperforms existing VLMs, including general-purpose models (e.g., GPT-5.5, Qwen3-VL) and spatial-enhanced VLMs. While GPT-5.5 achieves a $\delta_1$ accuracy of only ~0.41 on average, DepthVLM-4B reaches 0.868.
Surpassing Pure Vision Models: Despite being a unified model, DepthVLM surpasses state-of-the-art specialized pure vision models. For example, it outperforms DepthAnythingV3 (0.877 vs. 0.884 average $\delta_1$ ) and Metric3Dv2 (0.812 vs. 0.884).
Inference Efficiency: DepthVLM is substantially faster than prior VLM-based approaches. It generates a 256×192 depth map in 0.42s, compared to 2.48s for Youtu-VL and 13 hours for DepthLM (due to per-pixel queries).
Multimodal Capability: Evaluation on general visual benchmarks (MMBench, OCRBench, POPE) shows that DepthVLM matches or slightly improves upon its base VLM backbone, confirming that the addition of the depth head does not degrade general multimodal understanding.
3D Spatial Reasoning: The model demonstrates improved performance on complex downstream 3D tasks, such as point depth estimation, 3D object dimension estimation, and depth order ranking, suggesting that native dense geometry prediction serves as a solid foundation for high-level spatial reasoning.

Significance and Claims

The paper claims that DepthVLM represents a step toward a truly unified foundation model that jointly excels at low-level dense geometry prediction and high-level multimodal understanding.

Paradigm Shift: It challenges the notion that dense geometry prediction requires separate, specialized vision models or inefficient query-based VLMs. Instead, it shows that geometry can emerge natively within a single VLM backbone.
Practicality: By achieving high accuracy with minimal architectural changes and preserving general capabilities, DepthVLM offers a practical recipe for integrating 3D perception into existing VLM ecosystems.
Limitations: The authors modestly note that the current work focuses specifically on dense metric depth estimation. Broader 3D perception tasks, such as object detection and pose estimation, remain future work.

In summary, DepthVLM provides a lightweight, efficient, and effective solution for unlocking dense metric depth estimation in VLMs, bridging the gap between 2D multimodal understanding and 3D spatial intelligence.