💻 computer science

ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning

The paper introduces ARDepth, a novel auto-regressive framework for monocular depth estimation that progressively constructs depth representations across increasing spatial scales using Scale-Progressive Conditioning and Semantic-Aware Guidance to better capture hierarchical geometric structures compared to traditional global diffusion-based methods.

Original authors: Zijie Wang, Wei Zhang, Weiming Zhang, Xiao Tan, Weikai Chen, Xiaoxu Li, Guanbin Li

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zijie Wang, Wei Zhang, Weiming Zhang, Xiao Tan, Weikai Chen, Xiaoxu Li, Guanbin Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to draw a perfect 3D map of a room just by looking at a single photograph. For a while, the smartest computers tried to do this like a sculptor chipping away at a giant block of stone: they started with a noisy, blurry guess and slowly smoothed it out, step-by-step, until the shapes appeared. This method, called "diffusion," is like trying to fix a messy room by gently nudging every single item into place at the same time.

But the authors of this paper, ARDepth, noticed a problem with that approach. Real rooms aren't smooth blobs; they have sharp corners, thin table legs, and sudden jumps where a wall meets the floor. Trying to smooth everything out at once often blurs those sharp edges. It's like trying to draw a detailed map of a city by first drawing a giant, fuzzy circle and then hoping the streets and buildings magically sharpen up.

The Big Idea: Build Up, Don't Smooth Out
Instead of smoothing a blurry mess, ARDepth suggests building the depth map like a painter adding layers of detail, starting from the big picture and zooming in. They call this "auto-regressive generation." Think of it like assembling a LEGO set. You don't try to build the whole castle in one giant leap. First, you lay down the big, coarse baseplate (the general layout of the room). Then, you add the medium-sized walls. Finally, you snap on the tiny, intricate pieces like window frames and door handles.

The paper suggests that this "coarse-to-fine" approach is a much better way to handle the jagged, piece-by-piece nature of real-world geometry.

The Secret Weapons: Two Helpers
To make this LEGO-building process work perfectly, the team gave their model two special helpers:

The Scale-Progressive Guide (SPC): Imagine you are painting a mural. When you are painting the huge background sky, you need to see the whole canvas. But when you are painting a tiny flower in the corner, you need to zoom in close. The SPC module does exactly this. It feeds the model the right kind of visual clues at the right time. When the model is figuring out the big layout, it gets a "wide-angle" view of the image. When it's figuring out the tiny details, it gets a "zoomed-in" view. This ensures the model doesn't lose the big picture while trying to draw a sharp edge, and it doesn't get confused by tiny details while trying to place a whole building.
The Semantic Guide (SAG): Sometimes, a picture can be tricky. Is that dark patch a shadow or a hole? Is that blurry shape a person or a tree? The SAG module asks a super-smart AI assistant (a Vision-Language Model) to look at the picture and give a short, simple description of the scene, like "A sunny living room with a sofa in the center and a window on the left." This description acts like a compass, helping the model understand the story of the room so it doesn't get lost in the details. It's like having a tour guide whispering, "Remember, the table is in front of the window," so you don't accidentally put the table behind it.

What They Ruled Out
The authors explicitly argue against the idea that depth is just a smooth, continuous field that needs to be "denoised" (cleaned up) globally. They also show that just using a standard "flat" auto-regressive model (where you build the image one pixel after another in a long line, like reading a book) doesn't work well. Their experiments suggest that a flat approach captures the general shape but fails miserably at the sharp edges and thin structures, leaving the depth map looking blurry and wrong.

The Results: How Good Is It?
The team tested their new method on five different real-world benchmarks, including indoor rooms (NYUv2) and outdoor streets (KITTI). They compared it against the current champions, which are mostly the "smoothing" diffusion models.

The Numbers: On the NYUv2 dataset, the previous best diffusion model (Marigold) had an error score (AbsRel) of 5.5. ARDepth, using a similar amount of training data, lowered that error to 4.8. On the KITTI dataset, the error dropped from 9.9 to 8.4.
The "Zero-Shot" Test: They tested the model on data it had never seen before (zero-shot), and it still performed incredibly well, often beating models that were trained on massive amounts of data.
Scaling Up: When they increased the training data from 74,000 images to 174,000 images, the model got even better, showing that it can learn from more examples if they are available.

The Catch (Limitations)
While the results are promising, the paper is careful not to call it a perfect, solved problem. They admit that the current version of the model (the 8-billion parameter one) takes a bit longer to run—about 3.4 seconds per image on a powerful computer—compared to simpler, faster models. They also note that the "Semantic Guide" relies on an AI that might occasionally give a slightly noisy description, so they keep the text descriptions short and general to avoid confusion.

The Takeaway
The paper suggests that treating depth estimation as a structured, step-by-step construction project (building from coarse to fine) is a powerful alternative to the current "smooth and refine" methods. By combining this step-by-step building with smart visual and text-based guides, ARDepth creates depth maps that are sharper, more accurate, and better at handling tricky details like thin objects and sharp corners. It's not just a small tweak; it's a different way of thinking about how computers "see" the 3D world.

Technical Summary: ARDepth

Problem Statement
Monocular Depth Estimation (MDE) has recently been dominated by diffusion-based models, which treat depth recovery as a global refinement process via iterative denoising. While effective, the authors argue that this paradigm implicitly assumes depth is a globally smooth field, failing to explicitly reflect the piecewise, discontinuous, and scale-dependent nature of real-world scene geometry. Geometric structures—such as sharp occlusion boundaries, thin structures, and surface intersections—often emerge progressively across spatial scales rather than being uniformly refined. Furthermore, existing auto-regressive (AR) approaches applied to depth often use "flat" formulations that struggle to allocate sufficient capacity to near-field details while capturing the global layout, leading to a loss of fine-grained structural fidelity.

Methodology
The paper introduces ARDepth, a hierarchical auto-regressive framework that formulates depth estimation as a structured, scale-progressive generation process. Instead of uniform denoising, ARDepth constructs depth representations from coarse layouts to fine geometric details.

The architecture consists of three primary components:

Hierarchical Coarse-to-Fine Auto-Regression:
- The model utilizes a pretrained Visual Auto-Regressive (VAR) backbone.
- Depth maps are decomposed into a sequence of residual maps ( $R = \{r_1, ..., r_K\}$ ) with progressively increasing spatial resolutions.
- The transformer predicts these residuals sequentially. Coarse residuals encode global scene layout and dominant surfaces, while subsequent residuals capture high-frequency corrections like object boundaries and local discontinuities.
- During inference, the model generates residuals from coarse to fine, fusing them to produce the final full-resolution depth map.
Scale-Progressive Conditioning (SPC):
- To address the limitation of standard AR models that often rely on a single global visual feature, SPC injects multi-scale visual features at each generation stage.
- It constructs local feature tokens ( $v_k$ ) and a pooled global context ( $g_k$ ) from the input image.
- Using a cross-attention mechanism, the module adaptively fuses these local and global cues with the intermediate predicted residuals. This ensures that coarse stages receive global layout cues, while high-resolution stages are guided by localized, high-frequency visual evidence, improving spatial alignment and detail fidelity.
Semantic-Aware Guidance (SAG):
- To enhance global structural consistency, SAG introduces lightweight scene-level semantic priors.
- A multi-modal language model (VLM) generates a concise, image-specific description of the scene's global structure and object relationships (avoiding explicit pixel-level depth reasoning).
- This text is encoded and projected into a "start-of-sequence" (sos) token, which serves as the initial input for the transformer. This provides global semantic context to stabilize the generation process, particularly in textureless or occluded regions.

Key Contributions

Structured AR Formulation: The paper proposes formulating MDE as a scale-progressive auto-regressive generation task, moving away from the assumption of depth as a globally refined random field.
Stage-Aligned Conditioning: The introduction of SPC and SAG modules. SPC aligns visual cues with the hierarchical generation process to recover local details, while SAG provides global semantic priors to maintain scene coherence.
Comprehensive Validation: The authors provide extensive zero-shot, relative-depth, in-domain, and scaling studies to validate the approach.

Experimental Results

Zero-Shot Performance: On five real-world benchmarks (NYUv2, KITTI, ETH3D, ScanNet, DIODE), ARDepth achieves competitive or superior performance compared to state-of-the-art diffusion-based models (e.g., Marigold, GeoWizard) and significantly outperforms existing auto-regressive baselines (e.g., DepthART, VAR).
- For instance, on the 74K training setting, ARDepth improves AbsRel on NYUv2 from 5.5 (Marigold) to 4.8.
Ablation Studies:
- SPC: Removing SPC leads to significant performance degradation. The study confirms that adaptive attention-based fusion of local and global features is superior to simple concatenation or using only global context.
- SAG: Image-specific semantic descriptions (SAG Text) outperform generic prompts or blank text, confirming the value of compact scene-level guidance. However, overly detailed spatial descriptions can introduce noise.
- Hierarchy vs. Flat: A "Flat AR" baseline (single-stage raster-scan) performs substantially worse than the hierarchical formulation, demonstrating that coarse-to-fine factorization is critical for dense geometric prediction.
Scalability: The framework scales effectively with data (improving from 74K to 174K training samples) and model size (2B vs. 8B parameters). The 2B variant already outperforms Marigold, indicating the architectural design itself provides significant gains.
Qualitative Analysis: Visual comparisons show ARDepth produces sharper boundaries, better preserves thin structures, and maintains more coherent global layouts compared to diffusion-based models.

Significance and Claims
The paper claims that ARDepth validates auto-regressive generation as a promising alternative paradigm for geometric modeling. By explicitly modeling the hierarchical and scale-dependent organization of scene geometry, the approach achieves strong zero-shot generalization and structurally consistent predictions.

The authors position their work not as a replacement for diffusion models, but as evidence that with task-aware architectural design (specifically hierarchical generation and stage-aligned conditioning), auto-regressive frameworks can effectively capture dense geometric structures. They suggest this perspective offers a potential direction for extending auto-regressive foundation models (such as MLLMs) to include geometric perception capabilities, enabling unified modeling of spatial reasoning and multi-modal understanding.

Limitations
The authors acknowledge that the current 8B model incurs non-negligible inference costs due to sequential decoding, though a lighter 2B variant is available. They also note that SAG relies on off-the-shelf VLMs which may occasionally contain noisy spatial details, necessitating the use of compact scene-level descriptions rather than explicit depth reasoning. Finally, the current formulation is limited to single-image estimation, with video and multi-view extensions left for future work.

Technical Summary: ARDepth

More like this