💻 computer science

Fractal Autoregressive Depth Estimation with Continuous Token Diffusion

This paper proposes a Fractal Visual Autoregressive Diffusion framework that addresses the challenges of monocular depth estimation by reformulating it as a coarse-to-fine, next-scale autoregressive process using a VCFR module for cross-modal conditioning, conditional denoising diffusion for continuous space modeling, and a fractal recursive architecture for computational efficiency.

Original authors: Jinchang Zhang, Xinrou Kang, Guoyu Lu

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinchang Zhang, Xinrou Kang, Guoyu Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to draw a detailed map of a mountain range, but you only have a single, flat photograph of it. You need to figure out how far away every single rock and tree is. This is what computers call Monocular Depth Estimation.

For a long time, computers tried to do this by guessing the distance of every single pixel one by one, like a painter filling in a canvas dot-by-dot. This was slow, prone to mistakes, and often resulted in "blocky" or jagged maps because the computer had to guess the exact number for every tiny spot.

This paper introduces a new, smarter way to do it called Fractal Autoregressive Depth Estimation. Here is how it works, broken down into simple concepts:

1. The "Zoom-Out" Strategy (Fractal Autoregression)

Instead of trying to draw the whole mountain map at once or pixel-by-pixel, the computer starts with a tiny, blurry sketch.

The Analogy: Imagine you are sculpting a statue. You don't start by carving the eyelashes. You start with a big block of stone, rough out the general shape of the head, then the nose, then the mouth, and finally the details.
The Paper's Method: The computer predicts a very low-resolution (blurry) depth map first. Then, it uses that blurry map as a "guide" to predict a slightly sharper map. It repeats this process, getting sharper and sharper, until it has a high-definition map.
The "Fractal" Twist: To save time and money, the computer doesn't use a different brain for every step. It uses the same small brain over and over again, just at different zoom levels. It's like using the same stamp to print a pattern that gets bigger and more detailed each time. This makes the process incredibly efficient.

2. The "Translator" (Visual-Conditioned Feature Refinement)

There is a big problem: The input is a color photo (RGB), but the output is a distance map (Depth). They speak different "languages." A red car in a photo doesn't tell the computer how far away it is just by looking at the color.

The Analogy: Imagine trying to translate a book from English to French, but you keep forgetting the English words as you write the French ones. You need a translator who holds the English page in one hand and the French page in the other, constantly checking them against each other.
The Paper's Method: They built a special module (VCFR) that acts as this translator. At every step of the "sculpting" process, it looks at the current depth guess and the original photo features together. It fuses them to ensure the computer doesn't lose the visual details (like the edge of a building) while trying to guess the distance.

3. The "Smooth Painter" vs. The "Pixelator" (Continuous Diffusion)

Old methods tried to force the computer to pick a specific number (like "5 meters" or "6 meters") for every pixel. This is like trying to draw a smooth curve using only square Lego bricks; it always looks a bit jagged.

The Analogy: Instead of forcing the computer to pick a specific Lego brick size, this paper lets the computer use liquid paint. It allows the depth to be any smooth value, not just a fixed number.
The Paper's Method: They use a technique called Diffusion. Think of it like starting with a canvas covered in static noise (like TV snow) and slowly "denoising" it until a clear image emerges. By doing this in "liquid" (continuous) space rather than "blocky" (discrete) space, the resulting depth map is much smoother and more accurate, especially for tricky areas like glass windows or fog.

4. The "Panel of Judges" (Uncertainty-Aware Consensus)

Even the best artists make mistakes sometimes. If you ask the computer to guess the depth once, it might get a little wobbly.

The Analogy: Imagine you are trying to guess the weight of a watermelon. If you ask one person, they might be off. But if you ask 10 people, weigh their guesses, throw out the crazy outliers, and take the average, you get a much better answer.
The Paper's Method: The computer runs the generation process multiple times (like asking 10 judges). It then looks at where all the judges agree and where they disagree.
- Agreement: If all 10 guesses say a tree is 10 meters away, the computer is very confident.
- Disagreement: If 5 say 5 meters and 5 say 15 meters, the computer marks that spot as "uncertain."
- This gives the computer a built-in "reliability meter" that tells us which parts of the map we can trust and which parts are shaky.

Summary

This paper proposes a system that:

Builds depth from blurry to sharp (like sculpting).
Reuses the same efficient brain for every step (Fractal).
Translates photo features into distance constantly (VCFR).
Paints with smooth liquid instead of blocky pixels (Diffusion).
Asks itself multiple times and averages the answer to be sure (Consensus).

The result is a depth map that is faster to generate, smoother to look at, and more accurate than previous methods, making it a huge step forward for robots and self-driving cars trying to understand the 3D world.

1. Problem Statement

Monocular Depth Estimation (MDE) aims to predict pixel-wise depth from a single RGB image. While Autoregressive (AR) models have shown promise in image generation, applying them directly to MDE faces three critical challenges:

Modality Mismatch: Difficulty in modeling the structural correspondence between RGB appearance and depth geometry.
Inefficiency: Pixel-wise generation leads to high serial inference costs and slow processing.
Quantization Error & Instability: Traditional AR models often discretize continuous depth values into tokens. This introduces quantization errors, causing loss of fine-grained geometric details and unstable training. Additionally, error propagation during sequential generation undermines prediction reliability.

2. Methodology

The authors propose a Fractal Visual Autoregressive Diffusion (F-VARD) framework that reformulates depth estimation as a coarse-to-fine next-scale autoregressive generation process. Instead of predicting pixel-by-pixel, the model predicts depth at increasing resolutions (scales).

Key Components:

Fractal Recursive Architecture:
- The model organizes scale-wise generators into a self-similar, recursive hierarchy ( $g_4 \to g_1$ ).
- Instead of using independent generators for each scale, a single base Visual Autoregressive (VAR) unit is reused recursively.
- Process: The process starts with a coarse latent token (global context) and progressively refines it to higher resolutions. Each stage $g_i$ takes the depth state from the previous coarser scale ( $z_i$ ) and generates a higher-resolution latent ( $z_{i+1}$ ).
- Benefit: This drastically reduces parameter count and inference cost while maintaining the ability to model complex hierarchical dependencies.
Visual-Conditioned Feature Refinement (VCFR):
- To address the RGB-depth modality gap, a VCFR module is inserted at each recursive level.
- It fuses multi-scale visual features (extracted from the input image) with the current depth prediction state.
- The output is a Visual-Depth Joint Token, which serves as a conditional input for the next scale's generation, ensuring structural alignment and detail recovery.
Continuous Token Diffusion (Conditional Denoising Diffusion):
- Rather than discretizing depth into tokens (which causes quantization artifacts), the model operates in continuous latent space.
- At each scale, a conditional denoising diffusion objective models the probability distribution of the depth latent given the previous scale and visual conditions.
- Advantage: This avoids quantization errors, preserves smooth geometric transitions, and provides a more flexible representation for ambiguous regions (e.g., occlusions) compared to point-wise regression.
Uncertainty-Aware Robust Consensus Aggregation (URCA):
- To improve inference stability, the model performs multi-sample inference (generating $N$ stochastic depth maps).
- Alignment: Samples are first aligned via affine transformation (scale and shift) to a shared reference space.
- Consensus Fusion: A robust optimization minimizes an energy function that enforces consistency across stochastic samples and consistency along the coarse-to-fine generation trajectory.
- Output: Produces a fused depth map and a pixel-wise uncertainty proxy (reliability indicator) without requiring additional training.

3. Key Contributions

Fractal AR Framework: A novel recursive framework that reformulates MDE as a next-scale generation process, reusing a base generator to achieve an optimal efficiency-performance trade-off.
VCFR Module: A mechanism to construct visual-depth joint tokens, significantly improving cross-modal conditioning and structural alignment.
Continuous Diffusion Modeling: Integration of conditional denoising diffusion into the AR process to model depth distributions in continuous space, eliminating quantization errors inherent in discrete tokenization.
Robust Consensus Aggregation: An inference-time strategy that fuses multiple samples to enhance stability and provide a practical reliability metric for downstream tasks.

4. Experimental Results

The method was evaluated on standard benchmarks: KITTI (outdoor) and NYUv2 (indoor).

Performance:
- KITTI: Achieved state-of-the-art (SOTA) results among supervised methods, with an AbsRel of 0.044 and RMSE of 1.712, outperforming strong baselines like DepthAnything, ZoeDepth, and other diffusion-based models.
- NYUv2: Achieved an AbsRel of 0.049 and RMSE of 0.197, surpassing previous AR and diffusion methods.
Efficiency:
- While slower than lightweight CNNs (e.g., Monodepth2), the method is significantly faster than other diffusion-based approaches (e.g., Marigold, DiffusionDepth) due to the fractal reuse of modules.
- Runtime on 1024x320 input: 64ms (vs. 9.88s for Marigold).
Ablation Studies:
- Replacing standard AR with VAR reduced inference cost by ~98% while improving accuracy.
- Switching from discrete tokenization to continuous diffusion improved RMSE significantly.
- Multi-sample inference (increasing $N$ ) consistently improved accuracy and reduced uncertainty, with diminishing returns observed after $N=8$ .

5. Significance

This paper bridges the gap between autoregressive generation and diffusion modeling for dense prediction tasks.

Theoretical Shift: It demonstrates that autoregressive modeling does not require discrete state spaces; continuous diffusion can be effectively integrated into a scale-wise AR framework.
Practical Impact: By solving the computational bottleneck of multi-scale generation through fractal recursion and the accuracy bottleneck of quantization through continuous diffusion, it sets a new standard for high-fidelity, efficient monocular depth estimation.
Reliability: The inclusion of an uncertainty proxy makes the model more suitable for safety-critical applications (e.g., autonomous driving) where knowing the reliability of a prediction is as important as the prediction itself.