💻 computer science

DVD: Deterministic Video Depth Estimation with Generative Priors

This paper introduces DVD, a novel framework that deterministically adapts pre-trained video diffusion models into single-pass depth regressors using three core designs to achieve state-of-the-art zero-shot performance while requiring 163 times less task-specific data than existing baselines.

Original authors: Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Jing He, Zixin Zhang, Haodong Li, Yihao Liang, Kanghao Chen, Bin Ren, Xu Zheng, Shuai Yang, Kun Zhou, Yinchuan Li, Nicu Sebe, Ying-Cong Chen

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Jing He, Zixin Zhang, Haodong Li, Yihao Liang, Kanghao Chen, Bin Ren, Xu Zheng, Shuai Yang, Kun Zhou, Yinchuan Li, Nicu Sebe, Ying-Cong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D map of a moving world just by looking at a video. This is called Video Depth Estimation. It's like trying to figure out how far away every object is in a movie, frame by frame, so a robot or a self-driving car can understand the space around it.

For a long time, scientists have been stuck between two bad options, like a car stuck between a rock and a hard place:

The "Dreamer" (Generative Models): These models are like creative artists. They look at a video and "imagine" the 3D shape. They are great at understanding the general vibe and can work on any video without extra training. But, because they are "dreaming," they sometimes get confused. They might hallucinate (make things up), like suddenly making a wall wobble or a car float away. They are inconsistent.
The "Accountant" (Discriminative Models): These models are like strict accountants. They are very precise and consistent, but they only know what they've been taught. To learn the rules of the road, they need to study millions of labeled examples (like a student memorizing a textbook). If they see something weird or blurry, they get confused and might think a shadow is a wall. They are rigid and need massive amounts of data.

Enter DVD (Deterministic Video Depth Estimation).

The authors of this paper created a new framework called DVD. Think of DVD as a super-smart translator that takes the "Dreamer's" creativity and the "Accountant's" precision and combines them into one perfect system.

Here is how DVD works, using simple analogies:

1. The "Structural Anchor" (The Metronome)

Imagine the "Dreamer" model is a musician playing a jazz solo. It's free and creative, but sometimes it loses the beat.

The Problem: When you try to force this musician to play a specific note (predict depth), they might play it too softly (blurry) or too loudly (chaotic).
The DVD Fix: DVD introduces a Metronome (called a Timestep Anchor). It doesn't change the music; it just tells the musician exactly when to play to keep the rhythm steady. By locking the model to a specific "beat" in its training, DVD ensures the 3D map stays stable (doesn't wobble) but still keeps all the fine details (like the sharp edges of a table).

2. Latent Manifold Rectification (The "Sharpener")

Imagine you are trying to draw a picture of a moving car, but your pencil is too soft. The lines get smudged, and the car looks like a blurry blob. This is called "mean collapse"—the model averages everything out until nothing is sharp.

The Problem: Standard AI tends to smooth things out too much, losing the crisp edges of buildings or the fast movement of a runner.
The DVD Fix: DVD uses a Magic Sharpener (called Latent Manifold Rectification). Instead of just asking the AI "What is the average depth?", it asks, "How is the depth changing right here?" It forces the AI to pay attention to the edges and the speed of movement. It's like taking a blurry photo and using software to sharpen the edges and make the motion look smooth and natural, not jittery.

3. Global Affine Coherence (The "Seamless Stitcher")

Imagine you are watching a very long movie, but your screen is small. You have to watch it in chunks (windows).

The Problem: With old "Dreamer" models, when you switch from one chunk of the movie to the next, the size of the objects might suddenly change. A car might look tiny in the first chunk and huge in the second. It's like a video game where the world stretches and shrinks randomly.
The DVD Fix: DVD discovered a secret rule: the "stretching" between chunks is always predictable, like a simple zoom or a slide. It's not random chaos; it's just a math problem. DVD uses a Seamless Stitcher that automatically calculates the perfect zoom and slide to glue the chunks together. This means you can watch a 1-hour video, and the 3D map will stay consistent from start to finish without the world warping.

Why is this a Big Deal?

It's a Data Saver: The "Accountant" models need to study millions of videos to learn. DVD can learn the same skills by looking at 163 times less data. It's like a genius student who can learn a whole year's curriculum by reading just one chapter because they already understand the underlying logic.
It's Fast: Because it doesn't have to "dream" or "guess" multiple times (like the old models), it gives you the answer in one single pass. It's instant.
It Works Everywhere: It works on real-world videos, robot footage, and even weird AI-generated videos, all without needing to be retrained.

In Summary:
DVD takes a powerful video AI that usually "hallucinates" and turns it into a precise, reliable 3D map maker. It does this by giving the AI a steady rhythm, sharpening its blurry edges, and mathematically stitching long videos together so nothing ever looks weird or distorted. It's the best of both worlds: the creativity of a dreamer with the reliability of an accountant.

1. Problem Statement

Video depth estimation faces a fundamental trade-off between two existing paradigms:

Generative Models (e.g., DepthCrafter): Leverage pre-trained video diffusion models to capture rich spatio-temporal priors, enabling strong zero-shot generalization. However, they rely on stochastic sampling, which introduces temporal uncertainties, geometric hallucinations, and scale drift, making them unreliable for precise geometric tasks.
Discriminative Models (e.g., Video Depth Anything): Provide deterministic, efficient outputs but suffer from semantic ambiguity. They often misinterpret motion blur or textureless regions as structural boundaries and require massive amounts of labeled data to resolve these ambiguities, hindering scalability and zero-shot performance.

The Core Question: Can a framework be designed that combines the structural stability of discriminative models with the rich priors of generative approaches, while remaining efficient and scalable?

2. Methodology

The authors propose DVD (Deterministic Video Depth), a framework that adapts pre-trained video diffusion models into single-pass deterministic regressors. Instead of iterative denoising, DVD learns a direct mapping from RGB latents to depth latents.

The framework relies on three core technical designs:

A. Timestep as a Structural Anchor

In traditional diffusion, the timestep $t$ represents noise levels. DVD repurposes $t$ as a structural anchor ( $\tau_0$ ) to balance the trade-off between global stability and high-frequency details.

Mechanism: Instead of sampling $t$ dynamically, DVD fixes the timestep to an optimal value (empirically $\tau = 0.5$ ).
Rationale: Early timesteps ( $\tau \to 1.0$ ) bias the model toward low-frequency global structures (stable but blurry), while late timesteps ( $\tau \to 0.0$ ) focus on high-frequency details (sharp but unstable). Anchoring at the mid-range provides a persistent conditioning code that balances geometric coherence with detail recovery.

B. Latent Manifold Rectification (LMR)

Deterministic regression often suffers from "mean collapse," where the model predicts the conditional expectation, washing out high-frequency details and causing motion flickering in ambiguous regions.

Solution: DVD introduces a parameter-free supervision mechanism that enforces differential constraints in the latent space.
Components:
1. Spatial Rectification: Aligns spatial gradients ( $\nabla_h, \nabla_w$ ) between predicted and ground-truth latents to preserve sharp boundaries.
2. Temporal Rectification: Aligns temporal flows ( $\nabla_t$ ) to ensure coherent motion dynamics and prevent flickering.
Effect: This restores the differential geometry erased by standard $L_2$ regression, recovering sharp edges and consistent motion without heavy auxiliary modules.

C. Global Affine Coherence

For long-video inference, sliding-window approaches are necessary. Generative models often suffer from non-linear scale drift between windows.

Discovery: DVD's deterministic backbone exhibits Global Affine Coherence. The discrepancy between adjacent windows is not random but can be approximated by a linear scale-shift transformation (affine transformation).
Implementation: A lightweight, parameter-free affine-alignment strategy is used during inference. Overlapping windows are aligned using a closed-form least-squares solver to estimate a global scale ( $s$ ) and shift ( $t$ ).
Benefit: This enables seamless, long-duration video inference without complex feature matching or recurrent temporal modules, effectively eliminating stochastic scale drift.

D. Training Strategy

DVD employs an Image-Video Joint Training strategy. It trains on both static images (to preserve high-frequency spatial details) and dynamic video sequences (to enforce temporal coherence). This prevents the "catastrophic forgetting" of spatial details often seen when fine-tuning exclusively on video data.

3. Key Contributions

Paradigm Shift: DVD is the first framework to deterministically adapt pre-trained video diffusion models into single-pass depth regressors, breaking the ambiguity-hallucination dilemma.
Novel Mechanisms:
- Structural Anchor: Repurposing diffusion timesteps to control geometric operating regimes.
- LMR: A differential constraint mechanism to combat mean collapse and restore sharp boundaries.
- Global Affine Coherence: A mathematical property enabling robust, drift-free long-video inference.
Data Efficiency: DVD achieves state-of-the-art (SOTA) performance using 163 $\times$ less task-specific data than leading baselines (e.g., trained on 367K frames vs. VDA's 60M frames).
Open Source: The authors fully release the training suite and pipeline to the community.

4. Experimental Results

Extensive experiments were conducted on benchmarks including KITTI, ScanNet, Bonn, and Sintel.

Zero-Shot Performance: DVD achieves SOTA results across all benchmarks.
- ScanNet: AbsRel of 5.5 (vs. 5.8 for VDA).
- KITTI: AbsRel of 6.7 (vs. 7.2 for VDA).
- Boundary Metrics: Significantly outperforms baselines in Boundary F1 (B-F1) and Recall, proving superior preservation of fine-grained geometric details.
Long-Video Stability: On long sequences (>200 frames), DVD maintains consistent global scale and structure, whereas generative baselines (e.g., DepthCrafter) suffer from severe scale drift and flickering.
Efficiency:
- Inference Speed: DVD operates at speeds comparable to efficient discriminative models (single-pass) and is significantly faster than iterative generative models.
- Data Efficiency: Achieves superior accuracy with less than 1% of the training data required by Video Depth Anything (VDA).
Generalization: Demonstrates robust zero-shot generalization to diverse open-world domains, including animations, video games, and AI-generated content.

5. Significance

DVD represents a significant advancement in 3D scene understanding by successfully bridging the gap between generative and discriminative paradigms.

Scalability: It proves that foundation models can be adapted for precise geometric tasks without massive labeled datasets, lowering the barrier for deployment in data-scarce scenarios.
Reliability: By eliminating stochastic hallucinations, it makes video depth estimation viable for safety-critical applications like autonomous driving and robotics.
Efficiency: The combination of single-pass inference and minimal data requirements establishes a new, highly efficient standard for dynamic 3D perception.

In summary, DVD transforms video foundation models from "world simulators" into precise "geometric regressors," offering a robust, scalable, and data-efficient solution for video depth estimation.