💻 computer science

Depth Anything V4: Dynamic 4D Scene Reconstruction via Riemannian Flow Matching on 4D Gaussian Splatting

Depth Anything V4 (DAV4) introduces a novel framework for monocular dynamic 4D scene reconstruction that leverages Riemannian Flow Matching on 4D Gaussian Splatting parameters to ensure valid intermediate states and achieve superior performance without human-annotated depth labels.

Original authors: Jiaming Fan, Jian Lu, Jinling Jia, Chenbin Zhang

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiaming Fan, Jian Lu, Jinling Jia, Chenbin Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to capture a living room as it changes over time: a cat darts across the floor, a curtain sways in a draft, and a person walks through the frame. A standard photograph freezes a single instant, but a video only shows us a flat, two-dimensional sequence of images. For decades, scientists have struggled to turn those flat video frames into a true, three-dimensional model that moves and breathes like the real world. This is the challenge of dynamic 4D reconstruction. The goal is to build a digital twin of a scene that not only has depth but also understands how objects deform, rotate, and shift as time passes. To do this, researchers often use a technique called Gaussian Splatting, which represents a scene not as a solid mesh, but as a cloud of millions of tiny, colored, semi-transparent ellipses. Each of these tiny shapes holds specific information about its position, size, color, and how much it blocks the view. When you look at the scene from a new angle, the computer blends these shapes together to create a new image. The difficulty arises when these shapes need to move and change shape over time; if the computer guesses the wrong way to move them, the math breaks, and the 3D model collapses into nonsense.

A team of researchers has introduced a new system called Depth Anything V4, designed to solve this specific problem of creating moving 3D worlds from a single video camera. Previous versions of this technology were incredibly fast, capable of estimating depth in a fraction of a second, but they were built for static scenes or simple depth maps. They did not handle the complex, continuous motion of a dynamic environment well. The new approach prioritizes accuracy and consistency over raw speed, aiming to create high-quality 3D archives suitable for offline use, such as preserving historical sites or creating detailed virtual environments. The core innovation lies in how the computer learns to predict the movement of those millions of tiny 3D shapes. Instead of treating the movement as a simple straight line on a flat grid, the researchers realized that the rules governing these shapes are more like the rules of a curved surface. For instance, the size of a shape can only be positive, and its rotation must follow specific circular rules; if you try to move these values in a straight line, you might accidentally create a shape with a negative size or a broken rotation, which is physically impossible.

To navigate this curved mathematical landscape, the team developed a method called Riemannian Flow Matching. Think of it as a guide that knows the terrain perfectly. While a standard guide might try to walk in a straight line and then force the walker back onto the path if they step off, this new guide plans the route entirely along the curved surface itself. This ensures that every single step the computer takes during its learning process remains valid and physically possible. The researchers tested this by comparing their new system against a standard, rigid computer program that used the same data and the same amount of time to process the video. The standard program managed to reconstruct the scene with a certain level of accuracy, but the new system, using the curved-path guide, improved that accuracy significantly. The researchers isolated this improvement to prove it came from the new method and not just from having more data or extra processing time. They found that the new approach alone added a measurable boost to the quality of the reconstruction, confirming that respecting the curved nature of the data is essential for high-fidelity results.

The system works by first analyzing the video frames to understand the scene's structure and the motion of objects. It then uses a learned "prior," which is essentially a deep understanding of how 3D shapes typically behave, to generate a rough 3D model of the entire video sequence. This initial model is then refined through a process called test-time optimization, where the computer makes small adjustments to the model to ensure it matches the video frames perfectly. The researchers showed that their system could produce a high-quality 3D reconstruction in about 420 milliseconds per scene. While this is slower than the fastest existing tools, which take only 38 milliseconds, it is fast enough for offline applications where quality matters more than instant results. Furthermore, the researchers demonstrated that if this system were used to process a large number of scenes, such as ten thousand, the initial time spent training the system would be spread out, making the cost per scene very competitive with other methods that require hours of optimization for every single video.

One of the most compelling aspects of this work is how it handles uncertainty. In many computer vision tasks, the system gives an answer without admitting when it is unsure. Depth Anything V4, however, can tell you how confident it is in its reconstruction. By running the generation process multiple times with slight variations, the system can identify areas where the result is unstable, such as around fast-moving objects or in areas where the video is blurry. The researchers found that the system is very good at flagging these difficult spots, showing high uncertainty exactly where the visual information is ambiguous. This is crucial for applications where safety or precision is required, as it prevents the system from confidently presenting a wrong answer. The team also verified that their method produces fewer "invalid" 3D shapes compared to older methods that try to force straight-line math onto curved problems. In their tests, the new method produced valid shapes in nearly every single case, whereas older methods occasionally produced broken or nonsensical shapes that had to be discarded.

The study also addressed the importance of time in the learning process. The researchers tested what would happen if the system ignored the specific moment in the video and tried to learn a single, average way for the scene to move. They found that without this time awareness, the system struggled to keep the moving objects consistent from one frame to the next, resulting in a jittery and unstable 3D model. By explicitly teaching the system to pay attention to the time index of each frame, the model learned to track the flow of motion accurately, leading to a much smoother and more realistic reconstruction. This confirms that for dynamic scenes, understanding the passage of time is not just a minor detail but a fundamental requirement for success. The researchers concluded that while their system is not yet fast enough for real-time robotics or autonomous driving, it represents a significant step forward for creating high-quality, probabilistic 3D models of the world. It offers a way to capture the complexity of moving scenes with a level of detail and mathematical correctness that was previously out of reach, paving the way for better digital archives and more immersive virtual experiences.

Technical Summary: Depth Anything V4 (DAV4)

1. Problem Statement

The paper addresses the challenge of dynamic 4D scene reconstruction from monocular video. While the preceding Depth Anything series (V1–V3) established state-of-the-art performance in feed-forward monocular depth estimation, these models prioritize speed and produce point estimates without temporal modeling, relying on a discriminative regression paradigm.

Depth Anything V4 (DAV4) shifts the focus from speed to volumetric fidelity and temporal consistency, targeting offline content creation and 3D archiving. The core technical challenge lies in generating valid 4D Gaussian Splatting (4DGS) parameters (scale, rotation, opacity, position) that evolve over time. Standard Euclidean interpolation methods often produce invalid intermediate states (e.g., negative scales or non-unit quaternions) when traversing the non-Euclidean manifolds inherent to these parameters, necessitating post-hoc projections that degrade reconstruction quality.

2. Methodology

2.1 Core Framework: Riemannian Flow Matching (RFM)

The central innovation of DAV4 is the application of Riemannian Flow Matching (RFM) to the parameters of 4D Gaussian Splatting. Unlike standard flow matching that operates in Euclidean space, RFM defines probability paths directly on the non-Euclidean manifolds where Gaussian parameters reside:

Scale ( $s$ ): Modeled on $\mathbb{R}^3_{>0}$ using log-transforms and geodesic interpolation ( $s_t = s_z^{1-t} s_1^t$ ).
Rotation ( $q$ ): Modeled on $SO(3)$ using geodesic interpolation via Exp/Log maps for unit quaternions.
Opacity ( $\alpha$ ): Modeled on $(0, 1)$ using a logit transform to map to $\mathbb{R}$ .
Position ( $\mu$ ): Modeled on $\mathbb{R}^3$ .

The objective function minimizes the distance between the learned vector field $v_\theta$ and the conditional vector field $u_t$ on the manifold:
$L_{RFM} = \mathbb{E}_{t, G_z \sim p_0, G_1 \sim p_1} \left[ \|v_\theta(G_t, t, f_{img}, \gamma(\tau)) - u_t\|^2_{\mathcal{M}} \right]$
where $G_t$ represents the state at flow time $t$ , conditioned on image features $f_{img}$ and video time $\tau$ .

2.2 Architecture and Pipeline

Backbone: A frozen DINOv2-L backbone extracts semantic features from monocular video frames.
Perceiver Module: Maps the set of Gaussian encodings to fixed-size latent tokens.
Temporal Conditioning: The vector field is explicitly conditioned on the video time index $\tau$ via a learnable embedding $\gamma(\tau)$ , enabling the model to capture temporal evolution rather than a static average.
Self-Supervised Teacher: The teacher model is trained using self-supervised losses without human-annotated depth labels. The loss function combines:
- Rendering loss ( $L_{render}$ ) for photometric consistency.
- SE(3)-equivariance loss ( $L_{equiv}$ ) with rigidity scores.
- Optical flow consistency ( $L_{flow}$ ) using RAFT.
- Scale regularization ( $L_{scale}$ ) using median depth from DA2 (weak supervision).
- Regularization ( $L_{reg}$ ).

2.3 Inference and Sampling

The inference process employs a Coarse-to-Fine ODE Sampling strategy (Algorithm 1):

Coarse Stage: 10 steps to capture global scene layout.
Fine Stage: 20 steps to refine details.
Test-Time Optimization (TTO): Optional guidance is applied during the fine stage using a rendering loss gradient ( $\nabla G L_{render}$ ) to further refine the generated Gaussians.

3. Key Contributions and Controlled Experiments

The paper rigorously isolates the contribution of RFM from other factors (pre-training and TTO) through a series of controlled experiments using a deterministic MLP baseline. This baseline shares the exact same data, architecture (DINOv2-L + Perceiver), and TTO protocol but replaces the RFM objective with standard $L_2$ regression.

Isolated Gains (F-score on Dynamic-Sim):

TTO Contribution: Adding TTO to a standard 4D-GS baseline yields a +0.046 gain.
Pre-training Contribution: Large-scale pre-training on WebVid-10M and MegaDepth (without RFM) yields a +0.014 gain over 4D-GS+TTO.
RFM Contribution: The RFM formulation itself provides an isolated +0.044 gain over the deterministic MLP+TTO baseline (0.806 vs. 0.762).

This decomposition demonstrates that the improvement is not merely due to larger datasets or optimization techniques, but specifically stems from defining probability paths on the correct non-Euclidean manifolds.

4. Experimental Results

4.1 Dynamic Reconstruction

On Dynamic-Sim and KITTI Dynamic benchmarks, DAV4 (RFM + TTO) achieves an F-score of 0.806 and 0.652 respectively. This outperforms:

Fully optimized per-scene 4D-GS (5000 iterations): 0.781 (Dynamic-Sim).
Deterministic MLP + TTO: 0.762.
Euclidean Flow Matching + Post-hoc Projection: 0.684 (with 12.3% invalid Gaussians).

DAV4 reduces the percentage of invalid Gaussians (negative scales, non-unit quaternions) from 12.3% (Euclidean approach) to 0.01%, confirming the mathematical validity of the Riemannian approach.

4.2 Novel-View Synthesis (NVS)

Evaluated on ETH3D and ScanNet, DAV4 achieves 25.13 dB PSNR on ETH3D with only 20 TTO steps. This surpasses:

3D-GS with random initialization and 20 TTO steps (22.34 dB).
Fully optimized 3D-GS (5000 iterations, 24.12 dB).

The results indicate that the learned Riemannian prior provides a significantly better initialization than random initialization, allowing convergence to a better local optimum with fewer optimization steps.

4.3 Ablation Studies

Time Conditioning: Removing video time $\tau$ from the conditioning inputs reduces the F-score from 0.806 to 0.742 and temporal consistency (TC) from 0.92 to 0.78, proving the necessity of explicit temporal modeling.
Uncertainty Quantification: DAV4 provides well-calibrated uncertainty estimates via multi-hypothesis sampling. It achieves a Negative Gaussian Log-Likelihood (NGLL) of -1.58 and Expected Calibration Error (ECE) of 0.058, outperforming MC-Dropout and Ensemble methods.

4.4 Computational Cost

Pre-training: Requires 360 GPU-hours.
Inference: 420ms per scene (11x slower than DA3's 38ms).
Amortization: The paper notes that while DAV4 is not suitable for real-time applications, the pre-training cost amortizes effectively for large-scale deployment. For 10,000 scenes, the amortized cost drops to ~550ms per scene, becoming competitive with per-scene optimization methods while offering higher quality.

5. Significance and Claims

The paper claims that Depth Anything V4 represents a significant step toward probabilistic 4D scene understanding. Its primary significance lies in:

Geometric Validity: Demonstrating that defining flow matching directly on Riemannian manifolds (scale, rotation, opacity) is superior to Euclidean approximations with post-hoc projection, ensuring all intermediate states are valid and improving reconstruction quality.
Isolated Contribution: Providing empirical evidence that the Riemannian formulation itself contributes a substantial +0.044 F-score gain, independent of pre-training scale or test-time optimization.
Self-Supervised Paradigm: Achieving high-fidelity dynamic reconstruction without relying on human-annotated depth labels for training losses.
Uncertainty Awareness: Offering a framework that quantifies uncertainty via NGLL and ECE, which is essential for downstream decision-making in dynamic environments.

The authors acknowledge limitations, including the lack of real-time suitability (420ms inference), the high computational cost of pre-training, and an inherent initialization bias in NVS comparisons against random-initialized 3D-GS. However, they position DAV4 as a complementary advancement to the fast feed-forward depth estimation of V1–V3, specifically tailored for high-fidelity offline 3D archiving and content creation.