💻 computer science

3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation

This paper introduces a novel self-supervised framework for monocular video depth estimation in endoscopic navigation that recasts the task as an unconstrained multi-view 3D reconstruction problem, leveraging 3D foundation models and a three-constraint optimization strategy to achieve state-of-the-art spatial accuracy and global geometric consistency.

Original authors: Yuanye Liu, Ke Zhang, Junzhe Jiang, Li Zhang, Vishal Patel, Xiahai Zhuang

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuanye Liu, Ke Zhang, Junzhe Jiang, Li Zhang, Vishal Patel, Xiahai Zhuang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a room using only a series of 2D photos taken from a moving camera. If you look at each photo in isolation, you might guess the depth, but your guesses will likely be wobbly. One photo might say a chair is close, while the next says it's far away, causing the 3D model to "drift" or warp as you move through the room. This is exactly the problem doctors face when trying to create 3D maps of the inside of a patient's body using standard endoscopic cameras, which only have one lens and no built-in GPS to track their position.

This paper introduces a new way to solve this "wobbly map" problem. Here is how it works, broken down into simple concepts:

The Old Way: The "Stuttering" Approach

Previous methods treated a video like a stack of independent snapshots. They tried to guess the depth of each frame one by one.

The Analogy: Imagine trying to draw a continuous line on a piece of paper while your hand is shaking. You draw a dot, then another dot, then another. Because you aren't looking at the whole picture, your dots drift apart, and the line looks jagged and broken. In medical videos, this causes "temporal flickering" (the image jitters) and "geometric drift" (the 3D shape warps over time), making it unreliable for surgery.

The New Way: The "3D Jigsaw Puzzle"

The authors propose a paradigm shift. Instead of looking at the video as a sequence of time-bound frames, they treat the whole video as a single, giant 3D jigsaw puzzle.

The Analogy: Imagine you have a box of puzzle pieces (the video frames). Instead of trying to solve them one by one as they come out of the box, you dump them all out on the table at once. You realize that even though the photos were taken at different times, they all belong to the same 3D room. By looking at them all together, you can figure out exactly where every piece fits in 3D space.

The Secret Sauce: Three "Glues"

To make this puzzle work without a human telling them where the pieces go (since there is no "ground truth" or perfect map to compare against), the researchers use a system called 3D Consistency Optimization. They use three specific "glues" to hold the 3D model together:

The "Look-Alike" Glue (Image Consistency):
- How it works: The computer renders a fake image from its 3D model and compares it to the real photo.
- The Metaphor: It's like a sculptor constantly checking their clay statue against a reference photo. If the shadow on the statue doesn't match the photo, the sculptor knows they need to move the clay. This ensures the 3D model looks exactly like the real world.
The "Anchor" Glue (World-Coordinate Alignment):
- How it works: This is the most important part. It forces points from different photos to land on the exact same spot in a shared 3D space.
- The Metaphor: Imagine you are walking through a forest and taking photos of a specific tree. In Photo A, the tree is on the left. In Photo B, it's on the right. This "glue" acts like a magnetic anchor, saying, "No matter which photo you are in, that tree must be at this exact coordinate in the universe." This stops the map from drifting or warping as the camera moves.
The "Smoothness" Glue (Temporal Consistency):
- How it works: It checks that the edges and shapes in the video don't jump around wildly from one frame to the next.
- The Metaphor: Think of a movie with bad special effects where objects jitter or shake. This glue acts like a "shock absorber" for the video. It ensures that if a surface is smooth in one frame, it stays smooth in the next, preventing the annoying flickering that ruins the experience.

The Result: A Stable, High-Definition Map

By combining these three glues, the system creates a unified 3D representation of the surgical scene.

The Outcome: The paper claims this method is significantly better than previous techniques. In tests on real surgical videos, it produced maps that were much more accurate (less error) and much more stable (less jitter).
The "Zero-Shot" Superpower: The system is so good at understanding 3D geometry that it works well on new, unseen surgical videos without needing to be retrained. It's like a master carpenter who can build a perfect table using a new type of wood they've never seen before, simply because they understand the fundamental rules of wood and joints.

Summary

In short, this paper stops treating video depth estimation as a series of isolated guesses. Instead, it treats the video as a single, cohesive 3D reconstruction problem. By using a "3D Jigsaw" approach anchored by three consistency rules, it creates a stable, drift-free 3D map of the inside of the human body, which is crucial for helping surgeons navigate and understand the surgical scene.

Technical Summary: 3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation

Problem Statement
Reliable monocular video depth estimation is essential for downstream 3D reasoning and embodied AI in endoscopic navigation, including dense reconstruction, robot-assisted guidance, and surgical scene understanding. However, clinical deployment is hindered by hardware constraints; most standard endoscopes are monocular and lack reliable camera pose tracking. Existing self-supervised approaches typically treat video frames as independent entities or rely on weak temporal regularization. These methods fail to capture the holistic 3D geometry of the surgical scene, leading to geometrically inconsistent predictions, severe cross-frame drift, and temporal flickering. These errors accumulate rapidly in challenging endoscopic conditions, undermining the reliability of any downstream 3D reconstruction.

Methodology
The authors propose a paradigm shift that recasts sequential video depth estimation as an unconstrained multi-view 3D reconstruction problem. Instead of processing frames in isolation, the method treats an endoscopic video as a collection of multi-view observations anchored to a shared 3D scene. This approach leverages the geometric priors of recent 3D foundation models without requiring ground-truth camera poses.

The framework consists of two main stages:

3D Geometry Understanding Network:
- Architecture: The network utilizes an Alternating-Attention Transformer that interleaves frame-level intra-view attention with global inter-view attention to enforce spatial-temporal coherence. Input images are tokenized using DINOv2 features.
- Scene Representation: The underlying scene is represented by a set of 3D Gaussian primitives ( $P$ ), parameterized by 3D centers, opacity, anisotropic covariance, and spherical harmonic coefficients.
- Heads: The network employs three specialized heads:
  - Camera Head: Predicts per-frame extrinsic poses ( $T_t$ ) and intrinsics ( $K$ ).
  - Depth Head: Predicts an initial per-pixel depth map ( $D^{init}_t$ ) used solely to back-project image pixels into 3D space, providing initialization for Gaussian centers.
  - Gaussian Head: Fuses transformer features with shallow CNN appearance features to predict remaining Gaussian attributes.
- Rendering: A differentiable Gaussian splatting mechanism renders expected RGB images ( $\hat{I}_t$ ) and depth maps ( $\hat{D}_t$ ) from the 3D Gaussians and predicted camera parameters.
3D Consistency Optimization Framework:
The network is trained via a self-supervised objective that jointly optimizes depth maps and camera poses by enforcing three synergistic constraints on the rendered outputs:
- Image-level Photometric Consistency ( $L_{rgb}$ ): Minimizes the $L_1$ difference between the input frame $I_t$ and the rendered image $\hat{I}_t$ , ensuring appearance and structural consistency.
- World-Coordinate Geometric Alignment ( $L_{3D}$ ): This is the core mechanism for eliminating spatial drift. It unprojects rendered depth points from frame $t$ into a unified world coordinate system and reprojects them onto neighboring frames ( $t'$ ). A consistency loss is computed between the reprojected depth and the rendered depth of the target frame, strictly over valid pixel sets (handling occlusions and out-of-bounds projections). A robust Huber penalty is applied to mitigate dynamic tissue motion and outliers.
- Multi-Scale Temporal Gradient Consistency ( $L_{temp}$ ): To suppress temporal flickering while preserving geometry-aware discontinuities, this loss penalizes the absolute mismatch of spatial gradients between adjacent frames across multiple scales.

The final training objective is a weighted sum of these three losses. Crucially, this optimization is strictly a training-time objective; during inference, the model predicts depth in a single feed-forward pass without test-time optimization.

Key Contributions

Novel Paradigm: The authors introduce a self-supervised framework that reformulates monocular video depth estimation as a pose-free, multi-view 3D consistency optimization problem, unlocking the potential of 3D foundation models for clinical endoscopy.
Unified Optimization Strategy: They propose a strategy featuring explicit world-coordinate alignment and multi-scale temporal gradient constraints. This effectively anchors sequential frames to a globally coherent geometry, eliminating cross-frame 3D drift.
State-of-the-Art Performance: The method achieves superior accuracy on public benchmarks and demonstrates robust zero-shot generalization to unconstrained clinical environments.

Results
The framework was evaluated on two public surgical datasets: SCARED (with ground-truth depth and poses) and Hamlyn (complex in vivo procedures, used for zero-shot evaluation).

Quantitative Performance: On the SCARED dataset, the proposed method outperformed recent baselines (EndoDAC, VDA, EndoDAV, and AnySplat) across all metrics. Specifically, it reduced the Absolute Relative Error (Abs Rel) from 0.100 (AnySplat) to 0.060 and the RMSE from 10.400 to 5.503, while achieving a $\delta_1$ accuracy of 0.978.
Zero-Shot Generalization: On the Hamlyn dataset, the method also achieved the best results across all metrics, reducing Abs Rel from 0.299 to 0.108 and RMSE from 29.938 to 10.535.
Qualitative Analysis: Visual comparisons showed that the method better preserves scene structure and avoids the abnormal errors and temporal inconsistencies (flickering) observed in existing methods, particularly in scenarios with large motion.
Ablation Studies: Experiments confirmed that all three loss components ( $L_{rgb}$ , $L_{3D}$ , $L_{temp}$ ) provide complementary benefits. The inclusion of the geometric ( $L_{3D}$ ) and temporal ( $L_{temp}$ ) constraints significantly enhanced spatial accuracy and stability compared to baselines. Furthermore, applying the consistency optimization module to different backbones (VDA and AnySplat) consistently improved their performance.

Significance
The paper claims that by transforming ill-posed monocular video depth estimation into a robust multi-view 3D reconstruction problem, the proposed framework effectively bridges the gap between powerful 3D foundation models and pose-free clinical endoscopy. The results demonstrate that explicitly enforcing global 3D consistency and temporal coherence allows for reliable depth estimation in real-world operating rooms, significantly reducing geometric drift and temporal instability compared to state-of-the-art methods. This advancement paves the way for more reliable downstream applications in minimally invasive surgery, such as dense reconstruction and robot-assisted guidance.