💻 computer science

GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth

GemDepth is a novel video depth estimation framework that achieves state-of-the-art 3D consistency and spatial precision by integrating a Geometry-Embedding Module for explicit motion priors with an Alternating Spatio-Temporal Transformer to enforce rigorous temporal coherence.

Original authors: Yuecheng Liu, Junda Cheng, Longliang Liu, Wenjing Liao, Hanrui Cheng, Yuzhou Wang, Xin Yang

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuecheng Liu, Junda Cheng, Longliang Liu, Wenjing Liao, Hanrui Cheng, Yuzhou Wang, Xin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of the world using only a single video camera, like a GoPro strapped to a helmet. The goal is to figure out how far away everything is (depth) in every single frame of the video.

The problem with current "smart" cameras is that they often look at each video frame as if it were a standalone photograph. They are great at guessing depth for one picture, but when you string 30 pictures together to make a video, they get confused. The result? The 3D model looks like it's "flickering" or shivering, and fine details (like the edge of a leaf or a brick) turn into a blurry mess. It's like trying to draw a map while riding a rollercoaster without looking at the scenery, only guessing based on the last spot you saw.

The authors of this paper, GemDepth, say: "Stop guessing. Let's give the camera a sense of its own movement and the 3D shape of the world."

Here is how they fixed it, broken down into simple parts:

1. The "Motion Sense" (Geometry-Embedding Module)

Most video depth tools just try to smooth out the differences between frames, like blurring a shaky video to make it look stable. But that makes the image fuzzy.

GemDepth introduces a special module called GEM. Think of GEM as a GPS and Gyroscope for the camera.

Instead of just looking at pixels, GEM asks: "How did the camera move between this frame and the last one? Did it rotate? Did it zoom in?"
It calculates the camera's exact 6-degree-of-freedom pose (up/down, left/right, forward/back, and rotation).
It turns this movement data into a "geometric map" that gets injected into the AI's brain. This forces the AI to understand that if the camera turns left, the world should shift right in a specific, mathematically correct way. This stops the "shivering" because the AI now knows the rules of physics, not just the rules of smoothing.

2. The "Alternating Detective" (ASTT)

Once the AI knows how the camera moved, it needs to stitch the frames together perfectly. The authors built a new engine called ASTT (Alternating Spatio-Temporal Transformer).

Imagine a detective trying to solve a mystery by looking at a timeline of photos.

Step 1 (Temporal Alignment): The detective first looks across time. "If I see a red ball in frame 1, where is that exact same red ball in frame 2, considering the camera moved?" This ensures the object stays in the same 3D spot, even if the camera is spinning.
Step 2 (Spatial Refinement): Then, the detective looks within the frame. "Now that I know where the ball is, let me sharpen the edges of the ball so it doesn't look blurry."
The Magic: They do this back and forth (alternating). First, they align the movement, then they sharpen the details, then they align again. This ensures the video is both stable (no flickering) and sharp (no blur).

3. The "Two-Step Training" (Learning Strategy)

Training an AI to do this is hard because you need videos where you already know exactly how the camera moved (ground truth poses). But those videos are rare and expensive to make.

GemDepth uses a clever two-step training strategy:

Step 1 (The Gym): They train the AI on a huge pile of simulated videos (like video game footage) where the camera movement is known perfectly. Here, the AI learns the hard math of 3D geometry and how to track movement.
Step 2 (The Real World): Once the AI has learned the "rules of geometry" in Step 1, they freeze that part of its brain. Then, they teach the rest of the AI using real-world videos (like street scenes) where they don't know the exact camera movement. Because the AI already knows the geometry rules from Step 1, it can figure out the depth in these messy real videos very well, even without perfect data.

The Result

When they tested GemDepth, it was like comparing a shaky, blurry home video to a high-definition, stable 3D movie.

Sharper Details: It kept fine lines and textures crisp, whereas other methods blurred them out.
No Flickering: The 3D shapes stayed solid even when the camera spun or moved fast.
Efficiency: They achieved this "superpower" using less training data than other top methods, making it a very efficient solution.

In short, GemDepth stops the AI from just "guessing" depth frame-by-frame. Instead, it gives the AI a 3D compass and a step-by-step logic to build a video that is geometrically consistent, sharp, and stable.

Technical Summary: GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth

Problem Statement
Monocular depth estimation has advanced significantly through foundation models, yet these frameworks are primarily designed for single-frame inference. When applied to video, treating frames in isolation neglects crucial temporal dependencies, leading to severe inter-frame flickering and geometric inconsistencies. Existing video depth estimation methods generally fall into two categories, both with limitations:

Generative Models: While capable of preserving sharp details, they suffer from poor temporal coherence, high computational overhead, and prohibitive memory usage.
Discriminative Models: These extend single-frame estimators with temporal modules but rely on implicit 2D temporal smoothing. They lack explicit geometric awareness, struggling to maintain strict 3D consistency during complex camera rotations or drastic view changes. This often results in the suppression of high-frequency spatial cues, causing blurred boundaries and degraded structural fidelity.

The authors argue that explicit 3D geometry understanding—specifically the perception of camera motion and global 3D structure—is a prerequisite for genuine temporal consistency. Without explicit motion-aware priors, models fail to establish the latent point-level correspondences necessary to enforce consistency.

Methodology
GemDepth is a generalized framework designed to enhance video depth estimation by integrating explicit geometric constraints. It is instantiated on baselines such as DepthAnythingV2 and VideoDepthAnything. The framework consists of three core components:

Geometry-Embedding Module (GEM):
- Function: GEM explicitly models camera ego-motion to inject pose-aware geometric embeddings.
- Mechanism: It utilizes a lightweight EfficientPoseNet with a 4-layer alternating-attention transformer to process a learnable camera token. This estimates 6-DoF camera poses and a global scale factor.
- Normalization: To resolve monocular scale ambiguity, translations are normalized using a global scale factor, projecting all poses into a unified canonical frame.
- Output: The rotation, translation, and scale are encoded via geometric MLPs to create metric-aware camera feature representations ( $F_{cam}$ ). These are fused with visual features, providing the network with a coordinate system that enforces consistency through physical constraints rather than blind smoothing.
Alternating Spatio-Temporal Transformer (ASTT):
- Function: ASTT decouples dependency modeling into two specialized phases to enforce geometric consistency while refining details.
- Temporal Attention (Geometric Alignment): Leveraging GEM's motion priors, this phase establishes explicit point-level correspondences across the temporal axis. It performs trajectory-based feature aggregation to capture pure motion cues, ensuring robust spatial structural consistency and flicker-free sequences.
- Spatial Attention (Structural Refinement): Conditioned on aligned temporal features, this phase aggregates relevant 3D spatial features. It includes intra-frame spatial attention for local layout and inter-frame spatial attention for long-range dependencies, enhancing high-frequency representations and sharpening structural details.
- Execution: The module alternates between these mechanisms, ensuring geometric alignment precedes detail sharpening.
Two-Stage Training Strategy:
- Stage 1 (Geometric Optimization): The entire framework is jointly optimized using pose and depth supervision on a composite dataset of ~690K frames (including Virtual KITTI 2, TartanAir, etc.). This establishes robust geometric reasoning capabilities.
- Stage 2 (Depth Refinement): The GEM module is frozen to retain the geometric foundation. The remaining network is fine-tuned on ~250K frames from datasets without camera extrinsics ground truth (e.g., IRS, in-the-wild sequences). This compels the ASTT to adapt to pose noise, ensuring high-quality predictions even with imperfect geometric cues.

Key Contributions

GemDepth Framework: A generalized solution that incorporates a Geometry-Embedding Module (GEM) to inject pose-aware embeddings, enforcing rigorous 3D structural constraints.
ASTT Module: A novel transformer architecture that leverages geometric priors to decouple dependency modeling, resolving the conflict between detail preservation and temporal stability by alternating between geometric alignment and spatial refinement.
Data Efficiency: The framework achieves state-of-the-art (SOTA) performance with highly efficient data utilization, requiring significantly fewer training samples (fewer than 1M frames) compared to existing approaches that often require millions.

Experimental Results
Extensive evaluations across four benchmarks (Sintel, KITTI, Scannet, Bonn) demonstrate GemDepth's superiority:

Spatial Precision: GemDepth-VDA achieves an AbsRel error reduction of over 25% on Bonn and Scannet compared to existing SOTA methods. On Sintel, it surpasses VideoDepthAnything by 46.8%.
Temporal Consistency: GemDepth establishes new SOTA standards for temporal stability. On Scannet, it reduces the Temporal Alignment Error (TAE) by 56.14% (GemDepth-DAv2) and 17.54% (GemDepth-VDA) compared to their respective baselines.
3D Geometric Accuracy: In point cloud reconstruction, GemDepth-VDA achieves an F1 score of 90.43 on Bonn, significantly outperforming heavy 3D foundation models like DA3 (78.44) and VGGT, despite having only 0.58B parameters (roughly half the size of competitors).
Robustness: Ablation studies show the model maintains stable performance under low-to-moderate injected pose noise (up to 20%), demonstrating resilience to upstream geometric inaccuracies.
Efficiency: GemDepth achieves these results with a marginal computational overhead (<20%) compared to discriminative baselines and significantly lower latency than generative frameworks.

Significance and Claims
The paper claims that GemDepth sets a new benchmark for both spatial accuracy and temporal consistency in video depth estimation. By explicitly modeling camera motion and global 3D structure, the framework addresses the fundamental limitation of current methods that rely on implicit 2D smoothing. The authors assert that GemDepth is a versatile, generalized framework that can be seamlessly integrated into various monocular or video depth baselines. Its ability to achieve SOTA zero-shot performance with limited pose-labeled data highlights a shift toward more data-efficient and geometrically grounded approaches for 3D perception in dynamic environments. The work is positioned as a step forward in making depth estimation backbones more practical for real-world applications such as autonomous driving and augmented reality, where geometric consistency is critical.

1. The "Motion Sense" (Geometry-Embedding Module)

2. The "Alternating Detective" (ASTT)

3. The "Two-Step Training" (Learning Strategy)

The Result

More like this