💻 computer science

Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth

This paper introduces Depth2Pose, a novel benchmark and dataset that evaluates monocular depth estimation models based on their ability to support downstream relative camera pose estimation, offering a task-driven alternative to traditional ground-truth depth metrics that is particularly effective for challenging scenes where dense depth annotations are unavailable.

Original authors: Viktor Kocur, Sithu Aung, Gabrielle Flood, Yaqing Ding, Lukas Bujnak, Torsten Sattler, Zuzana Kukelova

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Viktor Kocur, Sithu Aung, Gabrielle Flood, Yaqing Ding, Lukas Bujnak, Torsten Sattler, Zuzana Kukelova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Perfect Map" Trap

Imagine you are trying to build a robot that can walk through a forest. To do this, the robot needs a "depth map"—a way to understand how far away trees, rocks, and bushes are.

For years, scientists have been training robots to guess these distances just by looking at a single photo. To see if they are doing a good job, they used a strict test: Compare the robot's guess to a "perfect map" (Ground Truth).

But here is the catch: Getting a "perfect map" is incredibly hard and expensive. You usually need special laser cameras (LiDAR) or very controlled environments. Because of this, the tests were only done on easy scenes (like tidy indoor rooms or specific driving streets) where these perfect maps existed.

The Flaw: The paper argues that judging a robot just by how close its guess is to a "perfect map" is like judging a navigator only by how well they memorized a map, rather than how well they can actually drive the car. A robot might make small errors in boring parts of the picture (like a blank wall) but still drive perfectly. Conversely, it might be perfect on a blank wall but fail to see a rock in the middle of the road, causing a crash.

The Solution: The "Driver's License" Test

The authors, Viktor Kocur and his team, propose a new way to test these depth estimators. Instead of asking, "How close is your guess to the perfect map?" they ask: "Can you help me figure out where the camera is moving?"

They call this new framework Depth2Pose.

The Analogy:
Imagine you are blindfolded and someone is moving you through a room. You are holding a piece of paper with a drawing of the room on it.

The Old Way: You compare your drawing to a photo of the room to see if the lines match perfectly.
The New Way (Depth2Pose): You use your drawing to guess which way the person is walking. If your drawing helps you guess the walking direction correctly, your drawing is "good enough," even if the lines aren't perfectly drawn.

In technical terms, they take two photos, guess the depth, and use that guess to calculate the camera's movement (pose). If the calculated movement matches the actual movement (which is easier to find than a perfect depth map), then the depth guess was useful.

The New Playground: The "D2P" Dataset

To prove their idea works, they built a new playground called the D2P Dataset.

Most existing tests are like a driving test on a smooth, empty highway. The D2P dataset is like a driving test in a jungle and a sculpture garden.

The Jungle (Vegetation): Trees, leaves, and branches are tricky. They are see-through, move in the wind, and have no clear edges. Old depth cameras struggle here.
The Sculpture Garden (Statues): These are weird shapes, often hanging in the air or in water. They don't look like the "standard" objects the robots were trained on.

The authors say that many robots that get "A+" grades on the smooth highway (standard benchmarks) crash immediately in the jungle or sculpture garden.

What They Found

The Correlation: On the easy, standard tests, their new "Driver's License" test (Pose) agreed with the old "Perfect Map" test. If a robot was good at guessing distances, it was also good at guessing movement.
The Surprise: When they moved to the D2P Dataset (the jungle and statues), the rankings changed completely.
- Some robots that were top-tier on standard tests failed miserably in the new scenes.
- Some robots that looked "okay" on standard tests actually handled the difficult scenes surprisingly well.
The Conclusion: A robot can be "globally accurate" (close to the perfect map on average) but still useless for the actual job (moving through a scene) if it gets the important parts wrong. The new test highlights these hidden weaknesses.

Why This Matters

The authors aren't saying the old tests are useless. They are saying we need a second opinion.

By using Depth2Pose, we can test robots in real-world, messy environments (like forests or crowded cities) without needing expensive laser scanners to create a "perfect map" first. We just need to know where the camera moved, which is much easier to figure out.

In short: They built a new test that stops robots from just memorizing maps and starts testing if they can actually navigate the real world.

Technical Summary: Depth2Pose

Problem Statement

Monocular depth estimation (MDE) has advanced significantly due to deep learning and large-scale training data, with predicted depth increasingly serving as a critical input for downstream geometric tasks such as Structure-from-Motion (SfM), visual localization, Simultaneous Localization and Mapping (SLAM), and dense 3D reconstruction. However, the evaluation of MDEs remains constrained by traditional protocols that rely on pixel-wise depth accuracy metrics (e.g., AbsRel, RMSE, $\delta_1$ ). These metrics aggregate errors globally, treating all image regions equally, which fails to reflect the varying geometric importance of different regions for downstream tasks. Furthermore, standard benchmarks depend on dense ground-truth depth, typically acquired via LiDAR or RGB-D sensors. This reliance restricts evaluation to specific environments, excludes challenging scenarios (e.g., large-scale outdoor scenes, vegetation, reflective surfaces), and creates a distributional overlap between training and testing data that obscures generalization capabilities.

Methodology

The authors propose Depth2Pose, a task-driven evaluation framework that assesses MDE quality based on its utility for relative camera pose estimation rather than direct depth accuracy. The core premise is that accurate depth predictions in geometrically informative regions should lead to more accurate relative pose estimates when used within depth-aware geometric solvers.

Evaluation Pipeline

Data Preparation: For a given scene, camera poses are reconstructed using a mature SfM pipeline (COLMAP) to serve as pseudo-ground truth (pGT). This avoids the need for per-pixel depth ground truth.
Image Pair Selection: Image pairs with at least 10% visual overlap are sampled based on the SfM model.
Feature Matching: Keypoints are detected and matched between image pairs using learned local features (specifically LoMa or SuperPoint+LightGlue).
Depth-Aware Pose Estimation: The matched 2D correspondences are augmented with depth values predicted by the MDE under evaluation. These are fed into depth-aware relative pose solvers (specifically variants of the RePoseD framework). The solvers jointly leverage point correspondences and depth to estimate relative camera motion ( $R, t$ ).
Metric Calculation: The estimated relative pose is compared against the reference pose from the SfM reconstruction. The evaluation metric is the mean Average Accuracy (mAA) of the pose error (rotation and translation) up to a threshold of $10^\circ$ .

Solver Variants

The framework evaluates multiple solver configurations to isolate the impact of depth:

Baseline (B): Classical 5-point solver using only 2D-2D correspondences (no depth).
Hybrid (H): Scale-invariant solver using a hybrid error (Sampson error for scoring, combined Sampson and symmetric reprojection error for optimization).
Reprojection (R): Solver using only symmetric reprojection error for both scoring and optimization, providing a more direct assessment of depth impact by not allowing the Sampson error to compensate for depth inaccuracies.
Affine-Invariant: Variants assuming affine-invariant depth for scenarios where metric scale is not preserved.

Key Contributions

Depth2Pose Framework: A novel evaluation paradigm that replaces ground-truth depth with ground-truth poses. This enables scalable evaluation in diverse real-world environments where dense depth sensing is impractical or impossible.
Task-Relevant Evaluation: The framework captures spatially non-uniform depth quality, emphasizing regions that are geometrically informative for downstream tasks (e.g., areas with stable feature correspondences) rather than penalizing errors in textureless or less critical regions.
D2P Dataset: A new dataset comprising 24 challenging real-world scenes (12 "Statues" and 12 "Vegetation") that lie outside the distribution of standard training data. These scenes feature complex geometry, heavy occlusions, and large depth variations, specifically designed to test generalization.
Extensible Protocol: A simple, reproducible evaluation pipeline that facilitates benchmarking in previously inaccessible scenarios without requiring per-pixel depth annotations.

Results

Correlation with Standard Metrics: On established benchmarks (ETH3D, ScanNet++, Sintel, LaMAR), the pose-based metric (mAA) correlates strongly with standard depth metrics ( $\delta_1$ ). However, the authors note that high $\delta_1$ scores do not strictly guarantee high pose accuracy, as pose estimation relies on a subset of stable correspondences rather than uniform pixel accuracy.
Generalization Gap: On the D2P dataset, the ranking of MDEs changes significantly compared to standard benchmarks. Methods performing best on standard datasets (e.g., MoGeV2) do not necessarily maintain top performance on D2P, while some methods with lower standard benchmark scores perform competitively. This highlights a generalization gap not captured by traditional evaluation.
Depth Utility: Current MDEs do not yet consistently outperform the "No Depth" baseline (using only 2D-2D correspondences) on the D2P dataset, suggesting that while depth-aware solvers offer a path for improvement, current depth predictions often contain errors that do not yet translate to robust pose estimation in challenging, out-of-distribution scenes.
Solver Sensitivity: The choice of solver (H vs. R) affects the correlation between depth error and pose accuracy. The R variant, which relies heavily on reprojection error, provides a stricter test of depth quality, whereas the H variant offers practical insights into system performance.

Significance and Claims

The paper claims that Depth2Pose offers a necessary shift from evaluating depth as an isolated reconstruction task to evaluating it as a component of geometric reasoning systems. By demonstrating that standard benchmarks fail to predict performance on challenging, real-world scenes (D2P), the authors argue that the community needs task-driven evaluation protocols.

The authors modestly state that their framework does not replace standard depth metrics but complements them. They acknowledge limitations, noting that the framework is applicable only where accurate camera poses can be recovered (e.g., via SfM) and that it cannot distinguish between accurate depth in matched regions versus unmatched regions (e.g., mirrors). Furthermore, they note that relative pose estimation is more sensitive to depth errors than complex pipelines like SLAM, which may tolerate higher errors through redundancy. Ultimately, the work aims to encourage the development of MDEs that are robust and useful for downstream geometric tasks in diverse, real-world environments.