💻 computer science

M2Depth: Unifying Monocular Depth Foundation Priors with Multi-View Stereo

The paper proposes M2Depth, a novel framework that unifies monocular depth foundation priors with multi-view stereo through a bidirectional mutual refinement strategy and prior-guided cost volume optimization, achieving superior depth map completeness, generalization, and accuracy across both dense and sparse-view settings.

Original authors: Byeonggwon Lee, Sanggi Lee, Siwoo Lee, Khang Truong Giang, Soohwan Song

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Byeonggwon Lee, Sanggi Lee, Siwoo Lee, Khang Truong Giang, Soohwan Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to build a detailed 3D model of a room using only a single photograph. You can guess where the walls are and how far the furniture sits, but you cannot know the exact distance without a second perspective to compare against. This is the fundamental challenge of reconstructing the three-dimensional world from flat images. For decades, scientists have relied on a technique called multi-view stereo, which stitches together multiple photos taken from different angles to calculate depth. While powerful, this method often stumbles when the scene is complex, when parts of the object are hidden, or when there are very few photos to work with. In these difficult spots, the computer struggles to find matching points between images, leading to gaps or distorted shapes in the final model.

At the same time, a new generation of artificial intelligence has emerged that can look at a single image and guess the depth of every pixel. These systems, trained on vast libraries of images, are remarkably good at understanding the general shape of a scene, even in places they have never seen before. However, because they rely on just one picture, they often get the scale wrong; a car might look the right shape but be the size of a toy, or a building might appear correct but be miles away. For years, researchers have tried to combine these two approaches: the geometric precision of multi-view stereo and the intuitive shape recognition of single-image AI. Previous attempts simply took the AI's guess and pasted it into the multi-view system, or vice versa. This one-way street often failed, as errors from one side would simply drag the other side down, leaving the final model either incomplete or geometrically misaligned.

A team of researchers has now proposed a new way to bridge this gap, creating a system where the two methods constantly talk to each other to correct their own mistakes. Instead of treating the single-image AI as a static guide, their framework allows the multi-view system and the single-image AI to refine each other in a continuous loop. The multi-view system uses its knowledge of how the camera moved to fix the scale and alignment of the AI's guess. In return, the AI provides a strong sense of the overall structure, helping the multi-view system fill in the blanks where photos are missing or the view is blocked. This back-and-forth correction happens at every stage of the process, from the initial rough sketch to the final detailed model.

The researchers tested this approach on standard datasets containing complex scenes, including indoor rooms and outdoor landscapes. They found that their method produced significantly cleaner and more complete 3D reconstructions than previous state-of-the-art techniques. Where older methods left holes in occluded areas or produced warped surfaces, this new system maintained sharp boundaries and accurate geometry. The improvement was particularly noticeable in challenging conditions, such as when the system was given only three photos taken from unfavorable angles, a scenario where traditional methods usually fail. Even without being specifically trained for such difficult tasks, the system generalized well, managing to reconstruct scenes with a level of detail and accuracy that rivals methods designed exclusively for sparse data.

A key part of this success lies in how the system handles uncertainty. When the multi-view system cannot find a clear match between images, it does not just guess; it looks to the single-image AI for structural clues. Conversely, when the AI's depth estimate is ambiguous or lacks a specific scale, the multi-view system anchors it with precise geometric data. This mutual refinement ensures that the final output is not just a blend of two imperfect guesses, but a unified structure where the strengths of one compensate for the weaknesses of the other. The result is a robust tool that can generate high-quality 3D models even from limited or difficult visual data, offering a significant step forward in how machines perceive and reconstruct the physical world.

Technical Summary: M2Depth

Problem Statement

Deep learning-based Multi-View Stereo (MVS) has advanced significantly but struggles to generalize to unseen scenes, particularly in regions with occlusions, limited view overlap, or complex geometries. Conversely, Depth Foundation Models (DFMs), such as DepthAnythingV2, offer robust monocular depth predictions with strong generalization capabilities but suffer from inherent scale ambiguity and a lack of explicit multi-view geometric consistency.

Existing approaches attempting to integrate these two modalities typically rely on static, one-way fusion schemes:

MVS-centric methods (e.g., MonoMVSNet) inject monocular priors as static guidance but fail to correct the prior's scale errors or fully leverage multi-view geometry to resolve ambiguities in textureless regions.
DFM-centric methods (e.g., MVSAnywhere) use multi-view information to condition a monocular model but often result in misaligned geometries due to loose coupling and the propagation of uncorrected scale errors.

These unidirectional approaches prevent dynamic compensation between modalities, leading to either noisy reconstructions or geometrically inconsistent, misaligned 3D structures.

Methodology

The authors propose M2Depth, a novel framework that tightly couples a DFM with a cascade MVS pipeline through a bidirectional mutual refinement strategy. The system operates iteratively across a four-stage coarse-to-fine pipeline, ensuring that monocular and MVS depth representations progressively improve one another.

1. Prior-Guided Cost Volume Refinement (Cost-Crafter)

Applied at the initial cascade stage, this module fuses the multi-view cost volume ( $CV_{multi}$ ) with a monocular cost volume ( $CV_{mono}$ ) derived from the DFM.

Discretization: The continuous monocular depth is discretized into $M$ depth bins to create a structural bin mask ( $M_{bin}$ ).
Tokenization & Attention: The module employs a compact bin-based tokenization strategy. It groups pixels sharing similar prior depth levels into representative "bin tokens."
Ordinal Depth Encoding: To preserve geometric structure, scalar positional bias terms are introduced for both pixels and bins. This allows the attention mechanism to favor interactions between features at similar depth levels without polluting semantic features.
Bidirectional Interaction: The module performs two passes of cross-attention:
- Pixel-to-Bin: Pixels aggregate global context from structurally similar depth bins.
- Bin-to-Pixel: Refined global responses are redistributed back to ambiguous pixels (e.g., textureless or occluded regions).
  This process regularizes the initial cost volume, establishing a robust geometric foundation for subsequent stages.

2. Mutual Depth Refinement (MDR)

Operating across all cascade stages, this module iteratively updates both the MVS depth ( $D_{mvs}$ ) and the monocular depth ( $D_{mono}$ ) using two symmetric branches:

MVS-Guided Alignment (MVS-GA): Aligns the monocular depth to the MVS geometry. It uses a geometric feature volume (including curvature-based confidence cues) to iteratively predict pixel-wise residuals, correcting spatially varying scale ambiguities in the monocular prediction.
Mono-Guided Refinement (Mono-GR): Refines the MVS depth using the scale-aligned monocular structure. This branch is particularly effective in low-texture or occluded regions where photometric matching is unreliable. It uses a confidence-gated discrepancy encoder to allow monocular cues to dominate only where MVS matching is uncertain.
Adaptive Anchoring: A persistent confidence map (updated via Exponential Moving Average) serves as an adaptive anchor. High-confidence MVS pixels act as rigid anchors, while low-confidence regions allow the monocular prior to hallucinate missing structures.

3. Training Objective

The framework is trained using three complementary losses at every stage:

Cross-Entropy Loss ( $L_{CE}$ ): Standard depth distribution accuracy.
Mutual Refinement Loss ( $L_{mut}$ ): A decayed $\ell_1$ loss supervising intermediate refinement iterations.
Order-Preserving Loss ( $L_{ord}$ ): Penalizes violations in the relative depth ordering of pixel pairs to maintain local structure.

Key Contributions

Bidirectional Mutual Refinement: A novel strategy where MVS depth resolves monocular scale ambiguity, and monocular depth enhances MVS structural completeness and detail, moving beyond static fusion.
Prior-Guided Cost Volume Refinement: A "Cost-Crafter" module that integrates monocular cost volumes and structural bin masks via attention-based mechanisms to promote local geometric consistency.
Robust Sparse-View Generalization: The framework demonstrates remarkable generalization to sparse-view settings (e.g., 3 views) without specific training, outperforming dedicated sparse-view methods while maintaining an efficient accuracy-efficiency trade-off.
State-of-the-Art Performance: The method achieves superior results in both 3D reconstruction and depth estimation across standard benchmarks.

Experimental Results

The method was evaluated on the DTU, Tanks and Temples (TNT), and RobustMVD benchmarks.

DTU Benchmark: M2Depth achieved the lowest errors in Accuracy (0.301 mm), Completeness (0.241 mm), and Overall (0.271 mm), outperforming both conventional MVS methods (e.g., MVSFormer++) and recent DFM-based hybrids (e.g., MonoMVSNet, MVSAnywhere).
TNT Benchmark: The method achieved the highest mean F-scores on both intermediate and advanced test sets, demonstrating strong generalization to large-scale, unconstrained scenes.
RobustMVD (Depth Estimation): On cross-dataset depth estimation, M2Depth achieved the best results on DTU and TNT, and competitive results on KITTI, outperforming pure DFM approaches and MVS-centric hybrids.
Sparse-View Reconstruction: Evaluated under the SparseRecon protocol (3 unfavorable views), M2Depth achieved the lowest average Chamfer Distance (0.79), outperforming dedicated sparse-view methods like UFORecon and SparseRecon, despite not being explicitly designed for sparse-view optimization. It also maintained a significantly better runtime and memory profile compared to these specialized methods.

Significance and Claims

The paper claims that M2Depth successfully resolves the limitations of prior hybrid methods by replacing static, one-way fusion with a tightly coupled, bidirectional interaction. By continuously aligning monocular priors with multi-view geometry, the framework mitigates scale ambiguity while compensating for MVS incompleteness in challenging regions.

The authors emphasize that their approach offers a practical direction for generalized 3D reconstruction. It leverages the structural completeness of foundation models without sacrificing the metric reliability of MVS. Crucially, the method generalizes robustly to sparse-view settings without requiring specific training or expensive test-time optimization, providing a superior accuracy-efficiency trade-off compared to both standard MVS pipelines and specialized sparse-view reconstruction techniques.