A Comparison of Multi-View Stereo Methods for Photogrammetric 3D Reconstruction: From Traditional to Learning-Based Approaches

1. Problem Statement

Photogrammetric 3D reconstruction is critical for applications like digital twins, scene understanding, and large-scale mapping. While traditional Multi-View Stereo (MVS) methods (e.g., COLMAP, Pix4D) offer high geometric accuracy, they suffer from:

• High computational costs and poor scalability for large datasets.
• Fragility in image registration (SfM), where failures in feature matching or camera pose estimation cascade through the pipeline, causing reconstruction failure.
• Sensitivity to textureless regions, repetitive patterns, and illumination changes.

Conversely, learning-based MVS methods have emerged as faster alternatives. However, there is a lack of systematic comparison between these new learning-based approaches (both geometry-guided and end-to-end) and traditional pipelines specifically within the context of aerial photogrammetry, which presents unique challenges compared to indoor or simulated datasets.

2. Methodology

2.1 Comparative Framework

The study evaluates three distinct categories of MVS methods:

1. Traditional: COLMAP (representing classical SfM + MVS pipelines).
2. Geometry-Guided Learning: Methods that integrate learned cost volumes with explicit geometric constraints and require pre-estimated camera poses (e.g., MVSNet, PatchmatchNet, MVSAnywhere, MVSFormer++).
3. End-to-End Learning: Methods that infer 3D structure, correspondence, and poses directly from images without explicit geometric modeling (e.g., Stereo4D, FoundationStereo, DUSt3R, MASt3R, Fast3R, VGGT).

2.2 Experimental Setup

• Datasets:
  • MARS-LVIG (Island Scene): UAV aerial imagery (5472×3648) with Ground Truth (GT) derived from high-precision LiDAR.
  • Pix4D (Urban Scene): Public urban aerial imagery (6000×4000) with GT generated by Pix4Dmapper.
• Input Handling:
  • Geometry-guided methods relied on COLMAP outputs for camera poses and sparse reconstruction.
  • End-to-end methods used undistorted images.
  • Images were rescaled to a maximum dimension of 512 pixels to ensure fair comparison across models with different input constraints.
• Progressive Subset Strategy: Experiments were conducted using varying numbers of input images (2, 10, 20, 50, and 100 frames) to test scalability.
• Evaluation Metrics:
  • Runtime: Total processing time in seconds.
  • Accuracy: Root Mean Square (RMS) and Mean Distance (MD) errors normalized by Ground Sampling Distance (GSD).
  • Coverage: Percentage of ground truth points successfully reconstructed within a 1-meter tolerance.
  • Tools: CloudCompare for quantitative analysis.

3. Key Contributions

• Systematic Benchmarking: Provides the first comprehensive comparison of state-of-the-art learning-based MVS (including recent end-to-end models like VGGT and DUSt3R) against traditional pipelines in aerial scenarios.
• Scalability Analysis: Investigates how performance degrades or improves as the number of input views increases, a critical factor for large-scale mapping.
• Identification of Failure Modes: Highlights specific artifacts (e.g., layering, discontinuities) in learning-based methods when applied to complex outdoor textures (vegetation, water) compared to controlled indoor datasets.
• Practical Guidelines: Offers actionable insights for practitioners on selecting methods based on trade-offs between speed, accuracy, and robustness.

4. Key Results

4.1 Processing Time (Efficiency)

• Traditional (COLMAP): Exhibits the highest computational cost, scaling poorly. For 100 images, runtime exceeded 1300 seconds. It frequently failed to register frames in difficult scenarios.
• Geometry-Guided: Dependent on COLMAP's success. If COLMAP fails to generate a sparse model, these methods fail entirely. Runtime is moderate but scales linearly with image count.
• End-to-End:
  • Fastest: Fast3R and VGGT demonstrated superior efficiency. Fast3R processed 100 images in ~16 seconds, and VGGT in ~60 seconds.
  • Slowest: MASt3R and DUSt3R showed poor scalability, with MASt3R taking over 3 hours for 100 images.
  • Stereo-only (FoundationStereo/Stereo4D): Fast for 2 views but unable to handle multi-view geometry effectively.

4.2 Accuracy (Geometric Fidelity)

• Sparse Views (2 images): DUSt3R achieved the highest accuracy (lowest RMS/MD) in both scenes, outperforming traditional methods.
• Dense Views (10-100 images):
  • DUSt3R accuracy degraded significantly as view count increased (RMS rose from 2.88 to 9.70 GSD in the Island scene), indicating issues with global consistency in overlapping regions.
  • VGGT maintained the most stable accuracy across increasing view counts.
  • COLMAP provided reliable geometric consistency but suffered from lower coverage due to registration failures.
  • Fast3R and MASt3R exhibited larger residuals and local misalignments.

4.3 Coverage

• DUSt3R: Achieved the highest coverage in sparse settings (84.89% on Island with 2 images) but suffered a "coverage collapse" when more images were added, likely due to conflicting depth predictions.
• VGGT: Demonstrated the most balanced and stable coverage, decreasing only moderately as input views increased.
• Traditional Methods: Often produced incomplete reconstructions due to registration failures, leading to lower overall coverage in challenging scenes.

4.4 Failure Modes

• Learning-Based Artifacts: End-to-end methods frequently produced layering artifacts and discontinuities, particularly in textureless areas (water, rocks) and repetitive patterns (vegetation, rooftops).
• Traditional Limitations: Prone to total pipeline failure if initial feature matching (SfM) is unsuccessful.

5. Significance and Conclusion

This study bridges the gap between traditional photogrammetry and modern deep learning approaches. The findings suggest:

1. End-to-end methods (specifically VGGT) offer the best compromise for large-scale aerial reconstruction, providing substantially faster processing and better robustness to registration failures than traditional methods, while maintaining competitive accuracy.
2. Geometry-guided methods remain heavily dependent on the quality of traditional SfM pipelines, limiting their standalone utility in complex aerial scenarios.
3. Current Limitations: While learning-based methods are faster, they still exhibit large residuals and structural inconsistencies (layering) that prevent their immediate adoption for high-precision metric mapping without further refinement.

Future Work: The authors propose integrating pose-aware priors (e.g., SLAM trajectories) to enhance large-scale accuracy and developing scalable inference strategies to handle massive image collections more efficiently.