Technical Summary: ActMVS – Active Scene Reconstruction with Monocular Multi-View Stereo

1. Problem Statement

Active scene reconstruction is essential for robots and Unmanned Aerial Vehicles (UAVs) to autonomously navigate and map unknown environments, particularly in scenarios like industrial inspection and disaster response where manual data acquisition is costly. While passive methods (e.g., SfM, SLAM) process pre-collected data, active reconstruction requires real-time planning of sensor trajectories to maximize coverage while ensuring collision-free navigation.

Current active reconstruction approaches predominantly rely on depth sensors (e.g., LiDAR, structured light) to generate high-confidence occupancy maps for safe spatial reasoning. However, these sensors introduce prohibitive costs, weight, and power constraints for resource-limited platforms. The authors aim to develop a vision-only monocular solution.

The core challenge lies in acquiring dense depth predictions at frame rates with metric-scale accuracy using only a single camera. Existing monocular depth estimation methods lack metric accuracy, while standard Multi-View Stereo (MVS) techniques face two critical limitations in active settings:

1. Insufficient Baselines: Using adjacent frames often results in sub-optimal depth due to insufficient parallax or covisibility.
2. Lack of Global Context: Recent learning-based MVS methods typically process only a limited number of reference frames (e.g., 8), leading to error accumulation and a lack of global consistency, which compromises planning safety and reconstruction quality.

2. Methodology: ActMVS Framework

ActMVS is introduced as the first framework for monocular active scene reconstruction. It integrates a hybrid scene representation with a novel view factor graph and global depth optimization to enable online, high-quality, globally consistent dense depth map generation.

A. Hybrid Scene Representation

The system maintains three concurrent representations:

• View Factor Graph ($G_f$): Models inter-frame geometric relationships. Nodes store view images, optimized depth maps, poses, and intrinsics. Edges encode voxel-frame visibility weights to indicate geometric overlap.
• Voxel Map ($M_v$): Built upon OctoMap, storing occupancy probabilities. It serves as both an occupancy map for collision-free planning and a visibility model for co-visible surfaces.
• Gaussian Splatting Map ($M_g$): Uses Gaussian surfels for high-fidelity surface reconstruction, generating color, depth, normal, and opacity maps via differentiable rendering.

B. Online Active Reconstruction Process

The framework operates iteratively in two phases:

1. Planning Phase (Next-Best-View):
   • Candidate viewpoints are sampled within safety and exploration boundaries.
   • A weighted score combining view completeness (visible unexplored voxels) and travel distance (computed via A* path search) determines the Next-Best-View (NBV).
   • Trajectories are interpolated along the A* path to ensure sufficient covisibility for subsequent MVS depth prediction.
2. Mapping Phase:
   • Visibility Modeling: Computes pairwise overlaps ($\Omega_{tj}$) via voxel covisibility, updating the View Factor Graph by connecting the current view to top-$k$ neighbors.
   • Depth Prediction: Adapts the feedforward MVS method MVSA to estimate dense depth using selected neighbors as reference views.
   • Global Depth Optimization: Performs $N_{iter}$ steps of global optimization over the depth map through pairwise depth warping within the graph.
   • Update: Updates the Voxel Map via probabilistic occupancy integration and refines the Gaussian Map by initializing primitives in low-opacity areas and minimizing photometric/depth losses.

C. Key Technical Innovations

1. View Factor Graph with Voxel-Frame Visibility Modeling:

   • To address occlusion issues in reference frame selection, the method computes per-frame frustum masks ($M_t$) by projecting voxel centers onto the camera plane and checking against rendered Gaussian depth.
   • This allows for efficient overlap calculation via a logical AND operation, filtering out obstructed voxels.
   • A correlation score balances covisibility ($\Omega_{ij}$) and camera translation to select reference frames that provide meaningful baselines (avoiding poses that are too close or too distant).

2. Global Depth Optimization:

   • To overcome the limited context of standard MVS (which processes ~8 frames), ActMVS enforces 3D consistency across the entire View Factor Graph.
   • Depth Consistency Factor: Aligns co-visible 3D structures between frame pairs by backprojecting depth, warping points, and applying a log-space Huber loss to valid correspondences.
   • Regularization: Incorporates Total Variation (TV) regularization to ensure smoothness and geometric plausibility.
   • The system solves this non-linear optimization via iterative backpropagation (Adam optimizer), propagating geometric constraints incrementally.

3. Experimental Results

Experiments were conducted on the Replica dataset (9 scenes) using the Habitat simulator, comparing ActMVS against state-of-the-art RGB-D baselines: ActiveGS and NARUTO.

• Quantitative Performance:

  • ActMVS achieves the best performance among all RGB-only methods across rendering metrics (PSNR, SSIM, LPIPS) and mesh quality metrics (Accuracy, Completion, Chamfer distance).
  • While RGB-D methods (ActiveGS) generally outperform, ActMVS closely approaches their performance, validating that multi-view constrained depth optimization compensates for the lack of depth sensors.
  • Ablation Studies:
    • w/o OPT: Removing global optimization causes significant surface degradation and depth instability.
    • w/o REF: Replacing voxel-based reference selection with simple adjacent-frame selection leads to MVS failures due to insufficient parallax.
    • w/o MVS: Replacing MVS with a monocular metric estimator (DepthAnythingV2) results in catastrophic reconstruction failures due to error accumulation.

• Qualitative Results:

  • Visual comparisons show ActMVS produces high-quality meshes and novel view renderings, outperforming NARUTO (which suffers from sparse viewpoint degradation) and approaching the visual fidelity of ActiveGS.

• Extended Simulations:

  • The framework was tested in AirSim with a quadrotor UAV using the Ego-planner for trajectory execution.
  • The system successfully navigated complex environments without collisions, demonstrating the feasibility of the path planning and reconstruction loop.
  • Note: The authors acknowledge that current 3DGS-based reconstruction requires significant video memory, exceeding the capacity of typical onboard devices, necessitating server-side processing for the current implementation.

4. Significance and Claims

The paper claims ActMVS as the first framework for monocular active scene reconstruction. Its significance lies in:

• Advancing Spatial Intelligence: Enabling robots/UAVs to perform safe, autonomous exploration and reconstruction using only a monocular camera, eliminating the need for expensive and heavy depth sensors.
• Bridging the Gap: Successfully addressing the trade-off between the metric accuracy of MVS and the global consistency required for online mapping.
• Competitive Performance: Demonstrating that a vision-only approach can achieve performance competitive with RGB-D methods, making high-fidelity active reconstruction accessible to resource-constrained platforms.

The authors conclude that while the current system requires significant computational resources, future work will focus on optimizing for real-time deployment on fully autonomous drones under physical constraints.