Technical Summary: Lightweight Neural Framework for Robust 3D Volume and Surface Estimation from Multi-View Images

Problem Statement

Accurate estimation of object volume and surface area from visual data is critical for applications ranging from marine ecology (coral growth monitoring) and medical diagnostics (anthropometry, dietary planning) to industrial analysis. However, existing approaches face significant limitations:

• Computational Cost: Traditional pipelines relying on Structure-from-Motion (SfM), dense Multi-View Stereo (MVS), or explicit watertight mesh generation (e.g., COLMAP, PMVS) are computationally intensive and require extensive post-processing.
• Data Sensitivity: These methods often fail or degrade significantly when input data is sparse (few views), noisy, or lacks accurate calibration.
• Workflow Inefficiency: State-of-the-art deep learning systems frequently focus on intermediate tasks like geometry reconstruction or image synthesis, relegating metric estimation to a separate, often error-prone post-processing step.
• Scale Ambiguity: In unconstrained RGB multi-view capture, absolute scale is ambiguous, complicating direct regression of physical metrics.

Methodology

The authors propose a fully feed-forward, lightweight neural framework that directly regresses scale-normalized volume and surface area (along with their uncertainties) from multi-view RGB images, bypassing iterative optimization and explicit mesh generation.

Pipeline Architecture

The system operates in four primary stages:

1. Pre-processing & Segmentation: The object of interest is isolated from the background using advanced promptable segmentation tools (Grounded-SAM2, combining Grounding DINO and SAM 2) or synthetic image generation. This yields masked multi-view images.
2. 3D Branch (Geometric Backbone): A general-purpose multi-view reconstructor (specifically MapAnything for efficiency, or VGGT) processes the masked views to generate a fused, sparse point cloud. Each point is augmented with a confidence score. The point cloud is normalized to a unit-extent sphere to handle scale ambiguity.
3. 2D Branch (Visual Features): A frozen, state-of-the-art 2D encoder (DINOv3 ViT-S/16) extracts view-aligned features from the masked RGB images. These features capture texture and edge information complementary to the 3D geometry.
4. Fusion & Regression:
   • 3D Encoding: The point cloud is processed by a Graph Isomorphism Network (GIN) to produce a compact 3D latent vector ($z_{3D}$).
   • 2D Encoding: Multi-view 2D features are aggregated via shallow MLPs and mean/max pooling into a global descriptor ($d_{2D}$).
   • Fusion: The 3D and 2D descriptors are concatenated and fed into a Fully Connected (FC) regression head.
   • Output: Instead of a single point estimate, the network outputs four parameters ($\gamma, v, \alpha, \beta$) defining a Normal-Inverse-Gamma (NIG) prior. This allows the model to predict the metric ($\gamma$) and explicitly quantify aleatoric (data noise) and epistemic (model uncertainty) uncertainties.

Training Strategy

• Datasets: The model is pre-trained on the Objaverse dataset for generalization, then fine-tuned on domain-specific datasets: Synthetic Corals (generated via Infinigen), THuman2.1 (human scans), and MetaFood (real food images).
• Ground Truth: Metrics are derived from watertight meshes (processed via ManifoldPlus) normalized to a unit bounding sphere.
• Loss Function: A composite objective combines:
  • Evidential Negative Log-Likelihood (NLL): Minimizes the NLL of the Student-t distribution derived from the NIG prior.
  • Evidential Regularization: Penalizes high confidence in regions of large error.
  • Deterministic Penalties: Includes L1 (Mean Absolute Error) and MAPE (Mean Absolute Percentage Error) to ensure robust point estimation and stability.

Key Contributions

1. Streamlined End-to-End Regression: The framework eliminates the need for watertight mesh generation and iterative optimization, enabling single-pass, high-speed inference suitable for large-scale deployment.
2. Integrated 2D-3D Feature Fusion: By fusing view-aligned 2D features (DINOv3) with 3D point cloud representations (via GIN), the model achieves robustness against sparse views and noise, outperforming methods relying on a single modality.
3. Deep Evidential Uncertainty Estimation: The model provides calibrated confidence estimates for volume and surface measurements, distinguishing between aleatoric and epistemic uncertainties, which is crucial for scientific reliability.
4. Extensive Cross-Domain Evaluation: The approach is validated on diverse, real-world scenarios including coral monitoring, dietary analysis, and anthropometry, demonstrating superior generalization compared to existing baselines.

Experimental Results

The framework was evaluated against state-of-the-art methods (e.g., VolETA, ININ, Trellis, 2D Gaussian Splatting, Agisoft) across synthetic and real datasets.

• Accuracy: The model achieves low Mean Absolute Percentage Error (MAPE) and Median APE (MdAPE). For instance, on the THuman dataset, incorporating 2D features reduced volume MAPE from ~20% to ~8% compared to using only 3D features.
• Sparse View Performance: The model excels in low-data regimes. On the MetaFood dataset, it outperformed baselines even with a single input image, where other methods (like Trellis) failed significantly (MAPE > 200%).
• Real-World Generalization: When tested on real coral specimens and human subjects, the fine-tuned model consistently outperformed commercial software (Agisoft) and 3D reconstruction baselines (2DGS, Trellis) in both mean and median errors, even with limited input views (20–30).
• Uncertainty Calibration: Risk-Coverage analysis demonstrated that thresholding predictions based on estimated uncertainty significantly reduces error (dropping risk by >70% at 0.6 coverage), confirming the model's ability to flag unreliable predictions.
• Efficiency: The framework is highly efficient, performing forward passes in <0.1 seconds on an NVIDIA A100 GPU for 50k-point clouds, with a memory footprint of ~3GB, compared to minutes or hours required for traditional reconstruction pipelines.

Significance and Claims

The paper positions this framework as a robust, adaptable, and high-speed alternative for quantitative shape analysis. Its primary significance lies in:

• Scalability: It enables efficient processing of large-scale surveys (e.g., reef monitoring) where traditional methods are too slow or resource-intensive.
• Robustness: It maintains high performance in "imperfect" scenarios characterized by sparse views, noise, and occlusions, where classical MVS and mesh-based methods often fail.
• Practicality: By bypassing the need for watertight meshes and heavy post-processing, it offers a direct path from visual data to geometric metrics, facilitating real-world applications in resource-constrained environments.

The authors acknowledge limitations, specifically the reliance on scale-normalized predictions (requiring external cues for absolute measurement) and the current scarcity of large, annotated real-world datasets for specific domains, though they note the model's strong performance even with limited fine-tuning data (e.g., 360 food samples).