EvalMVX: A Unified Benchmarking for Neural 3D Reconstruction under Diverse Multiview Setups

This paper introduces EvalMVX, a comprehensive real-world dataset featuring 25 objects with aligned ground-truth meshes captured under diverse lighting and view conditions, to establish a unified benchmark for quantitatively evaluating and comparing neural multiview stereo, photometric stereo, and shape-from-polarization reconstruction methods.

The Gist

Imagine you want to build a perfect, 3D digital twin of a real-world object—say, a shiny metal bell or a soft, fuzzy teddy bear. You have a camera, but how do you turn flat 2D photos into a detailed 3D model?

This paper introduces EvalMVX, which acts like a giant, fair taste-test competition for different 3D reconstruction recipes.

Here is the breakdown in simple terms:

1. The Problem: The "One-Size-Fits-All" Trap

Until now, scientists had different "recipes" (algorithms) for making 3D models, but they were tested in different kitchens:

• Recipe A (MVS): Uses standard RGB photos (like your phone camera). Good for matte objects, but struggles with shiny things.
• Recipe B (MVPS): Uses photos taken under many different lights. Great for seeing tiny bumps and wrinkles, but requires a lot of setup.
• Recipe C (MVSfP): Uses photos taken through a special polarized filter. Good for shiny surfaces, but the data is tricky to interpret.

The problem? No one had ever tested all three recipes on the same objects under the same conditions. It was like comparing a chef who cooks with a gas stove to one who cooks with a microwave without ever letting them use the same ingredients.

2. The Solution: The "Grand Buffet" (EvalMVX)

The authors built a massive new dataset called EvalMVX. Think of this as a Grand Buffet with 25 different "dishes" (objects).

• The Menu: They chose 25 objects ranging from simple, smooth shapes (like a face) to complex, bumpy ones (like a dragon).
• The Ingredients: The materials vary wildly: some are matte (like a rubber duck), some are shiny (like a silver dog), some are metallic (like a bell), and some are even see-through (like a glass duck).
• The Cooking Process: They photographed every single object from 20 different angles. But here's the magic: for every angle, they took photos under 17 different lighting conditions (including natural light and a special "one-light-at-a-time" setup) using a special polarized camera.

This means they have the "perfect ingredients" to test every recipe fairly. Plus, they have the Ground Truth—the actual, real 3D shape of the object (scanned with a laser)—so they can measure exactly how close the digital model is to reality.

3. The Taste Test: Who Won?

They ran 13 different "chefs" (algorithms) on this buffet. Here is what they found:

• The All-Rounder (MVPS): The method called SuperNormal was the MVP (Most Valuable Player). By using photos taken under different lights, it could figure out the shape of almost anything, from a smooth face to a shiny monkey. It was like a chef who could cook anything perfectly.
• The Speedster: GaussianSurfels was the fastest at making a model, but the quality was a bit "blurry" compared to the others. It's like a fast-food burger: quick to get, but not gourmet.
• The Shiny Specialist (MVSfP): Methods using polarized light were great at handling shiny, metallic objects where standard cameras get confused by reflections. However, they struggled a bit with the "noise" from the camera sensors.
• The Struggler: Standard methods (MVS) that only use regular photos often failed on shiny or transparent objects. They got "blinded" by the glare, just like trying to see a fish in a mirror.

4. The Big Takeaway

The paper concludes that there is no single "best" way to build 3D models.

• If you need speed, use the fast methods.
• If you need high detail on a complex object, use the "lighting" method (MVPS).
• If you are scanning shiny metal, the polarized method (MVSfP) is your best friend.

Why Does This Matter?

Imagine you are a video game developer, a doctor trying to model a patient's organ, or an engineer designing a car. You need to know which tool to pick. EvalMVX is the User Manual that tells you: "If your object is shiny, don't use Recipe A; use Recipe C. If you have a lot of time, use Recipe B for the best results."

It turns the confusing world of 3D reconstruction into a clear guide, helping researchers and developers choose the right tool for the job, saving time and improving the quality of digital worlds.

Technical Summary

1. Problem Statement

Current advancements in neural 3D surface reconstruction have led to three primary paradigms:

• Multiview Stereo (MVS): Uses RGB images to recover shape.
• Multiview Photometric Stereo (MVPS): Uses varying illumination to estimate surface normals and refine shape.
• Multiview Shape from Polarization (MVSfP): Uses polarization cues (Angle of Polarization) to recover shape, particularly effective for reflective surfaces.

The Gap: Existing real-world datasets are fragmented. They typically benchmark only one of these techniques (e.g., DTU for MVS, DiLiGenT-MV for MVPS, PANDORA for MVSfP) or lack aligned Ground Truth (GT) 3D meshes necessary for quantitative evaluation. Furthermore, synthetic datasets fail to capture real-world physical noise and complex reflectance properties. Consequently, there is no unified framework to quantitatively compare the working ranges, trade-offs, and performance boundaries of MVS, MVPS, and MVSfP on the same objects under diverse conditions.

2. Methodology: The EvalMVX Dataset

To address these limitations, the authors propose EvalMVX, the first real-world dataset designed for the simultaneous quantitative benchmarking of all three MVX techniques.

Data Acquisition

• Objects: 25 objects with diverse geometries (from smooth to highly complex) and reflectances (diffuse, specular, metallic, and translucent).
• Hardware: A Lucid Triton RGB polarization camera with a 16mm lens and a custom rig holding 16 high-power LEDs.
• Capture Protocol:
  • Views: 20 distinct viewpoints per object.
  • Illumination:
    • 1 image under static Environment Light.
    • 16 images under One-Light-At-a-Time (OLAT) conditions.
  • Total: 8,500 images (25 objects × 20 views × 17 light conditions).
• Ground Truth: High-fidelity 3D meshes are scanned using an EinScan-SP scanner.

Data Processing & Alignment

• Polarization Processing: Raw Bayer images are demosaiced into four polarization angles ($0^\circ, 45^\circ, 90^\circ, 135^\circ$) to compute Stokes vectors, Angle of Polarization (AoP), and Degree of Polarization (DoP).
• Unpolarized Extraction: Depolarized RGB images are synthesized from the polarization data to support MVS and MVPS benchmarks.
• GT Alignment: A differentiable rendering pipeline (PyTorch3D) is used to align the scanned GT meshes with the camera poses. This minimizes the Binary Cross-Entropy (BCE) loss between rendered silhouettes and observed masks, ensuring precise alignment for quantitative metrics.

3. Key Contributions

1. Unified Dataset: EvalMVX is the first dataset providing aligned GT shapes for 25 diverse objects, enabling simultaneous evaluation of MVS, MVPS, and MVSfP.
2. Comprehensive Benchmarking: The authors evaluate 13 state-of-the-art methods (4 MVS, 4 MVPS, 3 MVSfP, and baselines) across the dataset.
3. Working Range Analysis: The study identifies specific scenarios where each technique excels or fails, analyzing the trade-offs between capture effort (number of lights/views) and reconstruction quality.
4. Methodological Improvement: The authors propose an enhancement to the top-performing MVPS method (SuperNormal) by integrating per-view depth maps as a geometric continuity regularizer, significantly reducing reconstruction errors.

4. Key Results & Findings

Performance Overview (Chamfer Distance - CD)

• MVPS (Best Overall): SuperNormal achieved the lowest average CD (0.349 mm) and median CD (0.314 mm). It demonstrated robustness across diffuse and specular materials, leveraging surface normals for high-fidelity recovery.
• MVS: NeRO performed best on highly reflective metallic surfaces (e.g., SilverDog, Yoga) due to its integrated directional encoding, while NeuS2 was competitive on diffuse surfaces.
• MVSfP: PISR was the top performer among polarization methods. While generally less accurate than MVPS/MVS, MVSfP showed superior robustness on objects with complex reflectance but simple geometry (e.g., the translucent Duck).

Specific Insights

• Reflectance vs. Geometry:
  • MVPS excels on complex geometries regardless of material.
  • MVSfP is highly effective for complex materials (specular/metallic) but struggles with complex geometry.
  • MVS is highly sensitive to surface reflectance; standard methods fail on strong specularities without specific handling (like NeRO).
• Failure Cases:
  • MVPS: Struggled with metallic and translucent objects when the initial normal estimation (via Uni-MS-PS) was inaccurate due to complex reflectance.
  • MVSfP: Performance degraded due to sensor noise and inconsistencies between observed AoP and the theoretical azimuth map.
  • Shadows: All methods struggled with cast shadows (e.g., the "Swan" object).
• Efficiency: GaussianSurfels was the fastest but least accurate. SuperNormal offered the best trade-off between speed and quality.

Proposed Enhancement

The authors introduced Integrated Depth Regularization for SuperNormal. By integrating surface normals to create a depth map and using it as a regularization term, they reduced the average CD from 0.349 mm to 0.324 mm, effectively mitigating mesh discontinuities caused by inaccurate normal inputs.

5. Significance and Future Directions

• Standardization: EvalMVX provides a critical standard for comparing neural reconstruction methods, moving beyond single-task benchmarks.
• Guidance for Practitioners: The results offer clear guidelines for selecting methods based on scene constraints (e.g., use MVPS for high detail, MVSfP for metallic objects with sparse views).
• Open Problems Identified:
  1. 3DGS Limitations: 3D Gaussian Splatting representations currently lack geometric consistency compared to SDF-based approaches.
  2. Reliability of Priors: Learning-based normal estimation and polarization cues are prone to domain gaps and sensor noise; robustness needs improvement.
  3. Efficiency: Current normal acquisition methods are computationally expensive; lightweight solutions are needed for scalability.

The paper concludes by acknowledging the rapid evolution of the field and plans to release an online platform for continuous benchmarking, with future work aiming to incorporate depth and infrared data.