💻 computer science

Can These Views Be One Scene? Evaluating Multiview 3D Consistency when 3D Foundation Models Hallucinate

This paper introduces \benchmark, a robust benchmark and a family of COLMAP-based metrics that demonstrate how current 3D foundation models can hallucinate geometry from unrelated or noisy inputs, offering significantly more reliable consistency evaluation than existing neural methods by better aligning with human judgment.

Original authors: Soumava Paul, Prakhar Kaushik, Alan Yuille

Published 2026-05-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Soumava Paul, Prakhar Kaushik, Alan Yuille

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Magic 3D" Illusion

Imagine you are a detective trying to figure out if a group of photos were taken of the same room.

Scenario A: You have 10 photos of a living room from different angles. Easy! They clearly show one room.
Scenario B: You have 5 photos of a living room and 5 photos of a kitchen mixed together.
Scenario C: You have 10 photos of static noise (like TV snow).

In the past, computers were bad at this. But recently, new "AI 3D models" have been invented that are incredibly fast at turning 2D photos into 3D shapes. The problem is, these AI models are too eager to please.

Even if you feed them a mix of a kitchen and a living room, or just random TV snow, the AI will still try to build a 3D model. It will "hallucinate" (make up) a 3D shape that looks plausible, even though it's physically impossible.

The paper argues that the current tools used to grade these AI models are broken. They look at the AI's "hallucinated" 3D shape and say, "Wow, this looks consistent! Great job!" when the input was actually nonsense.

The Analogy: The Over-Confident Architect

Think of the AI reconstruction models (like DUSt3R, MASt3R, VGGT) as over-confident architects.

If you give them a clear blueprint (real photos of one room), they build a perfect house.
If you give them a blueprint that is half kitchen and half living room, they don't say, "This doesn't make sense." Instead, they build a weird, impossible house that somehow combines both.
If you give them a blank piece of paper (random noise), they still build a house, perhaps a floating dome or a flat plane, just because they are trained to always build something.

The current grading systems (like MEt3R) are like inspectors who only look at the finished house. They see a house standing there and give the architect an "A+." They fail to realize the architect was handed a blank piece of paper to start with.

What the Authors Did: The "SysCON3D" Stress Test

The authors created a new testing ground called SysCON3D. Instead of just testing the AI on good photos, they deliberately broke the inputs to see if the grading tools could catch the lies. They tested three types of "fake" inputs:

The Mix: Photos from two different rooms mixed together.
The Copy: The exact same photo repeated 10 times.
The Noise: Pure random static (like TV snow).

The Result: The old grading tools failed miserably. They gave high scores to the noise and the mixed rooms, thinking they were perfect 3D scenes. The AI architects had successfully tricked the inspectors.

The Solution: Two New Approaches

The authors propose two ways to fix this broken grading system.

1. The "Smarter Inspector" (Neural Metrics)

They realized the problem wasn't just the architect; it was how the inspector counted the mistakes. The old inspectors just took an average of all the errors. If the AI made a huge mistake on one photo but was perfect on the others, the average looked okay.

The authors created a new family of inspectors that look at the distribution of errors. Instead of just averaging, they ask, "Are there any weird outliers?"

The Winner: A new metric called MASt3R-W-IMQ. It is much better at spotting that something is wrong (up to 3x better than the old standard), but it still has a weakness: if the AI hallucinates a shape that looks too smooth, even this smart inspector can get fooled.

2. The "Skeptic" (Classical Geometry)

The authors also went back to "old school" methods (using tools like COLMAP). These methods don't try to guess the 3D shape using AI magic. Instead, they look for hard evidence:

Do the features in the photos actually match up?
Can we mathematically prove these cameras were looking at the same object?

If the answer is "No" (like with TV snow or mixed rooms), the Skeptic simply says, "I cannot build a model." It refuses to give a score.

Why this is good: In the real world, if a computer says "I can't verify this," that is a very useful piece of information. It tells you, "This input is garbage."
The Result: These "Skeptic" metrics aligned 4 times better with human judgment. When humans looked at the fake inputs, they said, "This is nonsense." The old AI metrics said, "This is great." The Skeptic metrics said, "This is nonsense."

The Human Test: Do We Agree?

The authors asked real humans to watch videos generated by different AI methods and vote on which ones looked like a real, consistent 3D scene.

The Old AI Metrics: Often disagreed with humans. They liked methods that looked pretty but were geometrically broken.
The New "Skeptic" Metrics: Agreed with humans almost perfectly. When humans saw a glitchy, impossible video, the Skeptic metrics gave it a low score.

The Main Takeaway

The paper concludes that we cannot blindly trust AI models to evaluate their own work or the work of other AI models.

AI Architects are great at building, but they will build a castle on a cloud if you ask them to.
AI Inspectors that rely on the Architect's output will praise the castle on the cloud.
We need Skeptics: We need tools that check if the foundation (the photos) actually supports the building. If the photos are nonsense, the tool should admit it can't find a 3D scene, rather than inventing one.

In short: If you feed a computer nonsense, it will try to make sense of it. To evaluate 3D consistency, we need tools that are brave enough to say, "This is nonsense," rather than trying to force a pattern where none exists.

Technical Summary: Can These Views Be One Scene? Evaluating Multiview 3D Consistency when 3D Foundation Models Hallucinate

Problem Statement

Multiview 3D evaluation typically assumes that input images are observations of a single, static 3D scene. However, in Novel View Synthesis (NVS) and sparse-view reconstruction, this assumption is frequently violated. Inputs or generated outputs may contain artifacts, outlier frames, repeated views, or noise, yet still receive high 3D consistency scores from current evaluation metrics.

Existing ground-truth-free metrics, such as MEt3R, rely on learned 3D reconstruction backbones (e.g., DUSt3R, MASt3R, VGGT, Fast3R) to infer geometry and measure consistency. The core problem identified in this work is that these backbones are trained to produce geometry, not to reject invalid inputs. Consequently, they can "hallucinate" dense 3D structure and cross-view support for inputs that do not define a coherent scene (e.g., unrelated scene mixtures, repeated images, or Gaussian noise). Metrics built upon these backbones inherit this failure, reporting spuriously high consistency for invalid inputs because they evaluate feature residuals on hallucinated correspondences rather than verifying the existence of geometric evidence.

Methodology

1. SysCON3D Benchmark

To systematically evaluate metric robustness, the authors introduce SysCON3D, a controlled benchmark that injects specific inconsistencies into multiview image sets from the Mip-NeRF360 dataset. The benchmark includes:

L0 (Consistent): Clean views from a single scene.
L1–L3 (Cross-Scene Mixtures): Increasing densities of foreign views (single outlier, controlled mixture, random mixture).
SysCON3D-N (Noise): Patched Gaussian noise and full Gaussian noise.
The benchmark tests whether metrics can correctly assign lower consistency scores to these inconsistent groups compared to clean sets.

2. Parametric Neural Metric Family

The authors propose a unified framework decomposing neural metrics into three components:
$m(\mathcal{I}) = \mathcal{A}(\rho(B(\mathcal{I}), \phi(\mathcal{I})))$

Backbone ( $B$ ): A reconstruction model (MASt3R, Fast3R, VGGT) that produces 3D geometry.
Residual ( $\rho$ ): A function computing feature disagreement (using DINOv2/FeatUp) via either Warp-based (view-pair warping) or Point-Consistency (PC) (3D point projection) methods.
Aggregation ( $\mathcal{A}$ ): A function summarizing residuals. The authors compare standard mean aggregation (used in MEt3R) against distributional aggregations (MMD, Inverse Multiquadric (IMQ), Energy distance) which preserve the shape of the residual distribution and are more sensitive to outliers.

3. COLMAP-Based Metrics

To address the limitations of learned backbones, the authors introduce metrics based on classical geometric verification using COLMAP. These metrics treat reconstruction failure as a signal of inconsistency:

Geometric-Photometric Consistency (GPC): Measures the agreement between geometric and photometric depth maps, weighted by pixel validity.
Integrated Consistency Mass (ICM): Quantifies the total reliable dense 3D support.
Coverage-Weighted GPC (W-GPC): Combines GPC with the angular coverage of registered cameras.
Crucially, if COLMAP fails to register cameras or produce dense maps (due to lack of geometric evidence), the metric score drops to zero, explicitly penalizing invalid inputs.

4. Human Alignment Study

The authors conducted a structured human study involving 11 participants (including 2 experts) to evaluate 8 NVS methods. Participants compared anonymized NVS videos based on 3D consistency, visual realism, and plausibility. Rankings were aggregated using a weighted Elo protocol to establish a ground-truth human preference for 3D consistency.

Key Results

Robustness on SysCON3D

Backbone Hallucination: All tested backbones (VGGT, MASt3R, DUSt3R, Fast3R) produced dense point clouds and non-zero overlap for Gaussian noise and cross-scene mixtures, confirming they hallucinate geometric support rather than abstaining.
Aggregation Impact: Switching from mean aggregation to distributional aggregation (specifically IMQ) significantly improved robustness. The best neural variant, MASt3R-W-IMQ, achieved a 71% overall win rate on SysCON3D, compared to 23% for the baseline MEt3R (MASt3R-W-Base).
Limitations: Even with improved aggregation, neural metrics struggled with single outliers (L1) and Gaussian noise, as the hallucinated geometry occurred before the aggregation step.

Human Alignment

Neural Metrics: MASt3R-W-IMQ showed moderate alignment with human judgments (Spearman $\rho \approx 0.48$ at $K=6,9$ ), outperforming MEt3R but still failing to perfectly rank methods.
COLMAP Metrics: The W-GPC metric achieved the strongest alignment with human judgments, reaching a Spearman correlation of 0.98 on DL3DV and 0.98 on Mip-NeRF360. This represents up to a 4 $\times$ improvement over MEt3R.
Failure Modes: COLMAP metrics correctly assigned near-zero scores to inconsistent inputs where neural metrics failed, demonstrating that explicit geometric verification is a more reliable signal for scene-level consistency.

Significance and Claims

The paper claims that the reliability of ground-truth-free 3D evaluation is fundamentally compromised when using learned reconstruction models as evaluators in uncurated settings. The central finding is that modern 3D foundation models hallucinate geometric support for inputs that lack valid multiview evidence, causing downstream metrics to validate invalid data.

The authors argue that:

Distributional aggregation improves neural metrics but cannot fully correct unsupported geometry produced by the backbone.
Classical geometric verification (via COLMAP) provides a complementary, failure-aware signal. By treating the inability to register or reconstruct as a negative score, these metrics align significantly better with human judgments of 3D consistency.
Practical Recommendation: For robust evaluation of NVS outputs, the authors recommend reporting W-GPC alongside ICM and registration rates. Neural metrics like MASt3R-W-IMQ can serve as secondary diagnostics but should not override explicit geometric verification, especially in safety-critical applications where hallucinated geometry could be mistaken for evidence.

The work does not claim to have solved the reconstruction problem itself but rather provides a diagnostic framework and robust metrics to prevent the misinterpretation of hallucinated 3D consistency in generated content.