Technical Summary: ASV3D – Adapting Single-View 3D Reconstruction with Extra Imagery

1. Problem Statement

Single-view 3D reconstruction aims to recover a complete 3D representation of an object from a single image. While recent advances in conditional generative models (e.g., diffusion models like Wonder3D and Era3D) have improved performance, they face significant limitations when reconstructing unfamiliar objects. The primary issue is the under-constrained nature of the single-image setting: synthesizing views of occluded or unseen regions often leads to geometric misalignment, hallucinations, and cross-view inconsistencies.

Existing attempts to mitigate this by using multiple input images (sparse-view or multi-conditioning methods) often fail because they treat all inputs as a unified condition. This approach can introduce redundancy or ambiguity, particularly when target views require specific visual evidence that only certain input images provide. Furthermore, many sparse-view methods require camera pose estimation, which is often unavailable or difficult to compute for images captured in different environments.

The core challenge addressed by this paper is: How can an additional image of the same object be effectively integrated into a single-view 3D reconstruction pipeline without requiring camera pose estimation or retraining the entire model?

2. Methodology: ASV3D

The authors propose ASV3D, a framework designed to adapt pre-trained single-view 3D reconstruction models to utilize a primary image ($y$) and an additional image ($z$) captured under potentially different conditions (cameras, lighting, backgrounds). The framework operates without requiring camera pose or calibration information.

2.1 Core Mechanism: Consistency-Based Gating

Instead of fusing all input images into a single condition, ASV3D employs a consistency-based gating mechanism ($f$). This mechanism dynamically selects the most informative conditioning image (either $y$ or $z$) for each specific target view ($n$) during the multi-view generation process.

The gate $f: \{1, ..., N\} \rightarrow \{y, z\}$ is formulated to minimize the deviation of the denoising process over time steps. Specifically, for a target view $n$, the gate selects the condition $c$ that minimizes the squared error between the predicted mean $\mu^n_\theta$ and the average mean $\hat{\mu}$ over the late diffusion steps ($T/2$ to $1$):
$$f(n) = \arg\min_{c \in \{y, z\}} \mathbb{E}_{t \in T} \left\{ \| \mu^n_\theta(x^{1:N}_t, t, c) - \hat{\mu} \|^2_2 \right\}$$
This ensures that the generation of each view is conditioned on the input image that provides the most consistent visual evidence for that specific perspective.

2.2 Adaptation Strategies

ASV3D introduces two strategies to leverage the additional image:

1. Zero-Shot Adaptation:

   • Utilizes the pre-trained generative model (e.g., Wonder3D) without updating its weights.
   • Applies the consistency-based gate $f$ to select the optimal condition ($y$ or $z$) for generating each of the $N$ target views.
   • Synthesizes the multi-view set based on these selected conditions, which are then used for 3D reconstruction.

2. Optimised Adaptation:

   • Further refines the pre-trained U-Net model ($\epsilon_\theta$) using the additional image $z$ via a test-time adaptation process.
   • Denoising Loss ($\mathcal{L}_{denoise}$): Adapts the model to the selected conditions to improve reconstruction fidelity.
   • Contrastive Loss ($\mathcal{L}_{cont}$): Specifically targets views conditioned only by the additional image $z$. It treats the generated view as an anchor, the additional image $z$ as a positive sample, and other generated views as negative samples. This enforces cross-view consistency and prevents the model from overfitting to specific poses.
   • The total optimization objective is $\mathcal{L}_{opt} = \mathcal{L}_{denoise} + \lambda \mathcal{L}_{cont}$.

3. Key Contributions

• ASV3D Framework: The first work to adapt single-view 3D reconstruction using an additional image without requiring camera pose estimation or model retraining from scratch. It handles inputs from different cameras and environments.
• Consistency-Based Gating: A novel mechanism that automatically identifies the most suitable conditioning image for each target view, avoiding the suboptimal results of unified multi-conditioning strategies.
• Dual Adaptation Strategies:
  • A zero-shot scheme that improves reconstruction quality immediately by leveraging the gate.
  • An optimised scheme that enhances visual fidelity and cross-view consistency through contrastive learning.
• Empirical Validation: Extensive evaluations on benchmark (Google Scanned Objects) and real-world datasets demonstrating consistent improvements over state-of-the-art baselines.

4. Experimental Results

The authors evaluated ASV3D on two state-of-the-art baselines: Wonder3D and Era3D.

• Quantitative Performance:

  • On the GSO dataset, the optimised ASV3D-Wonder3D variant achieved state-of-the-art results across all metrics: Chamfer Distance (CD), Intersection-over-Union (IoU), PSNR, SSIM, and LPIPS.
  • Specifically, ASV3D-Wonder3D (optimised) reduced CD from 0.0218 (baseline) to 0.0120 and increased IoU from 0.5272 to 0.5738.
  • The method outperformed recent sparse-view reconstruction methods (e.g., FreeSplatter, EscherNet) despite not requiring pose estimation or joint processing of all inputs.

• Qualitative Improvements:

  • Geometry: ASV3D recovered more complete geometries and preserved fine structures (e.g., rims, planar faces) that were often deformed or missing in baseline reconstructions.
  • Consistency: The method resolved cross-view inconsistencies, such as generating objects visible in the additional view but hallucinated in the wrong location by the baseline.
  • Real-World Robustness: In real-world tests with diverse backgrounds and lighting, ASV3D maintained stable geometry and spatial coherence, whereas baselines often failed when the primary and additional images were captured at different distances or angles.

• User Study:

  • A study with 32 participants showed a strong preference for ASV3D over Wonder3D in both 3D reconstruction alignment and multi-view generation realism (Mean scores: 3.75 vs. 2.94 for reconstruction; 3.97 vs. 3.39 for generation).

• Ablation Studies:

  • Confirmed that the consistency-based gate is superior to simple multi-conditioning (concatenating inputs), as the latter can introduce misleading information.
  • Demonstrated that combining denoising and contrastive losses yields the best performance, with $\lambda=0.2$ being the optimal weight.

5. Significance and Claims

The paper claims that ASV3D addresses a critical gap in single-view 3D reconstruction: the inability to effectively utilize auxiliary information without complex pose estimation or retraining. By introducing a consistency-based gating mechanism, the authors demonstrate that the quality of the conditioning strategy is more important than the mere quantity of input images.

The authors modestly note that their work is the first to adapt single-view reconstruction using additional imagery in this manner. They highlight that their method improves robustness under unconstrained multi-view inputs, making it suitable for real-world applications where camera poses are unknown.

Finally, the authors suggest a potential future direction: novel-view input acquisition, where the method could theoretically suggest which auxiliary view to capture next to further improve the reconstruction of a specific object. However, this is presented as a future research direction rather than a current capability.