SyncFix: Fixing 3D Reconstructions via Multi-View Synchronization

1. Problem Statement

Sparse-view 3D reconstruction methods, such as NeRF and 3D Gaussian Splatting (3DGS), often suffer from severe artifacts when trained with limited input views. These artifacts include "floaters" (floating noise), fragmented surfaces, blurred textures, and geometric distortions.

Recent state-of-the-art approaches attempt to fix these issues by leveraging 2D generative priors (diffusion models). However, these methods typically treat refinement as an independent, per-view task. They render a novel view, refine it using a diffusion model conditioned on a reference view, and then distill the result back into the 3D representation.

• The Core Issue: By refining views independently, these methods ignore cross-view geometric and semantic constraints. A structure might be hallucinated differently in View A compared to View B. When these conflicting signals are fused back into the 3D model, it leads to unstable optimization, appearance drift, and degraded geometry.
• Goal: The paper aims to enforce cross-view consistency directly during the refinement process, ensuring that all generated views agree on the underlying 3D structure before being distilled.

2. Methodology: SyncFix

SyncFix formulates the refinement problem not as independent 2D generation, but as a joint conditional generation problem over multiple views. It utilizes a Latent Bridge Matching (LBM) framework.

A. Core Formulation

Instead of learning marginal mappings $P(x^{(i)}_{GT} | x^{(i)}_D)$ for each view $i$, SyncFix learns a joint conditional distribution $P(X_{GT} | X_D)$, where $X_D$ is the set of distorted views and $X_{GT}$ is the set of clean target views. This forces the model to condition the correction of one view on the information present in all other views.

B. Latent Bridge Matching (LBM)

Unlike traditional diffusion models that inject noise and perform iterative denoising, SyncFix uses LBM to learn a deterministic transport map between the distorted latent distribution and the clean latent distribution.

1. Encoding: Distorted views ($x_D$) and clean targets ($x_{GT}$) are encoded into latent tensors ($z_D$ and $z_{GT}$) using a pre-trained VAE.
2. Bridge Construction: A continuous path (bridge) is defined between the source and target latents:
   $$Z_t = (1-t)Z_D + tZ_{GT} + \sigma\sqrt{t(1-t)}\epsilon$$
   where $t \in [0,1]$ is the time step and $\epsilon$ is Gaussian noise.
3. Velocity Prediction: A neural network $v_\theta$ is trained to predict the velocity field $v^* = Z_{GT} - Z_D$. The loss minimizes the difference between the predicted velocity and the true vector field.

C. Multi-View Synchronization via Cross-View Attention

To achieve joint conditioning without explicit camera pose supervision:

• Token Concatenation: Latent tokens from all views in a batch are concatenated.
• Global Attention: The transformer's self-attention layers are modified to allow tokens from different views to attend to one another. This creates a permutation-invariant mechanism where the refinement of View 1 explicitly depends on the features of View 2, View 3, etc.
• Reference View: An optional clean reference view (closest to the training view) can be concatenated to guide the refinement, though the method works without it.

D. Inference

During inference, the model initializes with distorted latents $Z_D$ and solves the learned differential equation in a single step (non-iterative) to obtain the refined latent $Z^*_{GT}$, which is then decoded to produce consistent images.

3. Key Contributions

1. Joint Refinement Framework: SyncFix is the first method to formulate 3D reconstruction refinement as a joint latent bridge matching problem, enforcing consistency during generation rather than post-hoc.
2. Permutation-Invariant Architecture: The model is trained on pairs of views ($N=2$) but naturally generalizes to an arbitrary number of views ($N > 2$) at inference due to its attention-based design.
3. New Evaluation Metric (CVSC): The authors introduce the Cross-View Semantic Consistency (CVSC) metric. Unlike standard image metrics (PSNR, LPIPS) that measure pixel alignment with a single ground truth, CVSC measures semantic feature agreement between matched keypoints across different views, directly quantifying 3D consistency.
4. Zero-Shot Generalization: The method generalizes to unseen datasets (e.g., Nerfbusters) and different reconstruction pipelines (e.g., feedforward AnySplat) without retraining.

4. Experimental Results

The paper evaluates SyncFix on the DL3DV and Nerfbusters datasets, comparing it against baselines like 3DGS, Difix3D+, and Fixer.

• Quantitative Performance:
  • Image Quality: SyncFix outperforms Difix3D+ by 0.77 dB in PSNR and significantly reduces LPIPS and DreamSim scores (indicating better perceptual quality).
  • Consistency: SyncFix achieves the highest CVSC scores, proving it generates views that are semantically consistent with each other.
  • View Count Scaling: Performance improves as the number of input views increases, with diminishing returns at high counts, demonstrating the benefit of additional constraints.
• Qualitative Results:
  • Artifact Removal: SyncFix effectively removes floaters and blurs while maintaining sharp details.
  • Geometric Stability: Unlike Difix3D+, which often hallucinates different shapes for the same object in different views (e.g., a door disappearing in one view but present in another), SyncFix maintains coherent geometry (e.g., consistent door frames, table structures) across all viewpoints.
• Ablation Studies:
  • Removing the multi-view joint training (using a single-view version) improves image quality over baselines but fails to maintain cross-view consistency.
  • Simply feeding multiple views into Difix3D+ (without joint training) does not improve consistency, confirming that the training objective (joint modeling) is the critical factor.

5. Significance and Conclusion

SyncFix addresses a fundamental flaw in current 3D reconstruction refinement: the misalignment between 2D generative priors and the multi-view nature of 3D scenes.

• Paradigm Shift: It moves the field from "refine-then-fuse" (which causes conflicts) to "synchronize-then-refine."
• Efficiency: By using Latent Bridge Matching, it achieves high-quality refinement in a single pass without the computational cost of iterative denoising.
• Broader Impact: The work suggests that generative models for visual scenes should operate on joint multi-view distributions rather than independent view marginals to produce physically plausible and geometrically stable 3D content.

The paper concludes that while pixel-wise metrics (PSNR) may sometimes penalize plausible generative completions that differ from a specific ground-truth rendering, SyncFix succeeds by prioritizing semantic and geometric consistency, which is the true hallmark of a high-quality 3D reconstruction.