Technical Summary: VidSplat

Problem Statement

Reconstructing 3D geometry from multi-view images is a fundamental task in computer vision, with recent breakthroughs achieved through Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). However, these paradigms rely heavily on multi-view photometric consistency, causing significant degradation when only a few input views are available. In sparse-view settings, the optimization becomes ill-posed and under-constrained.

Existing solutions generally fall into two categories: generalizable methods that pretrain on large datasets to learn cross-view correspondences, and scene-specific methods that overfit single scenes using monocular or stereo priors. While these approaches improve performance, they remain constrained to recovering only the visible regions of the given input views. They struggle to infer geometry in unseen, occluded, or weakly constrained regions beyond the input field of view, resulting in incomplete and fragmented surfaces. Furthermore, recent generative approaches for novel view synthesis often require costly pretraining, fail to recover extrapolated regions (producing voids), or generate visually plausible but geometrically inconsistent renderings.

Methodology: VidSplat

To address these limitations, the authors propose VidSplat, a training-free generative reconstruction framework. VidSplat leverages powerful video diffusion priors to iteratively synthesize novel views that compensate for missing input coverage, enabling the recovery of complete 3D scenes from sparse inputs (even a single image). The framework integrates generation and reconstruction through an iterative optimization loop involving two primary stages: initialization and training.

1. Optimization Framework

The process begins with sparse input views $\{V_{input}\}_0$.

• Initialization: An initial point cloud is constructed using DUSt3R. Based on this, the system samples visibility-based camera trajectories and renders RGB and mask images. These are fed into a camera-controlled video diffusion model (VDM) to synthesize 3D-consistent video clips. Keyframes from these clips form a generated set $\{V_{gen}\}_0$, which is merged with the input to create a denser point cloud for initializing 2D Gaussians.
• Iterative Training: During Gaussian training, the system performs multiple refinement cycles. In each cycle $t$, new trajectories are sampled, and new sequences are generated to form $\{V_{gen}\}_t$. These are merged with the current view set to expand coverage. Unlike the initialization phase, generated views are not used for re-initialization but are backprojected into 3D space via monocular depths to create additional Gaussian centers. The reconstruction is iteratively refined, and final surfaces are extracted using marching tetrahedra.

2. Visibility-Based Camera Pose Sampling

To effectively explore under-covered regions, VidSplat employs a novel sampling strategy. Instead of simple interpolation or circular paths, it constructs trajectories where the camera orbits intersection points between camera rays and the scene surface on a sphere. Trajectories are validated based on the depth and mask of keyframes to ensure they avoid near-plane occlusions (e.g., walls blocking the view) and cover an appropriate amount of unseen regions. To mitigate hallucination accumulation, the trajectory is extended by 25% during generation, and the tail frames are discarded.

3. Geometry-Guided Video Generation

The core innovation lies in a training-free, stage-wise denoising strategy designed to ensure 3D consistency. Standard video generation often introduces content drift and dynamics that violate geometric constraints. VidSplat addresses this by:

• Inversion: Employing diffusion inversion on rendered RGB and mask images to create noisy latents that encode layout priors induced by camera motion.
• Adaptive Blending: At each denoising step, the model blends the predicted latent with the inverted reference latent using a spatial mask $M(t)$.
• Three-Stage Control:
  1. Early Stage ($T_0 \to T_1$): The denoising direction in known regions is strictly constrained to follow the rendered references ($M(t) = M_{ref}$). This anchors the scene structure and prevents content drift.
  2. Middle Stage ($T_1 \to T_2$): The constraint is gradually relaxed, allowing the model to refine imperfect renderings.
  3. Final Stage ($T_2 \to 0$): The constraint is fully removed to synthesize realistic local details and repair artifacts.

4. Loss Function

The optimization objective combines losses for input views and generated views. For input views, it uses standard photometric loss, regularization, and a normal prior. For generated views, the photometric loss is replaced with a Laplacian loss to mitigate high-frequency artifacts, and a per-pixel confidence map derived from point cloud fusion is used to weight the supervision.

Key Contributions

1. Generative Framework: A novel framework that iteratively incorporates video generation into Gaussian Splatting reconstruction, allowing for continuous refinement and the recovery of complete 3D scenes from sparse inputs.
2. Geometry-Guided Denoising: A training-free, stage-wise denoising strategy that adaptively guides the video diffusion process toward underlying geometry, ensuring 3D consistency without requiring model finetuning.
3. State-of-the-Art Performance: The method achieves superior results in both surface reconstruction and novel view synthesis on widely adopted real-world benchmarks, outperforming existing sparse-view and generative methods.

Experimental Results

The authors evaluated VidSplat on diverse real-world datasets, including Tanks and Temples, Replica, and DL3DV, covering both indoor and outdoor scenarios.

• Surface Reconstruction: On the Tanks and Temples and Replica datasets, VidSplat achieved significantly better quantitative metrics (Chamfer Distance, F-Score, Normal Consistency) compared to baselines like FSGS, MAtCha, and GuidedVD. Visualizations demonstrate the ability to reconstruct complete, high-fidelity surfaces without holes, even with only 5 input views.
• Novel View Synthesis: On the DL3DV dataset, the method achieved the best PSNR, SSIM, and LPIPS scores for both indoor and outdoor scenes, successfully rendering high-quality details in regions sparsely covered by inputs.
• Video Generation: When compared to other camera-controlled video diffusion methods (e.g., ViewCrafter, TrajectoryCrafter), VidSplat demonstrated superior consistency with ground truth geometries and greater generative diversity.
• Single-View Generation: The framework successfully demonstrated the ability to generate complete 3D scenes from a single input image, recovering large invisible regions.

Significance

The paper claims that VidSplat represents a significant advancement in sparse-view 3D reconstruction by effectively bridging the gap between generative video models and geometric reconstruction. By introducing a geometry-guided denoising strategy, the authors overcome the critical limitation of previous generative methods that fail to maintain 3D consistency or infer unseen geometry. The framework's ability to function without training (training-free) and its robustness across different video diffusion backbones (including SVD and Hunyuan) highlight its practicality and scalability. Ultimately, VidSplat enables the recovery of complete, high-fidelity 3D scenes from extremely limited data, opening new possibilities for applications in digital content creation, VR/AR, and embodied intelligence where dense multi-view capture is impractical.