💻 computer science

GaussVid: Sparse-View Gaussian Splatting with 3D-Aware Video Diffusion Priors

GaussVid addresses the artifacts in sparse-view 3D Gaussian Splatting by introducing a 3D-aware video restoration framework that leverages a large-scale 3DGS dataset and a camera-conditioned geometric prior to ensure multi-view consistency and high-fidelity reconstruction.

Original authors: Xinhui Liu, Can Wang, Wei Jiang, Wei Wang, Dong Xu

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinhui Liu, Can Wang, Wei Jiang, Wei Wang, Dong Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to rebuild a detailed sculpture, but you are only allowed to look at it from a handful of angles. If you try to guess what the rest of the object looks like based on those few glimpses, your mind will inevitably fill in the gaps with mistakes. You might imagine a hand where there is none, or blur the sharp edge of a table into a soft smudge. This is the fundamental challenge facing a powerful new technology called 3D Gaussian Splatting. This method has revolutionized how computers create realistic, three-dimensional scenes from photographs, allowing viewers to walk through digital environments with stunning clarity. However, when the input data is sparse—meaning there are only a few photos to work with—the resulting 3D models often suffer from ghostly artifacts, floating blobs of color, and distorted structures that break the illusion of reality.

For years, researchers have tried to fix these errors by adding strict mathematical rules to the computer's thinking process, forcing it to keep surfaces smooth or colors consistent. More recently, scientists have turned to artificial intelligence models trained on millions of images, hoping these systems could "hallucinate" the missing details correctly. Yet, these image-based AI models often fail the test of three-dimensional logic. Because they were trained on flat pictures, they do not truly understand how a scene looks from different angles simultaneously. They might make a building look perfect from the front, but when you move to the side, the AI might have drawn a window in a place where a wall should be, creating a jarring inconsistency. The goal, then, has been to find a way to guide these powerful AI imaginations with a true understanding of 3D space.

A team of researchers has developed a new approach called GaussVid to solve this specific problem. Instead of trying to fix the 3D model directly or relying on flat-image AI, they treat the reconstruction process as a video restoration task. They realized that if you have a clear view of a scene at the beginning and the end of a camera movement, the AI can be taught to fill in the blurry, distorted middle frames with high precision. To do this, the team first built a massive training library. They took thousands of real-world video clips and deliberately degraded them, simulating the exact kind of ghostly errors and blurriness that happen when 3D models are built from too few photos. This created a paired dataset where the computer could see both the "broken" version and the perfect original, allowing it to learn how to repair the damage.

The core of their innovation lies in how they teach the AI to understand the camera's position. Previous attempts to use AI for 3D tasks often tried to estimate the 3D shape of the object first, hoping to use that shape as a guide. The researchers found this approach flawed because, in sparse-view situations, the estimated shape is often noisy and wrong, which only confuses the AI further. Instead, GaussVid bypasses the guesswork entirely. It feeds the AI the exact, known positions of the camera as it moves through the scene. The system breaks this camera information down into two distinct parts: the direction the camera is pointing and the distance from the center of the scene. It then injects this geometric data directly into the AI's processing layers, acting as a rigid, noise-free skeleton that holds the video generation in place. This ensures that as the AI fills in the missing details, it respects the true geometry of the scene, keeping the object consistent no matter which angle the viewer chooses.

To make the learning process effective, the researchers did not throw all the difficult examples at the AI at once. They designed a curriculum that started with easy tasks, where the missing views were close together and the errors were mild. As the AI mastered these, the difficulty gradually increased, introducing wider gaps between views and more severe distortions. This step-by-step approach prevented the system from becoming overwhelmed by the most chaotic examples before it had learned the basic rules of reconstruction. The results were striking. When tested on various scenes, from indoor rooms to outdoor landscapes, the new method produced images that were significantly sharper and more geometrically accurate than previous techniques. It successfully removed the floating artifacts and blurred structures that plagued earlier models, restoring fine details like the edges of shelves and the texture of wood.

The study demonstrates that by combining a video-based AI model with precise, known camera data, it is possible to restore high-quality 3D scenes even when the original input is extremely limited. The researchers showed that their method not only fixed the visual errors but also maintained a consistent structure across all viewpoints, a feat that image-only AI models struggled to achieve. By treating the problem as a guided video restoration task rather than a blind 3D guess, the team provided a reliable path forward for creating robust, artifact-free digital environments from sparse data. This work suggests that the key to better 3D reconstruction may not be in building more complex 3D models, but in teaching our AI tools to understand the camera's journey through space with absolute clarity.

Technical Summary: GaussVid

Problem Statement

While 3D Gaussian Splatting (3DGS) has achieved remarkable success in novel view synthesis under dense multi-view supervision, reconstructions from sparse views frequently suffer from significant artifacts. These include geometric ambiguities, overfitting, "floaters" (floating artifacts), blurred structures, and poor generalization to unseen viewpoints.

Existing solutions generally fall into two categories, both with limitations:

Regularization-based methods: These impose geometric priors (e.g., depth smoothness, normal consistency) or appearance constraints. However, they remain sensitive to noise and offer limited improvements under extreme sparse-view settings.
Diffusion-based methods: Recent approaches leverage 2D image diffusion models or video diffusion models as priors. While 2D priors improve semantic realism, they lack explicit 3D awareness, leading to cross-view inconsistencies and geometric distortions. Video priors offer temporal consistency but often rely on estimated 3D geometry (e.g., via VGGT) as a conditioning signal. Under sparse views, these estimated geometries are often noisy and ambiguous, providing weak or misleading guidance to the diffusion process.

Methodology: GaussVid

The authors propose GaussVid, a novel 3D-aware video restoration framework designed to enhance sparse 3DGS reconstructions. The core philosophy is to treat sparse 3DGS restoration as a frame-conditioned reconstruction problem, where high-quality intermediate frames are recovered using a video diffusion model conditioned on clean anchor frames and explicit camera geometry.

1. Data Construction (GaussVid Dataset)

Since no existing dataset provides paired artifact-corrupted and clean video clips with camera poses, the authors constructed a large-scale dataset from ~5,000 scenes in DL3DV:

Artifact Simulation: They perform controlled 3DGS optimization, intentionally terminating early (at 8,000 iterations) to capture natural sparse-view artifacts (floaters, blur) before the model degenerates into severe overfitting.
Clip Sampling: They sample 5-frame clips using a "clean–noisy–clean" strategy. The first and last frames serve as clean anchors ( $I_s, I_e$ ), while the three intermediate frames are the corrupted renderings ( $\hat{F}$ ) to be restored.
Difficulty Control: The sparsity interval ( $n$ ) is varied to create a curriculum of difficulty levels, ranging from mild to severe artifacts.

2. 3D-Aware Video Restoration Model

The method builds upon Wan2.1-VACE, a first-to-last frame conditioned video model, extending it with specific 3D awareness mechanisms:

Camera-Conditioned Geometric Prior: Instead of conditioning on estimated 3D geometry (which is noisy), the model conditions directly on camera poses (known from the rendering process). These poses are encoded as Plücker coordinates (ray direction and moment).
Plücker Token Encoder:
- Decomposition: The Plücker field is decomposed into two distinct streams: a moment stream (sensitive to translation) and a direction stream (sensitive to rotation).
- Cross-Gating Fusion: These streams are processed by dedicated 3D convolutions and fused via cross-gating to preserve their distinct physical properties.
Dual-Stream Camera Injection:
- To prevent noisy video features from corrupting the geometric priors, the authors employ a dual-stream mechanism.
- Noise-Invariant Modulation: Camera tokens are modulated independently of the diffusion timestep (using a fixed zero timestep), treating geometry as a noise-free prior.
- Asymmetric Attention: An asymmetric mask allows video tokens to attend to camera tokens (absorbing geometric context) while preventing camera tokens from attending to noisy video tokens. This shields the geometric signal from artifact patterns.

3. Training Strategy

The model is optimized within a flow matching framework with two key stabilizing strategies:

Progressive Latent Reconstruction Loss: An auxiliary L1 loss is applied to the estimated clean latent. This loss is weighted progressively, focusing supervision at lower noise levels where reconstruction is most accurate.
Difficulty-Aware Curriculum Learning: The training process gradually exposes the model to increasingly difficult samples (larger sparsity intervals $n$ ). This prevents the model from being overwhelmed by high-difficulty, noisy gradients before it learns basic reconstruction patterns.

Key Contributions

Novel Formulation & Dataset: The paper formulates sparse 3DGS restoration as a frame-conditioned reconstruction problem and constructs a large-scale, paired dataset with diverse camera trajectories and controllable artifact difficulties.
3D-Aware Video Prior: The authors introduce a camera injection module that encodes Plücker coordinates into separate moment and direction streams. This is integrated via a dual-stream, asymmetric attention mechanism, providing a reliable, noise-free geometric prior that does not degrade as views become sparser.
Stabilized Fine-Tuning: A progressive training strategy combining a curriculum learning schedule and a progressive latent reconstruction loss is proposed to stabilize the adaptation of the video diffusion model.

Results

Extensive experiments were conducted on the DL3DV and Zip-NeRF benchmarks, comparing against optimization-based methods (FSGS, CoR-GS), 2D diffusion priors (DiFix3D), and video diffusion priors (4DSloMo, GSFIXER).

Quantitative Performance: GaussVid achieves state-of-the-art performance in PSNR and SSIM, indicating superior pixel-level fidelity and structural consistency. It also maintains competitive LPIPS scores (perceptual quality), outperforming methods that hallucinate content or lose geometric consistency.
Qualitative Improvements: The method effectively removes floaters and structural distortions while preserving fine details (e.g., railing edges, textures) and ensuring multi-view consistency.
Ablation Studies:
- Removing camera conditioning leads to a significant drop in performance, confirming the necessity of explicit 3D awareness.
- Decomposing Plücker coordinates into moment and direction streams yields better results than monolithic encoding.
- The curriculum learning and progressive loss strategies are shown to be complementary, with the curriculum handling large-scale structural artifacts and the progressive loss refining fine-grained textures.

Significance

The paper claims that GaussVid serves as a general enhancement framework that can be seamlessly integrated to improve existing sparse 3DGS methods. By bridging the gap between 2D video generation and 3D multi-view constraints through camera-aware conditioning, the approach overcomes the view-inconsistency issues inherent in previous diffusion-based methods. Furthermore, the restored frames can be fed back to refine the 3DGS representation itself, offering a pathway to higher-quality 3D reconstructions from sparse inputs. The work demonstrates that leveraging video diffusion models with explicit, noise-free geometric priors is a viable and effective strategy for addressing the ambiguity of sparse-view 3D reconstruction.