GaussFusion: Improving 3D Reconstruction in the Wild with A Geometry-Informed Video Generator

GaussFusion introduces a geometry-informed video generator that refines 3D Gaussian splatting renderings by synthesizing temporally coherent, artifact-free frames from depth and normal buffers, thereby significantly improving 3D reconstruction quality in the wild while supporting real-time performance.

The Gist

Imagine you are trying to build a perfect 3D model of a room using only a few photos taken with a shaky hand. You use a popular tool called 3D Gaussian Splatting (let's call it "The Splat Machine") to turn those photos into a 3D world.

The Problem:
The Splat Machine is fast and cool, but it's a bit messy. Because your photos were shaky or incomplete, the 3D model it builds is full of glitches:

• Floaters: Random, ghostly blobs of color floating in mid-air where nothing should be.
• Flickering: The texture of the walls changes color every time you move your head.
• Blur: The details are fuzzy, like looking through a foggy window.
• Geometric Errors: The walls might look wavy or the floor might be tilted.

Existing tools try to fix this by just "smoothing out" the colors, like using a photo editor to blur away a blemish. But this doesn't fix the structural problems (like the floating ghosts or the wavy walls). It's like trying to fix a crooked house by just painting over the cracks.

The Solution: GaussFusion
The authors of this paper created GaussFusion, a new tool that acts like a super-smart 3D architect and a video editor rolled into one.

Here is how it works, using a simple analogy:

1. The "GP-Buffer": The Architect's Blueprint

Most tools only look at the colors of the 3D model (the paint on the walls). GaussFusion is smarter. It creates a special Blueprint (called the GP-Buffer) that includes:

• Color: What the wall looks like.
• Depth: How far away the wall is.
• Normals: Which way the wall is facing (is it flat or tilted?).
• Opacity: Is the wall solid or see-through?
• Uncertainty: A "worry meter" that tells the AI, "Hey, this part of the model is shaky and probably wrong."

Think of this like a doctor not just looking at a patient's skin (color) but also checking their X-ray (depth), blood pressure (normals), and medical history (uncertainty) to diagnose the real problem.

2. The "Video Generator": The Magic Repair Crew

Once the Blueprint is ready, GaussFusion uses a Video Generator (a type of AI that usually makes movies from text) to fix the 3D world.

Instead of just fixing one photo at a time, it treats the 3D model like a movie. It watches the "movie" of the 3D room as you walk through it. Because it has the Blueprint, it knows:

• "That floating ghost blob is wrong because the depth map says there's nothing there." -> It deletes the ghost.
• "That wall is too blurry because the uncertainty meter is high." -> It sharpens the wall.
• "The floor is wavy." -> It straightens the floor.

It essentially re-imagines the scene, filling in missing parts and cleaning up the mess, while making sure everything looks consistent as you move around.

3. Training the AI: The "Fake Disaster" School

To teach this AI how to fix things, the researchers didn't just show it perfect rooms. They built a school of disasters.
They took perfect 3D models and intentionally broke them in every way possible:

• They removed 95% of the photos (making the model guess).
• They started with bad measurements.
• They simulated different types of camera shakes.

This is like a firefighter training in a house that is already on fire, rather than just watching videos of fires. Because the AI saw every possible way a 3D model could go wrong, it learned how to fix any model, whether it was built by a slow, careful method or a fast, "one-shot" method.

The Result

• Before: A 3D room that looks like a glitchy video game with floating ghosts and blurry walls.
• After: A photorealistic, stable 3D room that looks like a high-definition movie.

Why is this a big deal?

• It's Fast: A special version of this runs at 16 frames per second, meaning you could use it in real-time virtual reality games or live video calls.
• It's Universal: It works on 3D models built by any method, not just one specific type.
• It's Smart: It doesn't just guess; it uses the actual geometry (the shape and structure) to know exactly what to fix.

In short: GaussFusion takes a messy, broken 3D reconstruction and uses a "smart blueprint" to guide a video AI in cleaning it up, turning a glitchy mess into a perfect, realistic world.

Technical Summary

1. Problem Statement

While 3D Gaussian Splatting (3DGS) has become a dominant representation for high-quality 3D reconstruction and real-time rendering, existing methods suffer from significant artifacts, particularly in "in-the-wild" scenarios (sparse views, incomplete coverage, or noisy camera poses). Common issues include:

• Visual Artifacts: Floaters (floating noise), flickering, blurring, and ghosting.
• Geometric Errors: Inconsistent depth, missing regions, and "needle-like" over-elongated Gaussians.
• Pipeline Limitations: Current refinement methods are often limited to specific reconstruction paradigms (either optimization-based or feed-forward) and rely solely on RGB conditioning. This lack of geometric awareness leads to ambiguity when correcting complex artifacts like large floaters or missing geometry. Furthermore, models trained on one paradigm often fail to generalize to the other.

2. Methodology: GaussFusion

GaussFusion addresses these limitations by introducing a geometry-informed video-to-video generation framework. Instead of treating 3D reconstruction as a static image problem, it treats the refinement of novel views as a video generation task conditioned on rich geometric data.

A. Gaussian Primitives Buffer (GP-Buffer)

The core innovation is the GP-Buffer, a pixel-aligned video representation that encodes multi-modal cues from the 3D reconstruction, going beyond simple RGB. It consists of five channels:

1. Color (C): Standard rendered RGB.
2. Opacity (A): Alpha map from the splatting process.
3. Depth (D): Expected depth rendered via weighted averaging.
4. Normals (N): Surface normals estimated from camera-space positions.
5. Uncertainty (U): Inverse projected 2D covariance, encoding local geometric uncertainty and anisotropy (identifying "noisy" or "flat" regions).

B. Geometry-Informed Video Generator

The model is built upon a Flow-based Video Generation architecture (specifically adapting the Wan video generator).

• Architecture: It utilizes a DiT (Diffusion Transformer) backbone.
• Geometry Adapter (GA): A novel module that injects the encoded GP-Buffer features into the DiT backbone. The GA uses self-attention to process geometric features and cross-attention to fuse them with the video latent stream. This allows the model to be "geometry-aware," using depth and normals to guide the generation of clean, artifact-free frames.
• Training Objective: The model is trained using Flow Matching, predicting the velocity field to transform noisy latents into clean, refined video frames.

C. Artifact Simulation Pipeline

To ensure the model generalizes across different reconstruction sources, the authors created a comprehensive Artifact Simulation pipeline to generate a dataset of 75,000+ paired videos (clean vs. corrupted).

• Diverse Degradations: Simulates sparse-view sampling, underfitting (fewer optimization steps), and diverse initialization strategies (SfM, random points, dense maps).
• Cross-Paradigm Training: Includes artifacts from both optimization-based (e.g., Splatfacto) and feed-forward (e.g., DepthSplat, MVSplat) pipelines, ensuring the model learns to fix a wide variety of failure modes.

D. 3D Reconstruction Updating

The refined novel-view frames are not just for visualization; they are used to update the underlying 3DGS representation. The system generates smooth camera trajectories, renders GP-Buffers, refines them with GaussFusion, and then optimizes the 3D Gaussians using the new high-quality frames as supervision, improving both texture fidelity and geometric consistency.

E. Efficiency Strategy

To enable real-time applications, the authors employ a two-stage finetuning strategy:

1. Distillation: Converts a multi-step teacher model into a compact 4-step student model using Distribution Matching Distillation (DMD).
2. Adapter Finetuning: Freezes the distilled backbone and only finetunes the Geometry Adapter layers.
   This results in a model capable of 16 FPS inference on a single GPU while maintaining high quality.

3. Key Contributions

1. Geometry-Informed Video Generation: A novel approach that conditions video generation on a multi-modal GP-Buffer (Color, Depth, Normals, Opacity, Covariance), significantly outperforming RGB-only baselines in artifact removal.
2. Generalizable Refinement: The first method to successfully generalize across both optimization-based and feed-forward 3DGS reconstruction paradigms, addressing the "domain gap" in existing refinement tools.
3. Comprehensive Artifact Simulation: A robust data generation pipeline that synthesizes diverse degradation patterns, enabling the model to handle real-world "in-the-wild" scenarios.
4. Real-Time Performance: An efficient 4-step variant that achieves state-of-the-art quality at 16 FPS, enabling interactive 3D applications.

4. Experimental Results

The method was evaluated on DL3DV and RE10K benchmarks against state-of-the-art baselines (Difix3D+, GenFusion, ExploreGS, MVSplat360).

• Quantitative Performance:
  • Novel View Synthesis: Achieved 22.55 PSNR and 0.832 SSIM on DL3DV, outperforming all baselines.
  • Feed-Forward Refinement: Consistently improved feed-forward models (DepthSplat, MVSplat) where other methods failed or degraded performance.
  • 3D Reconstruction: Improved the underlying 3D geometry, resulting in cleaner reconstructions with higher PSNR and SSIM compared to methods that only refine individual frames.
• Qualitative Results:
  • Effectively removed floaters, ghosting, and blur.
  • Successfully inpainted missing regions and corrected geometric inconsistencies (e.g., spiky primitives).
  • Demonstrated superior temporal coherence compared to frame-by-frame diffusion methods.
• Ablation Studies:
  • Confirmed that including geometric modalities (Depth, Normals, Covariance) in the GP-Buffer significantly boosts performance over RGB-only inputs.
  • Showed that the Geometry Adapter architecture converges faster and yields better results than simple latent concatenation.

5. Significance

GaussFusion represents a significant shift in 3D reconstruction refinement. By moving from RGB-only conditioning to geometry-aware video generation, it solves the ambiguity problem inherent in correcting 3D artifacts. Its ability to generalize across different reconstruction pipelines (optimization vs. feed-forward) makes it a versatile tool for real-world applications where data sources vary. Furthermore, the achievement of real-time performance (16 FPS) bridges the gap between high-fidelity generative models and interactive 3D systems, paving the way for dynamic, high-quality 3D content creation in VR, AR, and autonomous driving.