🤖 AI

GA-GS: Generation-Assisted Gaussian Splatting for Static Scene Reconstruction

GA-GS is a novel method for static 3D scene reconstruction from monocular videos with dynamic objects that leverages a diffusion model to inpaint occluded regions and introduces a learnable authenticity scalar to balance supervision between real and generated data, achieving state-of-the-art performance on a newly constructed Trajectory-Match dataset.

Original authors: Yedong Shen, Shiqi Zhang, Sha Zhang, Yifan Duan, Xinran Zhang, Wenhao Yu, Lu Zhang, Jiajun Deng, Yanyong Zhang

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yedong Shen, Shiqi Zhang, Sha Zhang, Yifan Duan, Xinran Zhang, Wenhao Yu, Lu Zhang, Jiajun Deng, Yanyong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a perfect, crystal-clear photo of a beautiful park. But there's a problem: every time you try to snap a picture, a flock of birds, a running dog, and a group of kids playing soccer zoom right in front of the lens.

If you just delete the birds and the kids from your photos, you're left with giant, ugly holes in the picture where they used to be. You know the park is there, but you can't see it.

This is exactly the problem computer scientists face when trying to build 3D models of the real world from video. Real life is full of moving things (people, cars, animals) that block the view of the static background (buildings, trees, walls).

This paper introduces a new method called GA-GS (Generation-Assisted Gaussian Splatting) to solve this. Here is how it works, explained with simple analogies:

1. The Old Way: "The Eraser"

Previous methods were like using a digital eraser. They would look at the video, find the moving people, and simply delete them.

The Problem: If a person stood in front of a building for a long time, deleting them leaves a giant blank spot. The computer has no idea what the building looks like behind the person. It's like trying to finish a puzzle but throwing away half the pieces because a cat walked on them.

2. The New Way: "The Creative Artist"

The authors' new method, GA-GS, is smarter. Instead of just erasing the moving objects, it asks an AI artist (a Diffusion Model) to imagine what should be behind them.

The Analogy: Imagine you are looking at a painting with a smudge on it. Instead of just scraping the smudge off, you ask a master painter to look at the surrounding colors and brushstrokes and paint over the smudge to guess what the original picture looked like.
The Result: Now, instead of a blank hole, you have a "guessed" version of the background. It might not be 100% perfect, but it's much better than nothing.

3. The "Trust Meter" (The Secret Sauce)

Here is the tricky part: The AI artist's guess is good, but it's not real. If we treat the guess exactly the same as the real photo, the final 3D model might look weird or hallucinated.

So, GA-GS introduces a "Trust Meter" (called an Authenticity Scalar).

How it works: Every tiny 3D dot (called a "Gaussian") in the scene gets a little tag.
- If the dot comes from a real part of the video (where no one was blocking the view), the tag says: "TRUST ME, I'M REAL!" (High confidence).
- If the dot comes from the AI's guess (where someone was blocking the view), the tag says: "I'M JUST A GUESS, TAKE IT EASY." (Low confidence).
The Magic: When the computer builds the final 3D scene, it listens to the "Real" dots more loudly and treats the "Guess" dots as a helpful suggestion rather than a fact. This balances the need for a complete picture with the need for accuracy.

4. The "Robot Camera" Test

To prove this works, the researchers needed a way to check if their "guesses" were actually correct. But how do you check a guess if you don't have the real answer?

The Solution: They built a special dataset using a robot.
1. The robot drove through a scene with people and cars moving around (the messy video).
2. Then, the robot drove through the exact same path again, but this time, they made sure the scene was empty and quiet (the "Ground Truth").
Why it matters: This gave them a "secret answer key." They could compare their AI's reconstruction against the real, empty scene to see exactly how well they filled in the holes.

Summary

In short, GA-GS is like a super-smart photo editor that:

Removes moving people from a video.
Uses AI art to fill in the missing background holes.
Keeps a "Trust Meter" to make sure the AI's guesses don't ruin the reality.
Proves it works by testing it on a robot that took "before and after" photos of the same scene.

The result is a clean, high-quality 3D model of a world, even if that world was full of people running around while the camera was filming. This is huge for things like self-driving cars (which need to know where the road is, even if a pedestrian blocks the view) and Virtual Reality.

1. Problem Definition

The paper addresses the challenge of reconstructing a clean, temporally consistent static 3D scene from a monocular video captured in a dynamic environment containing moving objects (e.g., pedestrians, vehicles).

Core Challenge: Existing 3D Gaussian Splatting (3DGS) methods struggle when dynamic objects occlude large portions of the background.
- Current Limitation: Standard approaches typically remove dynamic objects and rely solely on the remaining background pixels for supervision. This leads to incomplete reconstructions in heavily occluded regions because valid supervision data is discarded.
- The Dilemma: Simply removing dynamic objects sacrifices supervision data, while using generative models to fill gaps introduces "hallucinated" content that may be unreliable and inconsistent with the true scene geometry.
Goal: To recover a high-fidelity static background that accurately reconstructs regions permanently occluded by dynamic objects, balancing the reliability of real observations with the completeness of generative priors.

2. Methodology: GA-GS

The proposed GA-GS framework integrates real observations with generative priors through a three-stage pipeline:

A. Initialization and Masking

Geometric Initialization: Instead of relying on traditional Structure-from-Motion (SfM) pipelines like COLMAP, the method uses VGGT (a 3D foundation model) to estimate camera poses and per-pixel depth maps directly from the video.
Structure-Aware Sampling: To initialize Gaussian primitives, the method filters the dense point cloud from VGGT. It prioritizes points with high visual and geometric saliency (using image and depth gradients) and high depth confidence, creating a compact but informative initial set.
Dynamic Masking: A motion-aware module combining Flow-SAM (Segment Anything Model + Optical Flow) is used to generate precise, temporally consistent binary masks identifying dynamic regions in every frame.

B. Pseudo-Ground Truth Generation

To provide supervision for occluded areas, the method employs a diffusion-based video inpainting model (DiffuEraser):

The dynamic masks guide the diffusion model to "inpaint" (fill in) the occluded regions in the video frames.
The output is a "completed" static image sequence where dynamic objects are replaced by plausible background content.
This creates a pseudo-ground-truth supervision signal ( $I'$ ) that combines original unoccluded pixels with generated pixels for occluded areas.

C. Authenticity-Aware Opacity Blending

This is the core innovation to handle the reliability gap between real and generated data:

Learnable Authenticity Scalar ( $\theta$ ): Each Gaussian primitive is assigned a learnable scalar parameter $\theta \in [0, 1]$ $θ \in [0, 1]$ , representing the likelihood that the primitive originates from a real observation rather than a generated one.
- Initialization: Primitives initialized from original image pixels start with $\theta \approx 0.9$ ; those from inpainted regions start with $\theta \approx 0.1$ .
Opacity Modulation: During rendering, the effective opacity ( $\alpha'$ $α^{'}$ ) of a Gaussian is modulated by its $\theta$ $θ$ value ( $\alpha' = \theta \cdot \alpha$ $α^{'} = θ \cdot α$ ).
- This ensures that primitives from real regions dominate the rendering and loss calculation, while those from generated regions act as auxiliary support.
Loss Function: The training loss is a weighted combination of L1 and SSIM. A composite weight map assigns higher weights to real pixels and lower weights to generated pixels, ensuring the model prioritizes fidelity to the real scene while utilizing generated data to fill gaps.

3. Key Contributions

Novel Framework (GA-GS): A 3DGS-based pipeline that jointly leverages real observations and generative priors to reconstruct static scenes under persistent dynamic occlusions.
Authenticity-Aware Rendering: Introduction of a learnable authenticity scalar ( $\theta$ ) per Gaussian primitive. This mechanism dynamically balances the contribution of real vs. generated data, allowing the model to utilize generative priors without compromising reconstruction fidelity.
New Dataset (Trajectory-Match): Since existing datasets (like DAVIS) lack ground-truth static backgrounds for dynamic scenes, the authors constructed a new dataset using a robot-mounted camera. They captured paired sequences (one with dynamic objects, one with a clean static background) along identical trajectories, enabling quantitative evaluation of occluded region recovery.
State-of-the-Art Performance: The method achieves superior results on both the DAVIS benchmark and the new Trajectory-Match dataset, particularly in scenarios with large-scale, long-term occlusions.

4. Experimental Results

Datasets: Evaluated on DAVIS (8 sequences) and the custom Trajectory-Match dataset (Indoor, Outdoor, Street scenes).
Metrics: PSNR, SSIM, and LPIPS.
Performance:
- DAVIS: GA-GS outperforms baselines (WildGaussians, Robust3DGS, DAS3R) in PSNR for background reconstruction. Qualitative results show significantly better recovery in occluded regions (e.g., "car-shadow," "camel" scenes).
- Trajectory-Match: GA-GS achieves the highest scores across all metrics. Crucially, in occluded regions, it shows a massive improvement in PSNR (e.g., ~24.80 vs. ~16.93 for DAS3R), proving its ability to reconstruct hidden geometry.
Ablation Studies:
- Removing the Authenticity Parameter ( $\theta$ ) causes a significant drop in PSNR for occluded regions, confirming the necessity of weighting real vs. generated data differently.
- Removing the Diffusion Generation module also degrades performance, highlighting the value of generative priors for missing data.
- Pose Source: The method remains robust even when using VGGT for initialization instead of the more accurate COLMAP, demonstrating its practicality for real-world applications.

5. Significance

Solving the Occlusion Gap: GA-GS effectively solves the fundamental trade-off between supervision completeness and reliability in dynamic scene reconstruction. It allows 3D reconstruction to proceed even when large parts of the scene are permanently hidden.
Practical Application: By enabling the recovery of static backgrounds from monocular videos with moving objects, this technology is critical for autonomous driving (mapping static environments despite traffic), robotics, and virtual reality content creation.
Dataset Contribution: The introduction of the Trajectory-Match dataset fills a critical gap in the community, providing the first standard for quantitatively evaluating static reconstruction quality in the presence of dynamic occlusions.