G4Splat: Geometry-Guided Gaussian Splatting with Generative Prior

G4Splat is a novel 3D reconstruction method that leverages accurate metric-scale geometry derived from planar structures to guide a generative prior, effectively resolving multi-view inconsistencies and enabling high-quality scene completion in both observed and unobserved regions.

The Gist

The Big Picture: Fixing the "Blurry Puzzle" Problem

Imagine you are trying to build a 3D model of a room, but you only have a few photos of it taken from different angles.

• The Old Way: If you try to guess what the rest of the room looks like (the parts you didn't photograph), you might end up with a model that looks okay up close but turns into a floating, blurry mess in the empty spaces. It's like trying to complete a jigsaw puzzle by guessing the missing pieces without looking at the picture on the box.
• The New Way (G4SPLAT): This paper introduces a new method that acts like a super-smart architect. It doesn't just guess; it uses the "rules of the room" (geometry) to know exactly where walls, floors, and tables should be, even in the dark, unphotographed corners. Then, it uses a "creative artist" (AI) to fill in the colors and textures, but only after the architect has built a solid frame.

---

The Two Main Problems They Solved

The authors identified two reasons why previous methods failed:

1. The "Floating Ghosts" Problem: Without a solid geometric foundation, AI models tend to create "ghosts"—floating blobs of color that don't belong to anything. It's like painting a cloud that has no sky to hold it up.
2. The "Confused Artist" Problem: When AI tries to imagine what's behind a wall, it often gets confused. One angle might say the wall is red, and another says it's blue. This leads to a messy, inconsistent result.

The G4SPLAT Solution: The "Architect + Artist" Team

The paper proposes a two-step process that combines Geometry (the structure) with Generative AI (the creativity).

Step 1: The Architect (Geometry Guidance)

• The Metaphor: Imagine the room is built mostly of flat surfaces: floors, walls, and tables. These are like giant, flat sheets of paper.
• How it works: The method assumes that most man-made objects are flat (like a Manhattan city grid). It finds these flat "sheets" in the photos you do have.
• The Magic: Once it finds a flat wall in your photo, it can mathematically "slide" that wall across the whole room, even into the parts you didn't photograph. This gives the computer a perfectly accurate map of where the walls and floors are, down to the exact distance.
• Why it matters: This stops the "floating ghosts." The AI knows exactly where the floor is, so it won't accidentally paint a chair floating in mid-air.

Step 2: The Artist (Generative Prior)

• The Metaphor: Now that the Architect has built a solid skeleton of the room, we need an Artist to paint the details.
• How it works: They use a powerful AI (a video diffusion model) to "hallucinate" or imagine the missing parts of the room.
• The Twist: Usually, artists might paint wildly different things from different angles. But because G4SPLAT has the Architect's map, it tells the Artist: "Hey, you are painting the back of this specific table. Make sure the color matches the front, and don't make the table float."
• The Result: The AI fills in the missing areas with realistic textures, but because it's guided by the solid geometry, the result is consistent and sharp.

Why This is a Game Changer

1. It Works with Just One Photo: You don't need a video or many photos. Even if you only have a single picture of a room, G4SPLAT can use its "flat surface" logic to guess the rest of the room's shape and then fill in the details.
2. It Handles "Unseen" Areas: If you take a photo of a room but the camera is blocked by a sofa, G4SPLAT can still figure out what's behind the sofa because it understands the geometry of the room.
3. No More "Floaters": The final 3D model is clean. No more floating clouds of pixels. It looks like a real, solid object.

A Simple Analogy: Building a House

• Old Methods: You try to build a house by throwing bricks into the air and hoping they stick together. Sometimes they do, but often you get a pile of rubble or a house that looks like it's melting.
• G4SPLAT:
  1. The Blueprint (Geometry): First, you lay down a perfect, rigid steel frame based on the rules of physics and the few photos you have. You know exactly where the walls and roof go.
  2. The Painting (Generative AI): Then, you hire a painter to fill in the drywall and paint the rooms. Because the steel frame is already there, the painter knows exactly where the corners are and doesn't accidentally paint a window on the roof.

The Bottom Line

G4SPLAT is a new way to turn 2D photos into 3D worlds. It fixes the biggest problem in current AI 3D reconstruction: making sure the "imagined" parts look real and fit perfectly with the "seen" parts. By using the simple logic of flat surfaces (like walls and floors) to guide the complex AI, it creates 3D scenes that are accurate, consistent, and ready for use in robotics, virtual reality, and video games.

Technical Summary

1. Problem Statement

Recent advances in 3D Gaussian Splatting (3DGS) have enabled efficient, photo-realistic novel view synthesis. However, performance degrades significantly in sparse-view settings (few input images) or unposed videos due to insufficient geometric and photometric supervision.

Existing methods attempting to solve this face two critical limitations:

1. Lack of Reliable Geometric Supervision: Methods relying on monocular depth estimators suffer from scale ambiguity, leading to poor reconstruction quality even in observed regions. This undermines the geometric basis required to inpaint unobserved areas.
2. Multi-View Inconsistencies: Approaches leveraging generative priors (diffusion models) to hallucinate missing content often fail to maintain consistency across views. This results in severe shape–appearance ambiguities, floating artifacts (floaters), and degraded scene geometry.

The authors argue that accurate geometry guidance is the fundamental prerequisite for effectively utilizing generative models to enhance 3D reconstruction.

---

2. Methodology: G4SPLAT

G4SPLAT integrates accurate geometry guidance with generative priors through a two-stage pipeline: Plane-Aware Geometry Modeling and a Geometry-Guided Generative Pipeline.

A. Plane-Aware Geometry Modeling

To overcome scale ambiguity and provide reliable depth in both observed and unobserved regions, the method leverages the prevalence of planar structures in man-made environments (Manhattan World assumption).

• 2D Plane Extraction: Instance masks (from SAM) are combined with normal maps to extract 2D plane masks in individual views.
• Global 3D Plane Estimation: Local 2D masks are merged into globally consistent 3D planes using the 3D point cloud. Planes are fitted via RANSAC on high-confidence points (observed in $\ge$ 2 views).
• Plane-Aware Depth Map:
  • For planar regions, depth is calculated analytically by intersecting camera rays with the fitted global 3D planes, ensuring metric-scale accuracy.
  • For non-planar regions, a pre-trained monocular depth estimator provides relative depth, which is then linearly aligned to the absolute scale of the planar regions.
  • Result: A complete, scale-accurate depth map that extends reliable geometry to non-overlapping and unobserved areas.

B. Geometry-Guided Generative Pipeline

The method incorporates the derived geometry into the generative refinement loop to mitigate shape–appearance ambiguities.

1. Geometry-Guided Visibility: Instead of using noisy alpha maps, a Visibility Grid is constructed from the scale-accurate depth maps. This grid determines which voxels are visible in training views, providing precise visibility masks for inpainting.
2. Plane-Aware Novel View Selection: To ensure the generative model has sufficient context, novel viewpoints are selected to maximize coverage of global 3D planes. The camera is positioned to align with plane normals and minimize distance to the plane, ensuring complete structural observation.
3. Geometry-Guided Inpainting:
   • A video diffusion model (e.g., See3D) inpaints occluded regions in the novel views.
   • Consistency Mechanism: To prevent multi-view conflicts, color supervision is modulated based on geometry. For planar regions, the view with the most complete observation of that plane is prioritized. For non-planar regions, the first view where the region becomes visible is used. This reduces cross-view conflicts during training.

C. Training Strategy

The training consists of an Initialization Stage (using MAtCha with plane-aware depth) followed by iterative Generative Training Loops. In each loop, novel views are generated, inpainted, and merged into the training set, progressively refining the Gaussian field and correcting geometric misalignments.

---

3. Key Contributions

1. Scale-Accurate Geometric Constraints: A novel method to derive metric-scale depth by leveraging global 3D plane representations, effectively propagating depth from overlapping to non-overlapping and unobserved regions.
2. Geometry-Guided Generative Pipeline: A framework that integrates geometry throughout the generative process (visibility estimation, view selection, and inpainting supervision) to resolve shape–appearance ambiguities and ensure multi-view consistency.
3. Robust Generalization: The method supports diverse inputs, including single-view images and unposed videos, performing effectively in both indoor (Manhattan) and outdoor (non-Manhattan) scenarios.

---

4. Experimental Results

The method was evaluated on Replica, ScanNet++, DeepBlending, and Mip-NeRF 360.

• Quantitative Performance: G4SPLAT consistently outperforms state-of-the-art baselines (including 3DGS, 2DGS, MAtCha, GenFusion, Difix3D+, and GuidedVD) across all metrics:
  • Geometry: Significant improvements in Chamfer Distance (CD), F-Score, and Normal Consistency (NC). For example, on Replica with 5 views, it achieved a CD of 6.61 vs. 10.12 for MAtCha.
  • Rendering: Higher PSNR, SSIM, and lower LPIPS, indicating superior visual fidelity.
• Qualitative Performance:
  • Unobserved Regions: Successfully reconstructs missing areas with high geometric accuracy and smooth textures, whereas baselines often produce blurry or floating artifacts.
  • Observed Regions: Eliminates "floaters" (spurious Gaussians) common in sparse-view reconstructions.
  • Single-View & Unposed Video: Demonstrates strong capability in extreme sparse settings (1 input view) and casual video inputs, outperforming methods that fail without dense supervision.
• Efficiency: The method achieves high-quality reconstruction with a runtime comparable to other generative approaches, with an accelerated variant (Ours DS) reducing training time significantly.

---

5. Significance

G4SPLAT addresses a fundamental bottleneck in sparse-view 3D reconstruction: the trade-off between generative freedom and geometric fidelity. By establishing accurate geometry as the foundation for generative inpainting, the paper demonstrates that:

• Generative models can be effectively constrained to produce physically consistent 3D scenes rather than hallucinating inconsistent content.
• Planar priors are a powerful tool for extending depth estimation beyond the convex hull of input views.
• The approach has high practical value for Embodied AI, Robotics, and AR/VR, where reconstruction from limited, unposed, or single-view data is a common requirement.

The work sets a new state-of-the-art for sparse-view 3DGS, proving that integrating structural priors with generative models yields superior results in both geometry and appearance.