💻 computer science

GaMO: Geometry-aware Multi-view Diffusion Outpainting for Sparse-View 3D Reconstruction

GaMO is a zero-shot, geometry-aware multi-view diffusion framework that improves sparse-view 3D reconstruction by expanding the field of view from existing camera poses to ensure geometric consistency and broader scene coverage, all while significantly reducing computational runtime compared to existing methods.

Original authors: Yi-Chuan Huang, Hao-Jen Chien, Chin-Yang Lin, Ying-Huan Chen, Yu-Lun Liu

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yi-Chuan Huang, Hao-Jen Chien, Chin-Yang Lin, Ying-Huan Chen, Yu-Lun Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a room, but you only have three blurry photos of it taken from different corners. You try to guess what the rest of the room looks like, but your guess is full of holes, ghostly double-images, and weird distortions. This is the problem of Sparse-View 3D Reconstruction.

The paper introduces a new method called GaMO (Geometry-aware Multi-view Outpainter) that solves this problem by changing the strategy entirely. Instead of trying to "hallucinate" new angles of the room, GaMO simply expands the edges of the photos you already have.

Here is the breakdown using simple analogies:

1. The Problem: The "Jigsaw Puzzle" with Missing Pieces

Traditional methods try to solve this puzzle by inventing entirely new photos from angles you never saw.

The Flaw: Imagine trying to guess what's behind a wall by inventing a new picture of the wall. You might guess a door is there, but the real wall is solid. When you try to build the 3D model, these "invented" guesses clash with reality, creating ghosts (double images) and holes (missing parts).
The Old Way: It's like trying to complete a jigsaw puzzle by drawing new pieces that might not fit the picture.

2. The GaMO Solution: "Zooming Out" Instead of "Teleporting"

GaMO takes a different approach. Instead of teleporting to a new spot to take a photo, it acts like a smart photo editor that "outpaints" your existing photos.

The Analogy: Imagine you have a photo of a living room, but the camera was zoomed in tight on the sofa.
- Old Method: Tries to guess what the kitchen looks like from a completely different angle.
- GaMO: Takes your tight photo and gently stretches the edges to reveal the walls, the floor, and the kitchen that are already connected to your sofa. It fills in the missing borders of the same view.

3. How It Works: The Three-Step "Construction Crew"

Step 1: The Rough Sketch (Coarse Initialization)
First, GaMO takes your few photos and builds a very rough, low-quality 3D skeleton of the room. It's like a construction crew putting up wooden scaffolding. It's not pretty, but it tells the computer where the walls and floors roughly are.

Step 2: The Smart Expander (Geometry-Aware Outpainting)
This is the magic part. GaMO uses an AI (a diffusion model) to "outpaint" your photos.

The Trick: Usually, AI outpainting is just guessing. But GaMO uses that "rough sketch" from Step 1 as a guide.
The Metaphor: Imagine an artist painting a mural. Instead of guessing the colors, they are wearing goggles that see the 3D structure. When the AI paints the new edge of the photo, it knows, "Ah, the wall curves this way," so it paints the wall continuing smoothly, not randomly.
Result: It creates wide-angle photos that perfectly match the geometry of your original photos. No ghosts, no holes.

Step 3: The Polish (Refinement)
Now, GaMO takes these new, wide-angle photos and feeds them back into the 3D builder. Because the new photos are geometrically perfect and consistent with the old ones, the 3D model snaps into place beautifully. The "holes" are filled, and the "ghosts" disappear.

4. Why Is It Better?

Consistency: Because it expands what you already know rather than guessing what you don't know, the 3D model stays consistent. It's like extending a road you are already driving on, rather than trying to build a bridge to a random island.
Speed: Other methods that try to generate new angles take hours (like 3+ hours). GaMO does it in under 10 minutes. It's the difference between hand-crafting a sculpture and using a 3D printer.
Quality: It produces sharper images with fewer artifacts (no weird floating blobs or double walls).

Summary

Think of GaMO as a smart photo frame extender.
If you have a small photo of a room, other methods try to imagine a whole new room next to it (and often get it wrong). GaMO simply widens the frame of your existing photo to show more of the same room, using a rough 3D map to ensure the new edges line up perfectly with the old ones. This creates a complete, high-quality 3D world from just a few snapshots, fast and accurately.

1. Problem Statement

Reconstructing high-fidelity 3D scenes from sparse input views (e.g., 3, 6, or 9 images) is a fundamental challenge in computer vision. While dense multi-view methods (like standard 3D Gaussian Splatting) perform well with abundant data, sparse inputs lead to:

Broken Geometry: Missing regions and holes in unobserved areas.
Visual Artifacts: Ghosting, blurring, and inconsistent textures.
Geometric Inconsistency: Misalignment between generated novel views and the original scene structure.

Existing diffusion-based approaches attempt to solve this by generating novel views (new camera poses) to augment training data. However, the authors identify three critical limitations in these state-of-the-art methods:

Inadequate Coverage: They focus on interpolating between existing views rather than extending beyond the periphery, leaving holes at the scene boundaries.
Geometric Inconsistency: Generating entirely new viewpoints often introduces hallucinations that conflict with the sparse input geometry, leading to misalignment.
Computational Inefficiency: These pipelines require complex trajectory planning and are computationally expensive (often taking hours).

2. Methodology: GaMO

The authors propose GaMO (Geometry-aware Multi-view Outpainting), a framework that reformulates sparse-view reconstruction as an outpainting problem rather than a novel view generation problem. Instead of creating new camera poses, GaMO expands the Field of View (FOV) of existing input images to cover unobserved regions while preserving geometric consistency.

The pipeline consists of three stages:

A. Coarse 3D Initialization

Goal: Establish a geometric prior to guide the diffusion process.
Process:
1. Use DUSt3R to generate an initial point cloud from sparse inputs.
2. Train a coarse 3D Gaussian Splatting (3DGS) model.
3. Render Priors:
  - Opacity Mask: Render an opacity map with an enlarged FOV. Regions with low opacity indicate areas requiring outpainting.
  - Coarse Rendering: Render a low-fidelity color image with the enlarged FOV to provide appearance and structural cues.

B. Geometry-aware Multi-view Outpainter

This is the core innovation, utilizing a pre-trained multi-view diffusion model (MVGenMaster) in a zero-shot manner (no fine-tuning).

Multi-view Conditioning:
- Plücker Ray Embeddings: Encode camera rays (origin and direction) for both input views and the target expanded view to ensure geometric alignment.
- Augmented Canonical Coordinate Maps (CCM) & RGB: Input images are warped to align with the expanded FOV. The center retains original data, while the periphery contains warped geometric structures to guide the generation.
Mask Latent Blending:
- During the denoising process, the model blends the generated latent with the coarse geometry prior (encoded from the coarse render).
- Iterative Mask Scheduling (IMS): The blending mask is progressively dilated and then shrunk over specific timesteps. Early stages allow the model to hallucinate peripheral content freely; later stages enforce alignment with the coarse geometry to prevent drift.
Noise Resampling: After blending, noise is re-sampled to eliminate boundary artifacts and ensure smooth transitions between the original and outpainted regions.

C. 3DGS Refinement

The generated outpainted views (with larger FOVs) are used to refine the 3DGS model.
Loss Function: Combines standard reconstruction losses ( $L_1$ , D-SSIM) for input views with a Perceptual Loss (LPIPS) for outpainted views. The perceptual loss is crucial for filling holes and maintaining texture quality in generated regions without over-regularizing.

3. Key Contributions

Paradigm Shift: Proposes Outpainting (expanding FOV of existing views) as a superior paradigm to Novel View Generation for sparse-view reconstruction. This inherently preserves geometric consistency and avoids the misalignment issues of generating new camera poses.
Geometry-Aware Outpainting: Introduces a zero-shot framework that integrates coarse 3D geometry priors (opacity masks, Plücker rays) directly into the diffusion denoising loop via Mask Latent Blending and Iterative Mask Scheduling.
Efficiency: Achieves a massive speedup over existing diffusion-based methods. The entire pipeline runs in under 10 minutes (approx. 8 mins on an RTX 4090), compared to hours for competitors.
State-of-the-Art Performance: Demonstrates superior reconstruction quality across multiple datasets (Replica, ScanNet++, Mip-NeRF 360) with 3, 6, and 9 input views, significantly reducing holes and ghosting artifacts.

4. Experimental Results

Datasets: Evaluated on Replica (indoor), ScanNet++ (indoor), and Mip-NeRF 360 (unbounded outdoor).
Metrics: PSNR, SSIM, LPIPS, and Runtime.
Quantitative Performance:
- Replica (3 views): Achieved SSIM 0.865 and LPIPS 0.117, outperforming GuidedVD-3DGS (SSIM 0.864) and significantly beating vanilla 3DGS.
- Mip-NeRF 360 (6 views): Achieved the best PSNR (16.74) and SSIM (0.393) among all compared methods.
- Efficiency: Reduced runtime from 3 hours 20 minutes (GuidedVD-3DGS) to **8 minutes** (GaMO), a roughly 25x speedup.
Qualitative Performance: Visual comparisons show GaMO effectively fills "black holes" in unobserved regions, eliminates ghosting artifacts, and maintains sharp geometric boundaries where other methods fail.

5. Significance and Impact

Practicality: The method makes high-quality 3D reconstruction feasible for scenarios with very limited data capture (e.g., casual photography, quick scans) without requiring expensive hardware or hours of processing time.
Robustness: By leveraging existing geometry to guide generation rather than hallucinating new perspectives, GaMO offers a more stable and reliable solution for sparse-view problems.
Zero-Shot Generalization: The approach works without fine-tuning the diffusion backbone, making it easily adaptable to new scenes and datasets.
Future Direction: The paper highlights that while GaMO handles peripheral expansion well, it still struggles with regions completely occluded from all input views, suggesting future work could involve generating outpainted views from drastically different angles (e.g., bird's-eye view) to resolve severe occlusions.

In summary, GaMO provides a highly efficient, geometry-consistent, and high-quality solution for sparse-view 3D reconstruction by rethinking the problem as an outpainting task guided by coarse 3D priors.