💻 computer science

GaussianGrow: Geometry-aware Gaussian Growing from 3D Point Clouds with Text Guidance

GaussianGrow is a novel method that generates high-quality 3D Gaussians from 3D point clouds by iteratively growing them with text guidance, utilizing multi-view diffusion models for appearance synthesis and inpainting to ensure geometric accuracy and complete coverage of unobserved regions.

Original authors: Weiqi Zhang, Junsheng Zhou, Haotian Geng, Kanle Shi, Shenkun Xu, Yi Fang, Yu-Shen Liu

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weiqi Zhang, Junsheng Zhou, Haotian Geng, Kanle Shi, Shenkun Xu, Yi Fang, Yu-Shen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a beautiful, detailed 3D statue of a "fox wearing a work jacket." In the past, doing this with computers was like trying to sculpt a masterpiece while blindfolded, or worse, trying to build a house by guessing where the bricks should go without a blueprint.

GaussianGrow is a new method that changes the game. Instead of guessing, it uses a 3D point cloud as a blueprint. Think of a point cloud as a "digital cloud of dust" that perfectly outlines the shape of the fox, but has no color or texture yet. It's just the skeleton.

Here is how GaussianGrow works, broken down into simple steps with some fun analogies:

1. The "Growth" Concept

Traditional methods try to build the whole statue from scratch, often getting the shape wrong. GaussianGrow does something different: it grows the statue.

The Analogy: Imagine you have a wireframe of a fox (the point cloud). GaussianGrow is like a magical gardener who plants "3D pixels" (called Gaussians) directly onto that wireframe. Because the wireframe is already there, the pixels know exactly where to sit. They don't have to guess the shape; they just fill in the color and detail. This ensures the fox looks exactly like a fox, not a blob.

2. The "Artist" and the "Guide"

The system uses a text prompt (e.g., "fox in a workwear style") to tell the computer what the fox should look like.

The Analogy: Think of the text prompt as the Art Director. The computer is the Painter.
Usually, painters struggle to make sure the front, back, and sides of a painting all match up. If the painter looks at the front, they might paint a red jacket, but when they turn to the back, they accidentally paint a blue one.
GaussianGrow uses a special "Multi-View Diffusion Model" (a super-smart AI artist) that looks at the fox from all angles at once to ensure the jacket is red everywhere.

3. Solving the "Seams" Problem

When you look at an object from different angles, the edges where those views meet often look messy or glitchy (like a bad photo collage).

The Analogy: Imagine taking photos of a statue from the front, left, and right. When you try to stitch them together, the seam where the left photo meets the front photo might look jagged.
The Fix: GaussianGrow has a clever trick. It calculates exactly where the "seams" are and moves the camera to a new, perfect angle to take a "bonus photo" right there. It then uses this new photo to smooth out the glitch, making the transition seamless.

4. The "Blind Spot" Inpainting

Even with many photos, some parts of the object might be hidden (like the fox's ears if they are tucked down, or the back of its head).

The Analogy: Imagine you are trying to paint a statue, but you can't see the back of its head. You might leave it blank or paint it wrong.
The Fix: GaussianGrow plays a game of "Hide and Seek." It looks at the parts of the fox it can't see yet, calculates the perfect spot to move the camera to peek at those hidden spots, and then uses an AI "Inpainting" tool (like a magic eraser and brush) to fill in the missing details based on the text description. It keeps doing this until the whole fox is painted.

Why is this a big deal?

No More Manual Modeling: You don't need a human artist to spend hours building a 3D mesh (a digital skin) for the object. You just need a simple scan (the point cloud), which is easy to get with modern scanners or even phone apps.
Better Quality: Because the shape is guided by the real scan, the final result doesn't look wobbly or distorted. It looks crisp and real.
Text to 3D: You can type "A floral cheongsam" or "A tiger head," find a generic shape that matches, and GaussianGrow will instantly turn that generic shape into a high-quality, textured 3D object.

In summary: GaussianGrow is like a smart, automated construction crew. They take a rough outline (the point cloud), use a text description as their blueprint, and "grow" a high-definition, perfectly textured 3D object onto that outline, fixing any mistakes or blind spots along the way. It makes creating 3D worlds faster, easier, and much more accurate.

1. Problem Statement

While 3D Gaussian Splatting (3DGS) has revolutionized 3D rendering with high fidelity and real-time performance, generating 3D Gaussians from scratch remains a significant challenge.

Lack of Geometric Priors: Existing generative methods often struggle because they lack reliable geometric constraints, leading to poor generation quality or the "Janus problem" (multi-faced artifacts).
Limitations of Current Approaches:
- Point Map Prediction: Methods that predict point maps as geometric references often produce unreliable geometries, resulting in suboptimal Gaussian primitives.
- Mesh-based Texturing: Approaches relying on 3D meshes require extensive manual modeling and UV unwrapping, which introduces texture distortion and overlapping issues.
- Point Cloud Neglect: Few methods effectively leverage easily accessible 3D point clouds (from LiDAR or scanning) for appearance generation.

Core Goal: To develop a framework that generates high-quality, text-guided 3D Gaussians by "growing" them directly from 3D point clouds, ensuring geometric accuracy without relying on complex mesh reconstruction.

2. Methodology: GaussianGrow

The proposed framework, GaussianGrow, treats 3D Gaussian generation as a "growing" process from a point cloud prior. It consists of three main stages:

A. Initialization and Geometric Priors

Point-to-Gaussian Initialization: Each Gaussian center is initialized at the corresponding point position in the input point cloud $P$ .
Unsigned Distance Field (UDF): Instead of Signed Distance Fields (SDFs) which require watertight surfaces, the method learns an UDF from the point cloud using CAP-UDF. This handles open and complex topologies better.
Geometric Maps: The system computes depth, normal, and position maps from the UDF. These maps serve as robust geometric cues for guiding the diffusion models.

B. Stage 1: Appearance Generation & Overlap Refinement

Multi-View Diffusion: The system uses a multi-view diffusion model (Hunyuan3D-Paint) to synthesize consistent appearances from the input point cloud and text prompts.
Overlap Detection & Pose Optimization:
- Standard cardinal views (6 views) often create artifacts in overlapping regions due to fusion inconsistencies.
- GaussianGrow detects these overlap regions by intersecting visible Gaussian sets from different viewpoints.
- It then optimizes additional camera poses specifically to observe these overlap regions. The optimization maximizes the alignment between camera rays and the geometric normals of the Gaussians in the overlap, ensuring consistent appearance generation.
Two-Phase Optimization:
1. Optimize Gaussians for the 6 cardinal views.
2. Refine Gaussians in the overlap regions using the newly optimized camera poses.

C. Stage 2: Iterative Gaussian Inpainting

Problem: Some regions of the point cloud remain "unseen" or occluded even after multi-view synthesis.
Visibility-Based Pose Optimization: The system iteratively predicts a new camera pose that observes the largest remaining unseen region of the point cloud. This is achieved by minimizing an occlusion loss that quantifies how many unoptimized Gaussians are hidden by optimized ones.
Image Inpainting: From this new pose, the system renders an occluded view and uses a pretrained 2D diffusion model (Stable Diffusion + ControlNet) to inpaint the missing regions based on the text prompt and depth maps.
Spatial Inpainting: As a post-processing step, properties are propagated from optimized neighbors to nearby unoptimized Gaussians to handle noise and uneven density in raw scans.
Iteration: This process repeats (typically 6 iterations) until all Gaussians are generated and the object is fully covered.

3. Key Contributions

GaussianGrow Framework: A novel approach that generates 3D Gaussians by "growing" them from accessible 3D point clouds, naturally enforcing geometric accuracy without mesh reconstruction.
Overlap-Aware Refinement: A specific refinement operation that identifies overlap regions and optimizes camera poses to generate consistent appearances, reducing fusion artifacts.
Iterative Inpainting Scheme: A novel method to detect and complete invisible regions by iteratively optimizing camera poses to maximize visibility of unseen areas and using 2D diffusion for inpainting.
Text-to-3D Pipeline: Demonstrated capability to perform text-to-3D generation by retrieving point clouds via multi-modal models (Uni3D) and applying GaussianGrow.

4. Experimental Results

The method was evaluated on both synthetic (Objaverse) and real-scanned (DeepFashion3D) point clouds.

Text-Guided Visual Synthesis (Point-to-Gaussian):
- Outperformed state-of-the-art methods (TexTure, Text2Tex, SyncMVD, Paint3D, GAP) on the Objaverse dataset.
- Metrics: Achieved the lowest FID (36.07 vs. ~40+ for others) and KID, and highest CLIP scores.
- Qualitative: Preserved fine details better than mesh-based methods, avoiding UV distortion.
Text-to-3D Generation:
- When combined with retrieval (Uni3D), it achieved the best performance on the T3Bench benchmark across CLIP Similarity, R-Precision, and ImageReward.
- Even with generative scaffolds (LGM), it significantly outperformed direct generation methods like DiffSplat.
Ablation Studies:
- Removing Overlap Processing or Inpainting significantly degraded performance (FID increased from 36.07 to ~40.5).
- Using 10 views (6 cardinal + 4 optimized overlap views) yielded optimal results; adding more views provided diminishing returns.

5. Significance

Bridging Geometry and Appearance: GaussianGrow successfully bridges the gap between raw point cloud geometry and 3DGS appearance, solving the issue of unreliable geometry estimation in purely generative models.
Efficiency and Accessibility: By utilizing point clouds (which are easier to acquire via scanning than meshes), the method lowers the barrier for high-quality 3D content creation.
Robustness: The iterative inpainting strategy ensures complete coverage of complex objects, addressing a major limitation of fixed-viewpoint generation methods.
Practical Application: The framework enables flexible, text-driven 3D asset creation for applications in film, gaming, and AR/VR, offering a robust alternative to traditional mesh-based pipelines.