Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors

1. Problem Statement

Current 3D generation models (e.g., those conditioned on text or single images) often lack geometric controllability. While they can produce plausible 3D assets, they struggle to respect real-world 3D measurements or specific structural constraints.

• The Gap: In many real-world scenarios (e.g., robotics, AR/VR), partial point clouds are readily available from active sensors (LiDAR, depth cameras) or feed-forward predictors (like VGGT). However, existing generative pipelines treat these as mere conditions rather than hard structural constraints, failing to anchor the generation to observed geometry.
• The Challenge: Simply injecting point clouds as an additional condition is insufficient. The model needs to integrate the structural prior directly into the latent space to ensure the generated asset aligns with observed regions while plausibly completing unobserved areas.

2. Methodology: Points-to-3D

The authors propose Points-to-3D, a diffusion-based framework built upon the TRELLIS latent 3D diffusion model. The core innovation is redefining the initialization and completion process of the sparse structure latent to incorporate point cloud priors.

A. Core Architecture & Pipeline

The method operates on a Sparse Structure (SS) Latent representation, which encodes the geometric scaffold of a 3D object.

1. Input Processing:
   • Takes a point cloud prior ($P$), which can be either ground-truth sensor data or an estimate from a single image (via VGGT).
   • Voxelization & Encoding: The visible point cloud is voxelized into a binary occupancy grid ($M'$) and encoded using the TRELLIS VAE encoder ($E_s$) to produce a partial latent representation ($q_{vis}$).
2. Latent Formulation (The "Inpainting" Input):
   • Instead of starting with pure Gaussian noise, the model constructs a mixed latent input ($q_{comb}$).
   • Regions corresponding to the observed point cloud are preserved ($q_{vis}$).
   • Unobserved regions are filled with random noise ($\epsilon_s$).
   • A visibility mask ($m_s$) is concatenated to the input to explicitly tell the model which parts to preserve and which to generate.
   • Formula: $q_{comb} = m_s \odot q_{vis} + (1 - m_s) \odot \epsilon_s$.
3. Inpainting Network ($G_{inp}$):
   • A Flow Transformer is adapted to perform latent inpainting. It takes the mixed latent input and predicts the complete sparse structure latent ($q$).
   • Training: The model is trained on pairs of (partial visible latent + mask) $\to$ (ground-truth complete latent) using Conditional Flow Matching (CFM) loss. This teaches the model to infer missing geometry from visible cues while maintaining global consistency.

B. Staged Sampling Strategy

To ensure high-quality generation, the inference process is split into two stages:

1. Structural Inpainting Stage (Steps $1 \to s$): The model generates a coarse, globally consistent skeleton guided by the fixed visible regions.
2. Boundary Refinement Stage (Steps $s+1 \to t$): The visibility mask is replaced with an all-ones mask (effectively turning inpainting into standard denoising). This allows the model to refine the boundaries between the generated and observed regions, eliminating "holes" or artifacts caused by the initial downsampling, without altering the anchored geometry.

3. Key Contributions

• Explicit 3D Priors in Latent Space: Unlike previous methods that condition on 2D images or text, this work explicitly embeds 3D point cloud priors into the latent initialization of a diffusion model, enabling hard geometric constraints.
• Structure-Aware Inpainting: The framework reformulates 3D generation as a latent inpainting problem, allowing the model to learn global structural completion from partial observations naturally within the generative process.
• Dual-Input Flexibility: The system works with both accurate sensor point clouds and estimated point clouds from single images (via VGGT), making it applicable to real-world scenarios where high-precision sensors may not be available.
• Staged Sampling: The introduction of a two-stage sampling strategy (inpainting followed by boundary refinement) significantly improves the quality of the transition between observed and generated regions.

4. Experimental Results

The method was evaluated on object-level (Toys4K) and scene-level (3D-FRONT) benchmarks, comparing against SOTA baselines like TRELLIS, GaussianAnything, and VoxHammer.

• Geometric Fidelity:
  • On the Toys4K dataset, Points-to-3D with explicit point cloud priors achieved an F-Score of 0.964 and a Chamfer Distance of 0.013, significantly outperforming TRELLIS (F-Score 0.832, CD 0.034).
  • In the visible regions (areas covered by the input point cloud), the method achieved near-perfect alignment (F-Score 0.998, CD 0.007), proving it strictly adheres to the input constraints.
• Rendering Quality:
  • The method achieved higher PSNR, SSIM, and lower LPIPS/DINO scores compared to all baselines, indicating better visual fidelity and texture consistency.
• Robustness:
  • Even when using VGGT-estimated point clouds (which contain prediction noise), the method still outperformed other baselines, demonstrating robustness to imperfect priors.
• Ablation Studies:
  • The staged sampling strategy was proven critical; using only inpainting steps resulted in geometric "holes" at boundaries, while the refinement stage eliminated these artifacts.

5. Significance and Impact

• Bridging Perception and Generation: Points-to-3D effectively bridges the gap between 3D perception (reconstruction/sensing) and 3D generation. It allows generative models to act as "completers" rather than just "creators from scratch."
• Controllable 3D Creation: It offers a new paradigm for geometry-controllable 3D generation, essential for applications like digital twins, robotics (where partial scans exist), and AR/VR content creation where specific structural alignment is required.
• Efficiency: By leveraging the latent space of TRELLIS, it avoids the computational cost of iterative 3D optimization (like NeRF training) while providing precise geometric control.

In summary, Points-to-3D demonstrates that explicitly embedding point cloud priors into the latent diffusion process is a superior strategy for achieving structurally faithful, high-fidelity 3D generation, outperforming methods that rely solely on 2D or text conditioning.