UniPart: Part-Level 3D Generation with Unified 3D Geom-Seg Latents

1. Problem Statement

The paper addresses the challenge of part-level 3D generation, which aims to synthesize 3D objects as sets of coherent, semantically meaningful, and decomposable components (e.g., a chair with separate legs, seat, and backrest) from a single input image.

Existing methods face two primary limitations:

1. Implicit Segmentation: Methods that group latent vectors often lack control over segmentation granularity and part-level fidelity.
2. Explicit Segmentation: Methods that generate a whole object first and then segment it (using external segmenters like SAM) rely on large annotated datasets, suffer from geometry degradation during decode-encode cycles, and struggle to maintain global coherence while preserving local details.

The core difficulty lies in achieving structurally and semantically coherent part segmentation while preserving high-fidelity geometry, especially for small or thin components.

2. Methodology: UniPart

The authors propose UniPart, a two-stage latent diffusion framework that operates on a unified Geometry-Segmentation (Geom-Seg) latent space. The key insight is that part awareness emerges naturally during whole-object geometry learning.

A. Geom-Seg VecSet (Unified Latent Representation)

Instead of treating geometry and segmentation as separate tasks, the authors introduce Geom-Seg VecSet, a unified latent representation based on the VecSet architecture.

• Encoding: It encodes both the 3D mesh geometry and part segmentation labels into a single set of latent vectors.
• Decoding: The representation is decoded by two heads:
  • A Geometry Decoder ($D_{geom}$) reconstructs the mesh (similar to standard VecSet).
  • A Segmentation Decoder ($D_{seg}$) predicts part labels for each latent vector, inspired by the promptable segmentation pipeline of SAM2.
• Training: The VAE is trained to minimize reconstruction loss, segmentation loss, and KL divergence. Crucially, it is fine-tuned from a pre-trained geometric VecSet VAE, proving that adding segmentation information does not degrade geometric reconstruction quality.

B. Two-Stage Latent Diffusion Framework

UniPart employs a two-stage diffusion process to generate the final decomposable mesh:

Stage 1: Whole-Level Latent Diffusion

• Input: A single RGB image.
• Process: A global-level Diffusion Transformer (DiT) generates the Geom-Seg VecSet latent ($\hat{Z}_0$).
• Output: This stage produces the global geometry latent and a corresponding part latent mask. Unlike previous methods, this segmentation is explicit and learned jointly with geometry, providing flexible granularity.

Stage 2: Part-Level Latent Diffusion (Dual-Space Generation)

• Input: The segmented part latents ($X_i$) and the global object latent ($\hat{Z}_0$), conditioned on the input image.
• Dual-Space Mechanism: To solve the issue of part geometry degradation and placement, the model predicts part latents in two distinct spaces simultaneously:
  1. Global Coordinate Space (GCS): Encodes the part's scale, position, and compositional relationship to the whole object.
  2. Normalized Canonical Space (NCS): Encodes the high-fidelity intrinsic geometry of the part within a normalized $[0,1]^3$ frame.
• Architecture: The DiT uses Global-Local Attention to jointly attend to the whole-object context and local part details. It also injects Space Embeddings to distinguish between tokens from the GCS and NCS.
• Composition: During inference, the model denoises both spaces. The NCS latent is decoded into a high-quality canonical mesh, while the GCS latent provides the transformation (scale/position) to place the part correctly within the global object.

3. Key Contributions

1. Geom-Seg VecSet: A unified latent representation that jointly encodes object geometry and part semantics, enabling part-aware understanding directly from whole-object generative learning without external segmenters.
2. UniPart Framework: A two-stage latent diffusion model that performs joint geometry generation and part-latent segmentation, followed by dual-space part diffusion for seamless composition.
3. Dual-Space Generation: A novel procedure predicting part latents in both global and canonical spaces, which mitigates geometric degradation and ensures consistent part assembly.
4. Superior Performance: Extensive experiments demonstrating state-of-the-art results in segmentation controllability and part-level geometric quality compared to existing holistic and part-level methods.

4. Experimental Results

The authors evaluated UniPart on a curated dataset of 300K objects with part-level segmentation.

• Quantitative Metrics: UniPart outperforms baselines (including PartPacker, PartCrafter, OmniPart, HoloPart, and X-Part) across all metrics:
  • Chamfer Distance (CD): 0.72 (Lower is better), significantly outperforming the next best (X-Part at 0.82).
  • F-Score: 92.21 (Higher is better).
  • IoU: Competitive results in geometric alignment.
• Qualitative Results: Visualizations show that UniPart produces more reasonable part segmentations and higher-quality geometries, particularly for small or thin components where other methods fail or produce artifacts.
• Ablation Studies:
  • Removing Normalized Canonical Space (NCS) generation leads to significant degradation in fine details.
  • Removing Local Attention results in less harmonious spatial arrangements.
  • Removing Space Embeddings causes the model to confuse coordinate spaces, leading to assembly failures.
• Segmentation Quality: When compared against SOTA 3D segmentation models (SAMesh, PartField, P3-SAM) on a manually annotated test set, UniPart achieved the highest mean IoU (0.7222), proving the effectiveness of its implicit part-aware learning.

5. Significance

This work represents a significant advancement in structured 3D content creation. By unifying geometry and segmentation in a latent space, UniPart eliminates the need for error-prone post-processing segmentation or reliance on massive external annotated datasets. The dual-space generation strategy effectively solves the long-standing trade-off between global structural coherence and local geometric fidelity.

The method enables:

• Decomposable Editing: Users can manipulate individual parts of generated 3D objects.
• Physical Simulation & Robotics: Providing objects with distinct, physically valid parts is crucial for simulation and robotic manipulation tasks.
• Modular Design: Facilitates the creation of complex 3D assets from simple image prompts with inherent structural logic.

In summary, UniPart shifts the paradigm from "generate-then-segment" to "generate-with-segmentation," offering a more robust, efficient, and high-fidelity approach to part-level 3D generation.