Technical Summary: CubePart: An Open-Vocabulary Part-Controllable 3D Generator

Problem Statement

Interactive 3D assets in games and simulations require decomposition into specific semantic parts to support animation, physics, and scripted behaviors. However, current generative 3D models typically produce either monolithic meshes or arbitrary part decompositions that do not align with application-specific schemas. Existing methods relying on 2D grounding (e.g., image segmentation) fail to control hidden parts or define complete 3D semantic structures due to view dependency. Furthermore, existing part-aware generation approaches often rely on fixed taxonomies or closed vocabularies, limiting their ability to accommodate the diverse, open-ended part definitions required by different downstream applications. There is a critical need for a 3D-native, schema-driven interface that allows users to explicitly specify an open-vocabulary list of semantic parts during generation.

Methodology

CubePart is a two-stage generative framework designed to synthesize multi-part 3D meshes conditioned on a global text prompt and a user-defined part schema (an open-ended list of part names).

1. Data Engine

To support open-vocabulary training, the authors constructed a large-scale dataset comprising approximately 462K assets and 2.02M parts. This pipeline automates the creation of part-labeled data from unstructured meshes using:

• Preprocessing: Filtering degenerate geometry and retaining assets with 2–32 parts.
• VLM-Based Quality Filtering: Using Vision-Language Models (VLMs) to identify mesh defects, scan artifacts, and scene-level content.
• Set-of-Mark (SoM) Rendering: A novel annotation strategy where 14 orbital views are rendered as paired images: a textured view with part contours and numbered markers, and a part-colored view where each part is a distinct solid color. This pairing provides both semantic context and unambiguous part identification.
• VLM-Based Clustering and Naming: The VLM processes all view pairs simultaneously to cluster related parts based on function or logical relationships and assigns concise, descriptive names (e.g., "front left wheel" rather than a generic caption).
• Postprocessing: Converting parts into watertight meshes using Dual Marching Cubes on unsigned distance fields for training.

2. Generative Architecture

The framework employs a two-stage diffusion process based on VecSet (vector set) latent representations:

• Stage 1: Single-Part Mesh Generation (Global Synthesis)

  • Utilizes a Multi-Modal Diffusion Transformer (MM-DiT) conditioned on text prompts.
  • The text prompt is structured to explicitly enumerate the constituent parts (e.g., "This object contains the following parts: {list}").
  • This stage is pre-trained on ~4.7M mesh-text pairs and fine-tuned on the curated part-labeled dataset to ensure the generated global shape structurally aligns with the input schema.

• Stage 2: Multi-Part Mesh Generation (Decomposition)

  • Takes the full shape latent from Stage 1 and decomposes it into distinct part latents.
  • Part-Aware Prompting: The model is conditioned on the full list of parts and a specific target part name to guide segmentation boundaries.
  • Cross-Part Attention: To resolve geometric boundaries and prevent overlaps, the authors insert dedicated zero-initialized Transformer blocks (Cross-Part Attention Residual Blocks) into the pre-trained Stage 1 weights. These blocks facilitate efficient inter-part communication without disrupting the learned global geometric priors.
  • The output is a set of distinct meshes, one for each schema element, which assemble into a coherent object.

Key Contributions

1. Scalable Data Engine: A fully automated pipeline for constructing open-vocabulary, part-labeled 3D datasets using VLMs and Set-of-Mark prompting, resulting in a dataset over 11 times larger than previous efforts (PartVerse-XL).
2. Schema-Driven Architecture: A two-stage generative model that separates global shape synthesis from part-level decoding, enabling open-vocabulary, part-controllable generation while preserving global geometric coherence.
3. End-to-End Integration: A demonstration that the generated multi-part meshes can be directly integrated into game engines and driven by behavior scripts (e.g., Lua scripts for driving, flying, or character articulation) without manual post-processing.

Results and Evaluation

• Quantitative Performance: On the PartObjaverse-Tiny benchmark, CubePart outperforms baselines (including HoloPart, OmniPart, PartCrafter, and PartPacker) in both holistic and part-level metrics (Chamfer Distance and F-score). Specifically, the method achieves a holistic F-score of 0.974 and a part-level F-score of 0.743.
• Ablation Studies:
  • Removing schema-aware fine-tuning leads to missing or incorrectly emphasized parts.
  • Removing the Cross-Part Attention mechanism causes a severe drop in part-level accuracy, confirming the necessity of inter-part communication for resolving boundaries.
  • Modifying pre-trained local attention layers (as done in other works) degrades performance compared to using dedicated zero-initialized blocks.
• Qualitative Results: The system successfully generates complex objects (e.g., a turtle with a castle, a tank, a drone) where specific parts (wheels, turrets, propellers) are generated as distinct, structurally complete meshes matching the user's schema. It also demonstrates the ability to vary the granularity of parts (e.g., merging fenders into wheels vs. separating them) based on the input schema.

Significance and Claims

The paper claims that CubePart represents a significant step toward producing game-ready assets by shifting from monolithic generation to schema-driven, part-controllable synthesis. By allowing users to explicitly define the semantic structure of an object via text, the method bridges the gap between generative AI and the rigid requirements of interactive 3D environments. The authors emphasize that their approach enables the direct usage of generated assets for scripted or physically simulated behaviors, eliminating the need for manual decomposition and labeling by artists. The work positions text as a universal interface for controlling 3D structure, offering a flexible alternative to fixed taxonomies or view-dependent 2D controls.