Technical Summary: Rigel3D

Problem Statement

Recent advancements in 3D generative models have enabled the synthesis of high-quality static assets from images, text, and latent distributions. However, a critical limitation persists: these outputs lack the skeletal rigs, joint hierarchies, and skinning weights required for standard animation pipelines. Consequently, generated meshes often require substantial post-processing to become "animation-ready" for use in games, film, simulation, and embodied AI.

Existing automatic rigging methods (e.g., Pinocchio, RigNet, TARig, and recent template-free approaches) attempt to bridge this gap by predicting skeletons and skinning weights for completed static shapes. The paper argues that this post-hoc approach is fundamentally limiting. Because the geometry, topology, and part structure of generative outputs often differ significantly from the training distribution of downstream riggers, treating rigging as a separate step after shape generation leads to inconsistencies. The core challenge is to generate geometry and rig structure jointly, despite their heterogeneous nature: surface geometry is dense and local, while skeletons are sparse, hierarchical, and internal. Furthermore, practical animation workflows require semantic joint labels for motion retargeting, which template-free generators typically fail to provide.

Methodology

The authors introduce Rigel3D, a generative framework that produces animation-ready 3D assets (meshes, skeletons, skinning weights, and joint labels) in an end-to-end manner. The method extends the Structured LATent (SLAT) representation from TRELLIS by learning two coupled latent sets:

1. Surface SLats: Capturing geometry and appearance.
2. Skeleton SLats: Capturing internal articulation structure.

1. Rig-Aware Autoencoder

The core of the system is a rig-aware autoencoder that maps rigged meshes to these coupled latent representations and decodes them back into a complete asset.

• Encoders:
  • Surface Encoder: Adapts the TRELLIS surface encoder, voxelizing the input mesh and aggregating multi-view DINOv2 features to produce surface SLats.
  • Rig Encoder: Constructs bone features by skinning-weighted averaging of the DINOv2 features of vertices influenced by each bone. These features are rasterized onto the voxel grid and processed by a separate sparse encoder to produce skeleton SLats.
• Decoders:
  • Mesh Decoder: Extends the TRELLIS decoder to predict FlexiCubes parameters. Crucially, it incorporates cross-attention from surface tokens to skeleton tokens, conditioning surface reconstruction on the internal rig structure.
  • Skeleton Decoder: Uses an autoregressive transformer to decode skeleton topology. It represents skeletons as sequences of discrete tokens (joint coordinates and parent indices) in a breadth-first-search order. The generation is conditioned on both surface and skeleton SLats via cross-attention.
  • Skinning Weight Decoder: An attention-based module that predicts skinning weights by computing point-bone similarity. It embeds mesh points and bone coordinates into a shared space, refining them through self-attention and cross-attention to the latent representations. This allows the model to handle skeletons with variable numbers of bones.

2. Animation-Ready Asset Generation

To generate assets from images, Rigel3D employs a two-stage latent generative model (inspired by TRELLIS) applied to both surface and skeleton SLats:

• Stage I (Occupancy): A rectified-flow transformer generates the sparse voxel occupancy grid (which voxels are active) conditioned on input image features.
• Stage II (Latent Codes): A second flow transformer generates the local latent codes attached to the active voxels.
  The resulting latent representations ($Z_{surf}$ and $Z_{skel}$) are passed to the respective decoders to synthesize the mesh, skeleton, and skinning weights simultaneously.

3. Open-Vocabulary Joint Labeling

To support motion retargeting, the paper introduces a module that assigns semantic labels to generated joints.

• Instead of a closed-set classifier, the model embeds generated joints into a shared vision-language space (using a frozen OpenCLIP text encoder).
• It utilizes an autoregressive transformer that conditions on joint coordinates, hierarchy, and previous label embeddings to predict joint embeddings.
• This allows generated rigs to be matched to arbitrary candidate label sets (e.g., specific retargeting templates) via nearest-neighbor retrieval, decoupling the rig generation from a fixed vocabulary.

Key Contributions

1. Rigel3D Framework: An end-to-end generative method for creating animation-ready 3D assets, producing meshes, optional appearance, skeletons, and skinning weights jointly rather than sequentially.
2. Coupled Latent Representations: A rig-aware autoencoder utilizing coupled surface and skeleton SLats, featuring a skeleton-conditioned mesh decoder, an autoregressive skeleton decoder, and an attention-based skinning decoder.
3. Open-Vocabulary Labeling: A novel module enabling motion retargeting by matching generated joints to text labels in a shared vision-language space, supporting arbitrary templates without retraining.
4. State-of-the-Art Performance: Demonstrated superior performance over existing auto-rigging baselines (Anymate, Puppeteer) and concurrent generation methods (AniGen) across multiple metrics on large-scale rigged asset datasets.

Results and Evaluation

The method was evaluated on the Anymate and ModelsResource datasets.

• Quantitative Performance: Rigel3D outperformed baselines (Anymate, Puppeteer) on skeleton metrics (J2J, J2B, B2B) and skinning metrics ($\ell_1$, $\ell_2$, KL divergence). Notably, it showed significant gains in bone-level structure (B2B) and skinning distribution alignment (KL).
• Comparison with AniGen: When compared to the concurrent AniGen method, Rigel3D achieved substantial improvements in skeleton accuracy (e.g., a 32.8% relative reduction in J2J on ArticulationXL) while remaining competitive in skinning prediction.
• Qualitative Results: Visual comparisons demonstrated that Rigel3D produces more accurate joint placement, coherent connectivity, and smoother skinning transitions compared to post-hoc rigging methods.
• Labeling Accuracy: The open-vocabulary labeling module achieved high Top-1 and Top-3 accuracy in assigning semantic labels to joints, outperforming non-autoregressive variants.

Significance

The paper claims that Rigel3D narrows the gap between static 3D generation and rig-based animation workflows. By jointly modeling the interdependent geometry and rig structure, the method avoids the distributional mismatches inherent in post-hoc rigging. The introduction of open-vocabulary joint labeling further enhances practical utility by enabling direct compatibility with diverse downstream motion retargeting pipelines. The authors position this work as a step toward fully automated, production-ready 3D asset creation where the output is not just a static shape, but a fully rigged and animatable entity.

Limitations

The authors acknowledge several limitations:

• Generated rigs may still contain missing joints, spurious branches, or imperfect connectivity.
• Skinning deformations may be less natural under extreme poses, as the model is not supervised by long motion sequences.
• Open-vocabulary labels can be ambiguous for repeated structures (e.g., fingers, tails).
• The method inherits limitations of image-conditioned 3D generation, such as sensitivity to ambiguous or occluded input views.