SK-Adapter: Skeleton-Based Structural Control for Native 3D Generation

1. Problem Statement

While recent native 3D generative models (e.g., Trellis) have achieved remarkable speed and visual fidelity, they suffer from a critical limitation: the inability to prescribe precise structural articulations.

• Limitations of Current Prompts: Text prompts are often too ambiguous for specific geometric constraints (e.g., "bend knee 60 degrees"), and image prompts lack 3D depth information.
• Limitations of Existing 3D Control Methods: Previous approaches attempting to use skeletons often rely on a "2D lifting" strategy (e.g., SKDream). These methods project 3D skeletons into 2D views to condition multi-view diffusion models, which are then reconstructed into 3D. This introduces:
  • Spatial Ambiguity: Flattening 3D structures into 2D loses depth information and causes self-occlusion issues.
  • Quality Degradation: Multi-stage reconstruction pipelines often result in geometric artifacts and texture inconsistencies.
• The Gap: There is a lack of a method that can inject skeletal control signals directly within the native 3D domain without compromising the generative priors of the foundation model.

2. Methodology: SK-Adapter

The authors propose SK-Adapter, a lightweight, parameter-efficient framework that enables precise skeletal manipulation for native 3D generation. It is built upon the Trellis foundation model (a Structured Latent 3D Flow Transformer).

Core Architecture

1. Skeleton as First-Class Control: Instead of 2D projections, the 3D skeleton is treated as a set of sparse spatial tokens containing both joint coordinates and topological constraints.
2. Topology-Aware Encoding (GRPE):
   • The input skeleton is modeled as a graph $G=(V, E)$.
   • The authors employ Graph Relative Positional Encoding (GRPE) to encode the skeleton.
   • Two types of embeddings are learned:
     • Topological Distance ($D_{ij}$): The shortest path length between joints (clipped to a max distance).
     • Edge Relations ($R_{ij}$): Semantic roles such as parent, child, sibling, or end-effector.
   • These embeddings create structural attention biases that allow the model to understand the kinematic hierarchy of the skeleton.
3. Skeletal Cross-Attention Mechanism:
   • The encoded skeletal features ($f_{skel}$) are injected into the frozen pre-trained backbone via Cross-Attention layers.
   • Query (Q): Derived from the voxel features of the frozen backbone.
   • Key (K) & Value (V): Derived from the encoded skeletal tokens.
   • This allows each 3D voxel to dynamically "attend" to the most relevant skeletal joints, mapping topological constraints directly onto the 3D coordinate grid.
4. Parameter-Efficient Fine-Tuning (PEFT):
   • The backbone (Trellis) is frozen to preserve its high-quality generative priors.
   • Only the Skeleton Encoder, Cross-Attention layers, and Zero-initialized Linear projection layers are trained.
   • The zero-initialized linear layer ensures that at the start of training, the adapter contributes a "null signal," preventing catastrophic forgetting and ensuring training stability.

Training and Data

• Training Objective: The model is trained using Latent Flow Matching (LFM) in the compact latent space of a pre-trained 3D Voxel Autoencoder.
• Objaverse-TMS Dataset: To address the lack of synchronized data, the authors curated a new dataset, Objaverse-TMS, containing 24,000 high-quality text-mesh-skeleton triplets. Unlike previous datasets that used autonomous (algorithmic) rigging, these skeletons are expert-annotated, ensuring anatomical correctness and physical plausibility.

Editing Capabilities

The framework supports tuning-free local editing. By combining masked voxel regions with modified skeleton prompts (e.g., changing joint angles or adding limbs), the model can regenerate specific parts of an asset while maintaining coherence with the rest of the object.

3. Key Contributions

1. First Native 3D Skeleton Control: SK-Adapter is the first framework to achieve skeletal control directly within the native 3D generation space, bypassing the lossy 2D projection bottleneck.
2. Objaverse-TMS Dataset: The creation of a large-scale, expert-annotated dataset of 24k text-mesh-skeleton triplets, solving a major data bottleneck for structure-guided 3D generation.
3. Flexible 3D Editing: The method enables precise, region-specific editing (addition, re-posing, replacement) guided by skeleton prompts, a capability previously unattainable with existing methods.
4. Architecture Design: The introduction of a Topology-Aware Cross-Attention mechanism that effectively bridges the gap between sparse skeletal graphs and dense 3D voxel latents.

4. Experimental Results

The method was evaluated on the TMS-eval benchmark (140 instances across humanoid, animal, and object categories) against baselines like SKDream (2D lifting) and SpaceControl (training-free spatial guidance).

• Structural Alignment (ReRigging Score): SK-Adapter achieved a score of 0.2228, significantly outperforming SKDream (0.2818) and SpaceControl (0.2740). A lower score indicates better adherence to the input skeleton.
• Textual Alignment (CLIP Score): Achieved the highest score (26.16), demonstrating robust cross-modal alignment.
• Visual Fidelity:
  • PickScore: 21.01 (highest), indicating strong human preference for the generated geometry.
  • KD-DINO: 0.7778 (lowest), indicating the generated distribution closely matches ground-truth references without the geometric artifacts seen in multi-stage pipelines.
• Efficiency: Generation time is <15 seconds per asset, compared to ~40 seconds for SKDream.
• Ablation Studies: Removing the Cross-Attention mechanism caused structural alignment to degrade drastically (ReRigging Score doubled), and removing Topological Encoding reduced the model's ability to understand complex hierarchies.

5. Significance and Impact

• Bridging Semantics and Topology: SK-Adapter successfully bridges the gap between abstract semantic synthesis (text-to-3D) and explicit topological articulation (skeleton-to-3D).
• Production Readiness: By generating assets that strictly adhere to user-defined skeletal constraints, the output is immediately suitable for downstream applications in animation and gaming pipelines (e.g., rigging and skinning).
• Paradigm Shift: The work demonstrates that precise structural control in 3D does not require compromising generative quality or relying on inefficient multi-stage reconstruction. It establishes a new standard for controllable native 3D generation using lightweight adapters.