Technical Summary: SAMoR – Motion Modelling for Articulated Objects of Any Skeleton and Topology

1. Problem Statement

Modeling motion for articulated objects with arbitrary skeleton topologies (e.g., humans, quadrupeds, birds, insects, and mechanical rigs) remains a significant challenge. Existing state-of-the-art motion generators, particularly text-to-motion systems, are typically tied to a fixed human skeleton (e.g., SMPL) with a specific joint count, ordering, and connectivity. While extensions to animal categories exist, they generally assume a predefined topology.

Cross-topology approaches have emerged but face limitations:

• Single-token approaches (K=1): Collapse distinct motion details into a global representation, losing limb and appendage dynamics.
• Per-joint approaches (K=J): Treat each joint independently, preventing the sharing of a discrete motion vocabulary across rigs with different joint counts.
• Lack of Shared Vocabulary: There is no fixed-size discrete motion codebook that can be reused across heterogeneous skeletons for tasks like motion transfer or token-based generation.

The core difficulty lies in the fact that while joint-level motion does not correspond cleanly across species (due to differing joint counts and connectivity), functional joint groups (e.g., a human arm, a wolf foreleg, and a bird wing) share semantic motion structures.

2. Methodology: SAMoR

The authors propose SAMoR (Skeleton-Aware Motion Representation for Articulated Objects), a cross-topology motion representation that encodes motion segments into a small, fixed number of K=8 part tokens shared across arbitrary skeletons.

Architecture Overview

SAMoR is implemented as a Skeleton-Aware Part VQ-VAE (Vector Quantized Variational Autoencoder) with the following components:

1. Input Representation:

   • Motion Features: Per-joint positions and velocities ($x_t^j \in \mathbb{R}^6$). Local rotations are omitted in the cross-topology setting due to inconsistent rest poses across heterogeneous assets.
   • Skeleton Conditioning: The encoder and decoder are conditioned on the target skeleton via three signals:
     • Pose Anchors: Per-joint 3D locations (first frame for cross-topology; canonical rest pose for HumanML3D).
     • Joint Name Embeddings: CLIP embeddings of joint names (e.g., "forearm", "wing_L1").
     • Graph Structure: Learned biases injected into spatial attention logits based on kinematic tree hop distance, chain membership, and edge type.

2. Encoder (Graph-Transformer):

   • Processes motion chunks using a stack of transformer blocks alternating between temporal self-attention (along time $T$) and spatial self-attention (along joints $J$).
   • The spatial attention utilizes the graph structural biases defined above.
   • A strided 1D convolution downsamples the temporal axis by 4$\times$.

3. Part Pooling (Joints to Tokens):

   • Cross-Attention Pooling: $K=8$ learnable query vectors compress the $J$ encoded joints into $K$ fixed-size part tokens.
   • Specialization Mechanism: Without supervision, these queries tend to collapse into identical distributions (effectively $K=1$). To prevent this, the authors introduce Attention Supervision.
   • Attention Supervision Loss ($L_{attn}$): During training, canonical functional-group labels (derived from joint names) guide the cross-attention maps. Each query is encouraged to attend to a distinct functional group (e.g., left leg, tail, head). Joints without matching labels ($y_j = -1$) contribute zero to the loss but are still routed by the learned attention.
   • Joint-Name Dropout: To ensure robustness to rigs with missing or non-semantic names, joint name embeddings are randomly dropped during training. This forces the model to rely on motion patterns and skeleton topology rather than text labels alone.

4. Quantization:

   • The $K$ part tokens are quantized using Residual Vector Quantization (RVQ) with a shared codebook. This creates a discrete motion vocabulary ($Z \in \{1, \dots, V\}$) shared across all rigs.

5. Decoder:

   • Mirrors the encoder but expands the $K$ quantized part tokens back to $J$ target joints via inverse cross-attention.
   • The decoder is conditioned on the target skeleton's pose anchor, joint names, and graph biases. This allows the same source tokens to be decoded onto a completely different rig (cross-topology transfer) without retraining.

Downstream Tasks

• Text-Conditioned Generation: A MaskGIT transformer is trained over the $(T/4) \times K$ token grid, conditioned on text and the target skeleton.
• Part-wise Editing: Specific part tokens can be masked and in-filled, allowing localized editing of functional groups (e.g., changing leg motion while preserving arm motion).

3. Key Contributions

1. Graph-Transformer-based Motion Tokenization: A VQ-VAE that compresses arbitrary-skeleton motion into $K=8$ shared part tokens, yielding a single discrete motion vocabulary across heterogeneous skeletons.
2. Attention Supervision for Query Specialisation: A novel loss function that breaks query symmetry by anchoring each query to a distinct functional joint group, preventing the collapse of $K=8$ queries into a single global representation. Combined with joint-name dropout, this generalizes to unseen skeletons.
3. Unified Heterogeneous Motion Corpus: A curated dataset of $\sim71k$ characters and $\sim100k$ motions from HumanML3D (humans), Truebones Zoo (65 animal species), and animated Objaverse-XL assets. This includes automated canonical joint-matching and functional-group annotations.
4. Demonstrated Downstream Capabilities: The shared part-token space supports cross-topology motion transfer, text-conditioned generation, and localized part-wise editing.

4. Experimental Results

The authors evaluated SAMoR on held-out characters with unseen skeletons.

• Cross-Topology Reconstruction:

  • SAMoR achieved a normalized MPJPE of 2.75 $\times 10^{-2}$.
  • This is 5.8$\times$ lower (better) than the strongest adapted variable-$J$ tokenizer baseline (MoGenTS, 16.14 $\times 10^{-2}$).
  • It significantly outperforms both single-token global representations ($K=1$) and per-joint tokenizations ($K=J$).
  • Ablation studies confirmed that removing attention supervision ($L_{attn}$) nearly doubled the error, and removing RVQ increased error by 44%.

• HumanML3D Benchmark:

  • When instantiated in a HumanML3D-compatible setting, SAMoR remains competitive with fixed-skeleton specialists.
  • Text-to-Motion: FID of 0.039, comparable to MoGenTS (0.033) and better than MoMask (0.045).
  • VQ-VAE Reconstruction: SAMoR-H3D achieved 17.3 mm MPJPE, outperforming MoMask (29.5 mm) and T2M-GPT (58.0 mm).

• Cross-Topology Transfer:

  • Qualitative results (Fig. 3) show SAMoR preserves appendage dynamics (limbs, tails, wings) across topology shifts, whereas $K=1$ baselines retain only coarse body rhythm.
  • Round-trip consistency (Source $\to$ SMPL $\to$ Source) was highest for SAMoR, indicating the part-token representation preserves motion structure through intermediate topologies.

5. Significance and Claims

The paper claims that SAMoR addresses the fundamental limitation of current motion models being tied to fixed topologies. By identifying that functional joint groups are the appropriate abstraction level for cross-topology motion (rather than individual joints or the whole body), SAMoR enables:

• Reusability: A single discrete codebook can be used for humans, animals, and mechanical rigs.
• Transfer: Motion can be transferred directly at decode time to unseen skeletons.
• Editing: Localized editing of functional groups is possible without retraining.

The authors emphasize that the intermediate granularity ($K=8$) is crucial; $K=1$ loses detail, while $K=J$ prevents vocabulary sharing. The attention supervision loss is identified as the critical component that prevents query collapse, enabling the model to learn functional clusters from a mix of motion, structure, and (optional) text labels.

Limitations Acknowledged:

• In the cross-topology setting, SAMoR operates on joint positions rather than rotations because heterogeneous assets lack a unified rest-pose reference. Recovering bone rotations for Linear Blend Skinning (LBS) requires an Inverse Kinematics (IK) post-process, which introduces twist ambiguity.
• The $K=8$ grouping is a fixed design choice; automatically learning the number and structure of groups is suggested as future work.