Technical Summary: Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation

1. Problem Statement

The field of human motion analysis is currently characterized by severe fragmentation. Existing paradigms are divided into two distinct, non-overlapping camps:

• Perception Models: These models (e.g., MotionBERT, SiC) excel at understanding motion from raw video but are typically limited to outputting text descriptions or static 2D/3D poses without the capacity to generate new, plausible motion sequences.
• Generation Models: These models (e.g., MotionGPT, MotionGPT3) are capable of generating motion sequences from text or abstract inputs but often lack the ability to perceive or process raw visual inputs (video). Furthermore, many generative Multi-modal Large Language Models (MLLMs) are restricted to single-frame, static poses using dense parametric models (like SMPL), failing to capture temporal dynamics.

Additionally, existing motion vocabularies are constructed solely from skeleton data, severing the link to the visual domain. This "read-only" vs. "write-only" dichotomy forces researchers to use disparate architectures for inherently interconnected tasks, preventing models from leveraging the synergistic relationship between perception and generation.

2. Methodology

The authors propose Superman, a unified, multi-task, and multi-modal generative framework that reformulates human motion analysis as a conditional sequence generation problem. The framework consists of two primary stages:

A. Vision-Guided Motion Tokenizer (VGMT)

The cornerstone of the approach is a novel tokenizer that creates a discrete, cross-modal motion vocabulary. Unlike prior works that rely on skeletons alone, VGMT is built upon a Vector Quantized Variational Autoencoder (VQ-VAE) architecture designed to fuse visual appearance and 3D skeletal geometry.

• Dual-Stream Encoding:
  • Visual Stream: Extracts pose-centric visual features from video frames using a vision backbone (e.g., HRNet) and a Visual-Skeleton Attention (VSA) module. The VSA adaptively aggregates features by predicting sampling offsets and weights based on projected 2D joint locations, enhancing robustness to occlusion.
  • Skeletal Stream: Captures intrinsic motion geometry by processing the 3D pose sequence through lightweight 2D convolutions on a joint-temporal grid.
• Hybrid Codebook Quantization: The system employs a hybrid codebook where each token is a bimodal entity comprising paired visual and geometric prototypes. For each time window, the token index is selected by minimizing the joint Euclidean distance between the fused visual and skeletal features and the codebook prototypes. This ensures the discretization process is grounded in both appearance and geometry.
• Training: The tokenizer is trained end-to-end with a VQ objective combining reconstruction loss and modality-wise commitment losses.

B. Unified Multi-Task Modeling with MLLM

A single decoder-only Multi-modal Large Language Model (specifically Qwen2.5-VL-7B) serves as the central sequence processor.

• Motion-Aware Fine-Tuning (MAFT): An optional lightweight module injects skeletal geometry into the MLLM's visual stream. It enhances Vision Transformer (ViT) features using a VSA block, fusing grid tokens (queries) with pose tokens (keys/values) via cross-attention.
• Task Unification: The MLLM is trained jointly on a mixture of task-specific formats using a standard autoregressive cross-entropy loss. It handles three distinct tasks by varying conditioning inputs:
  1. 3D Pose Estimation (PE): Translates video clips into sequences of pose tokens.
  2. Motion Prediction (MP): Autoregressively predicts future pose tokens given a historical sequence.
  3. Motion In-betweening (MIB): Generates intermediate tokens between start and end keyframes.

3. Key Contributions

1. Unified Framework: The proposal of a single generative framework that leverages one MLLM for multi-task, multi-modal human motion analysis, effectively bridging the gap between motion perception and generation.
2. Vision-Guided Motion Tokenizer: The introduction of a novel tokenizer that creates a robust, cross-modal pose vocabulary. By designing a hybrid codebook where tokens are bimodal entities (paired visual and geometric prototypes), the quantization process is simultaneously guided by video appearance and 3D skeleton geometry.
3. State-of-the-Art Performance: The demonstration that this unified method achieves state-of-the-art or competitive performance across diverse tasks (PE, MP, MIB), validating the efficacy of learning a rich, joint representation.

4. Experimental Results

The framework was evaluated on standard benchmarks, primarily Human3.6M and the unseen 3DPW dataset.

• Performance on Human3.6M: Superman achieved state-of-the-art results across all three tasks.
  • Pose Estimation: It outperformed traditional multi-task models (e.g., Human-in-Context) by 11.97% and LLM/MLLM-based methods (e.g., PoseLLaVA) by 10.91% in terms of N-MPJPE.
  • Motion Prediction & In-betweening: It secured the lowest Mean Per Joint Position Error (MPJPE) compared to both traditional and LLM-based baselines.
• Generalization: Trained exclusively on Human3.6M, Superman demonstrated exceptional zero-shot generalization on the 3DPW dataset, significantly outperforming all baseline models in both motion prediction and in-betweening tasks.
• Efficiency: While the large-scale Qwen2.5-VL-7B dominates resource consumption, the newly introduced components (MAFT and VSA) account for less than 0.03% of the total computation cost, demonstrating high scalability.
• Ablation Studies:
  • Tokenizer: Joint learning from vision and geometry proved essential; visual-only models failed, and skeleton-only baselines were significantly outperformed by fused configurations.
  • Training Strategy: The unified multi-task training strategy consistently outperformed specialized models trained on single tasks, indicating positive knowledge transfer.
  • Scaling: Increasing model size (3B to 7B) and codebook capacity consistently reduced error rates, confirming the framework's scalability.

5. Significance

The paper claims that Superman represents a more efficient and scalable path for generative motion analysis. By conceptualizing motion as a universal language and grounding it in both visual and geometric modalities, the work challenges the fragmented ecosystem of current human motion analysis. It demonstrates that a single, unified model can fluidly process multi-modal inputs—from full videos for pose estimation to skeleton sequences for prediction—and generate coherent, structured 3D motion, thereby unifying the "reading" and "writing" of motion within a coherent architecture.