The Quest for Generalizable Motion Generation: Data, Model, and Evaluation

1. Problem Statement

Despite significant advances in 3D human motion generation (MoGen), existing text-to-motion models suffer from a fundamental generalization bottleneck.

• Data Scarcity & Bias: Traditional optical Motion Capture (MoCap) datasets are high-fidelity but limited in scale and semantic diversity (mostly indoor, locomotion-heavy actions). Conversely, web-scale video data offers broad semantic coverage but suffers from low motion fidelity and artifacts when converted to 3D.
• Evaluation Gaps: Current benchmarks rely heavily on distribution-based metrics (e.g., FID) that fail to capture fine-grained generalization capabilities or align with human judgment on complex, long-tail instructions.
• The Gap: Models struggle to generalize to novel, diverse, or complex instructions (e.g., martial arts, multi-step behaviors) because they cannot effectively leverage the semantic richness of video generation models while maintaining the physical plausibility of motion data.

2. Methodology

The authors propose a holistic framework addressing three pillars: Data, Modeling, and Evaluation.

A. Data: ViMoGen-228K

A large-scale, heterogeneous dataset comprising 228,000 motion samples designed to balance high fidelity with semantic diversity. It integrates three sources:

1. Optical MoCap (171.5K pairs): Aggregated from 30 standardized datasets (e.g., HumanML3D, AMASS). These provide high-quality physical priors but are text-annotated via a pipeline using Gemini 2.0 to render depth videos for description.
2. In-the-Wild Video (41.9K triplets): Extracted from a massive internal video database (10M clips) using rigorous filtering. Motion is extracted via visual MoCap (CameraHMR, SMPLest-X) and annotated.
3. Synthetic Video (14.7K triplets): Generated using state-of-the-art Video Generation (ViGen) models (Wan2.1) prompted with a long-tail vocabulary to create controlled, high-quality human videos that are easier to capture than real-world chaotic scenes.

B. Model: ViMoGen & ViMoGen-light

A Flow-Matching based Diffusion Transformer (DiT) architecture that unifies priors from MoCap and ViGen.

• Dual-Branch Architecture:
  • Text-to-Motion (T2M) Branch: Conditions on text embeddings and MoCap priors. Ensures physical plausibility and robustness for well-represented actions.
  • Motion-to-Motion (M2M) Branch: Conditions on "video motion tokens" (extracted from ViGen outputs or simulated via noise perturbation of ground truth). Transfers semantic knowledge from video models to handle novel/rare actions.
• Adaptive Gating Mechanism: A key innovation where the model dynamically selects between branches.
  • Inference: Uses a Vision-Language Model (VLM) to check alignment between the text prompt and a generated video. If aligned, the M2M branch refines the motion; if not, it falls back to the robust T2M branch.
  • Training: Uses a curriculum approach (prioritizing T2M for annotated data, M2M for auto-labeled data) and simulates visual MoCap errors (jitter, trajectory noise) to train the M2M branch without requiring online video generation.
• ViMoGen-light: A distilled variant that eliminates the dependency on video generation models at inference. It is trained via knowledge distillation from the full ViMoGen teacher using a synthetically diverse dataset of 14,000 prompts, inheriting generalization capabilities while remaining computationally efficient.

C. Evaluation: MBench

A hierarchical benchmark designed for fine-grained assessment, moving beyond single-score metrics.

• Three Pillars: Motion Quality, Motion-Condition Consistency, and Motion Generalizability.
• Nine Dimensions: Includes temporal quality (jitter, foot sliding), frame-wise quality (pose naturalness, body penetration), and semantic alignment.
• Open-World Vocabulary: Uses a curated list of rare and complex prompts (e.g., "bodysurfing," "jousting") to test true generalization rather than dataset memorization.
• Human Alignment: Validated against human preference annotations to ensure automated metrics correlate with human judgment.

3. Key Contributions

1. ViMoGen-228K: The first large-scale dataset to systematically combine high-fidelity optical MoCap, in-the-wild video, and synthetic video data, featuring both text-motion pairs and text-video-motion triplets.
2. ViMoGen Architecture: A novel gated diffusion transformer that adaptively fuses semantic knowledge from video generation models with physical priors from MoCap data, achieving superior generalization without sacrificing motion quality.
3. ViMoGen-light: An efficient, distilled model that achieves state-of-the-art generalization without the computational overhead of running video generation models during inference.
4. MBench: A comprehensive, human-aligned benchmark that provides granular evaluation across quality, consistency, and generalization, addressing the lack of unified standards in the field.

4. Results

Extensive experiments on MBench and standard benchmarks (HumanML3D) demonstrate significant improvements:

• Generalization: ViMoGen achieves a Motion Generalizability score of 0.68, significantly outperforming the next best baseline (0.55). It successfully generates plausible motions for out-of-distribution prompts (e.g., martial arts, complex interactions) where other models fail or produce generic movements.
• Consistency: It achieves the highest Motion-Condition Consistency (0.53), indicating superior semantic alignment between text and motion.
• Quality: While generalization often trades off with quality, ViMoGen maintains competitive scores in physical plausibility (low foot sliding, low jitter) and achieves a new SOTA on HumanML3D (R-Precision Top-1: 0.542, FID: 0.114).
• Ablation Studies: Confirm that the adaptive gating mechanism is crucial; using only T2M limits generalization, while using only M2M degrades quality. The combination yields the best results.

5. Significance

This work represents a paradigm shift in 3D motion generation by bridging the gap between high-fidelity motion capture and broad semantic video generation.

• Scalability: It demonstrates that leveraging video generation priors is a viable path to solving the data scarcity and semantic bias problems in MoGen.
• Efficiency: The distillation strategy (ViMoGen-light) proves that complex multimodal knowledge can be compressed into efficient models, making generalizable motion generation accessible for real-time applications.
• Standardization: By introducing MBench, the paper sets a new standard for evaluating motion models, shifting the focus from simple distribution matching to rigorous, human-aligned assessment of generalization and physical realism.

The authors conclude that this framework lays the groundwork for a general-purpose motion foundation model, capable of understanding and generating human behavior across a vast spectrum of contexts.