QuadFM: Foundational Text-Driven Quadruped Motion Dataset for Generation and Control

1. Problem Statement

Despite advances in quadrupedal robotics, a critical bottleneck exists in foundational motion resources. Current datasets are limited to basic locomotion primitives (e.g., walk, trot, sit) and lack:

• Diversity: They fail to cover interactive behaviors, emotional expressions, or complex athletic maneuvers.
• Semantic Grounding: There is a lack of rich, open-vocabulary language annotations linking natural language commands to specific motions.
• Executability: Existing text-to-motion methods often decouple high-level generation from low-level control, producing kinematically plausible but dynamically infeasible motions that fail on real hardware.

The goal is to bridge the gap between natural language intent, diverse motion generation, and robust physical execution on real quadruped robots.

2. Methodology

The paper proposes a two-pronged solution: a massive new dataset (QuadFM) and a unified training framework (Gen2Control RL).

A. The QuadFM Dataset

QuadFM is a large-scale, ultra-high-fidelity dataset containing 11,784 curated motion clips and 35,352 natural language descriptions. It is constructed via a multi-source pipeline:

1. Motion Acquisition:
   • Real Dog Mocap: High-fidelity capture of fundamental gaits and transitions.
   • Video-to-Motion: Using LLMs (Qwen2.5VL) and video generators (Wan) to synthesize diverse instinctive/social behaviors, followed by animal pose estimation (SMAL).
   • Artist Design: Keyframe animations for stylized, intent-driven interactions (e.g., dancing, begging) that are hard to capture from real animals.
   • Teleoperation: Direct control of the robot to capture native, hardware-executable motions.
2. Robot-Executable Processing:
   • Kinematic Retargeting: Maps animal keypoints to the robot's morphology using Inverse Kinematics (IK) with regularization for smoothness.
   • RL Tracking Correction: A Reinforcement Learning (RL) policy tracks the retargeted trajectories in a physics simulator to enforce rigid-body dynamics, ensuring the motions are physically feasible and stable.
3. Triple-Layer Annotation: Every clip includes:
   • Fine-grained action labels.
   • Context-aware interaction narratives.
   • Executable natural-language commands.

B. Gen2Control RL Framework

To solve the "sim-to-real" gap and ensure semantic controllability, the authors propose Generation-to-Control Reinforcement Learning (Gen2Control RL). This is a joint co-optimization framework:

• Components:
  1. Text-to-Motion Generator: A diffusion-based model (initialized with MotionGPT3) that generates latent motion representations conditioned on text and current robot state.
  2. General Motion Controller: An RL policy (PPO) that tracks the generated reference motions on the robot.
• Training Loop:
  • The generator produces a reference motion sequence.
  • The controller attempts to track it in simulation.
  • Feedback Mechanism: The tracking success (reward) of the controller is used as a feedback signal to update the generator via policy gradients. This biases the generator to produce motions that are not only semantically correct but also dynamically executable.
• Deployment: The system runs end-to-end on an NVIDIA Jetson Orin, achieving real-time inference (<500ms latency).

3. Key Contributions

1. QuadFM Dataset: The first large-scale, ultra-high-fidelity motion-language dataset for quadrupeds, featuring 11,784 clips with triple-layer annotations, covering locomotion, interaction, and emotion.
2. Gen2Control RL Framework: A unified paradigm that jointly trains a text-conditioned generator and a low-level controller, ensuring semantic alignment with physical executability.
3. Real-World Validation: Successful deployment on a Unitree Go2 X robot with NVIDIA Orin, demonstrating real-time, language-driven motion generation with sub-500ms latency.

4. Experimental Results

Dataset Analysis

• Diversity: Compared to the baseline DogML, QuadFM offers significantly broader kinematic coverage. While DogML is concentrated in periodic gaits, QuadFM expands into dynamic, expressive manifolds (jumping, stretching, dancing).
• Scale: QuadFM contains 3.64M frames (normalized) and 11,784 unique atomic motions, vastly exceeding the diversity of existing resources.

Performance Evaluation

• Human Evaluation: In a blind test with 30 participants, Gen2Control RL significantly outperformed the baseline MotionGPT3 across all metrics:
  • Stability: 7.58 vs. 6.27
  • Text-Motion Alignment: 7.98 vs. 5.77
  • Smoothness: 7.46 vs. 5.62
  • Naturalness: 7.40 vs. 5.54
• Ablation Study:
  • Training solely on DogML resulted in poor generalization to QuadFM's validation set (high body position error).
  • Training on QuadFM alone achieved performance comparable to DogML-trained models on DogML data, proving high quality and generalization.
  • Synergy: Co-training with a small portion (25%) of DogML data improved performance on DogML benchmarks without sacrificing QuadFM performance, confirming the complementarity of the datasets.

Real-World Deployment

• The system achieved <500ms end-to-end latency from voice input to motion execution on the Unitree Go2 X.
• The robot successfully executed diverse commands (e.g., "Pee here," "Do push-ups," "Back Kick") with stable physical interaction.

5. Significance

This work represents a major step forward in Embodied AI for quadrupedal robots:

• Bridging the Gap: It moves beyond simple gait control to enable open-vocabulary, context-aware, and expressive interactions.
• Reliability: By coupling generation with control during training, it solves the common issue of generating motions that look good in simulation but fail on hardware due to dynamic constraints.
• Foundation for Future Research: QuadFM provides the necessary "foundation model" data to train next-generation quadruped agents capable of complex, human-like social and physical interactions, paving the way for more intuitive human-robot collaboration.