Commanding Humanoid by Free-form Language: A Large Language Action Model with Unified Motion Vocabulary

1. Problem Statement

The core challenge addressed is enabling humanoid robots to execute whole-body actions based on free-form natural language commands. While Large Language Models (LLMs) have advanced reasoning and Vision-Language-Action (VLA) models have succeeded in robotic manipulation (e.g., gripper-based tasks), applying these to humanoids remains difficult due to:

• High Degrees of Freedom (DoF) & Complex Dynamics: Humanoids require precise control of balance and contact, unlike simple manipulators.
• Data Scarcity: There is a lack of large-scale, physically executable humanoid motion datasets compared to human motion datasets.
• The "Retargeting Gap": Existing methods often map text to human motion and then "retarget" it to robots. This introduces kinematic mismatches and projection errors, leading to physically implausible or unstable robot behaviors.
• Trade-offs: Current approaches struggle to balance language generalization (understanding diverse instructions) with physical fidelity (ensuring the robot doesn't fall or violate dynamics).

2. Methodology: Humanoid-LLA

The authors propose Humanoid-LLA, an end-to-end framework that maps expressive language directly to executable humanoid actions. The framework consists of three tightly integrated components:

A. Unified Human-Humanoid Motion Vocabulary

To bridge the gap between human data and robot execution, the authors construct a shared discrete motion space.

• Canonicalization: Human motions (SMPL) and humanoid motions (e.g., Unitree G1) are mapped to comparable high-dimensional representations (263D and 227D respectively).
• Cross-Embodiment Tokenization: They employ a VQ-VAE with implicit partitioning. Unlike standard tokenizers, this model enforces bidirectional cross-modal reconstruction.
  • Mechanism: A token generated from a human motion must decode to the same motion primitive when applied to a humanoid, and vice versa.
  • Result: A compact, shared vocabulary where a single discrete token represents a consistent motion primitive across both embodiments, enabling the LLM to reason over a unified latent space.

B. Vocabulary-Directed Action Distillation

To ensure the discrete tokens generated by the LLM result in physically stable robot behavior, a distillation process is used.

• Teacher Policy: A "privileged" tracking controller is trained in simulation using Reinforcement Learning (PPO) to follow dense, retargeted humanoid reference motions with high physical fidelity.
• Student Policy: A Vocabulary-Directed Student Controller (modeled as a Conditional VAE) is distilled from the teacher. Instead of tracking dense trajectories, this student policy takes motion tokens (from the unified vocabulary) as input and generates full-body actions.
• Benefit: This shifts the control paradigm from continuous trajectory tracking to a discrete token sequence, allowing the robot to execute actions selected within the vocabulary space while maintaining dynamic robustness and contact stability.

C. Large Language Action Model (LLA) Training

The core LLM is trained in two stages to map text to these motion tokens:

1. Supervised Fine-Tuning (SFT) with Chain-of-Thought (CoT):
   • The model is trained on large-scale text-motion datasets (AMASS).
   • Augmentation: A Vision-Language Model (VLM) generates Motion Chain-of-Thought annotations. Instead of just "walk," the model learns to decompose tasks (e.g., "1. Lift leg, 2. Swing forward, 3. Place foot").
   • The LLM learns to output a reasoning trace followed by a sequence of motion tokens.
2. Reinforcement Learning Fine-Tuning (RLFT) with Humanoid Feedback:
   • The model is further optimized using Group Relative Policy Optimization (GRPO).
   • Reward Design: A composite reward function ensures:
     • Format Compliance: Correct output structure.
     • Distributional Reward: The generated motion distribution matches physically plausible trajectories and aligns semantically with the text.
     • Tracking Reward: The actual rollout of the generated tokens by the vocabulary-directed controller in simulation is evaluated for position and acceleration errors.
   • This closed-loop training injects physical priors directly into the token generation process.

3. Key Contributions

1. Humanoid-LLA Framework: The first end-to-end system enabling free-form text-to-humanoid whole-body control, bypassing the need for intermediate kinematic retargeting.
2. Unified Motion Vocabulary: A novel cross-embodiment VQ-VAE tokenizer that creates a shared latent space for humans and robots, mitigating data scarcity and embodiment mismatch.
3. Two-Stage Fine-Tuning: A strategy combining SFT with Motion CoT for semantic reasoning and RLFT with physical feedback for dynamic feasibility, achieving robust generalization without sacrificing physical stability.

4. Experimental Results

The method was evaluated in simulation and on real-world Unitree G1 and Booster T1 humanoids.

• Quantitative Performance:
  • Generation Quality: Humanoid-LLA achieved a significantly lower FID (2.626 vs. 8.682 for the next best) and higher R-Precision (0.447 vs. 0.295), indicating better motion diversity and text-motion alignment.
  • Physical Fidelity: It achieved an 87.6% success rate in execution, outperforming baselines like RLPF (80.0%) and LangWBC (76.0%). It also showed the lowest Mean Per-Joint Position Error (MPJPE) of 56.43mm.
• Ablation Studies:
  • Removing CoT reasoning drastically reduced performance, proving the importance of structured task decomposition.
  • Removing RLFT lowered physical success rates, confirming the necessity of simulation-based physical feedback.
• Real-World Demonstrations: The robot successfully executed complex, unseen instructions such as "Walk in a curving figure-eight," "Shake hands," and "Execute a military parade march," demonstrating strong zero-shot generalization.

5. Significance

This work represents a significant step toward general-purpose embodied intelligence. By unifying human and robot motion representations and grounding LLM reasoning in physical dynamics, Humanoid-LLA overcomes the traditional trade-off between language flexibility and physical safety. It provides a scalable pathway for deploying humanoid robots in unstructured environments where they must interpret and execute complex, open-vocabulary human commands without extensive task-specific programming.