💻 computer science

MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulation

MotionWAM is a real-time foundation world action model that unifies whole-body loco-manipulation control for humanoids by leveraging intermediate video denoising features to predict coordinated motion tokens, thereby overcoming the speed and consistency limitations of traditional hierarchical policies and significantly outperforming Vision-Language-Action baselines on complex tasks.

Original authors: Jia Zheng, Teli Ma, Yudong Fan, Zifan Wang, Shuo Yang, Junwei Liang

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jia Zheng, Teli Ma, Yudong Fan, Zifan Wang, Shuo Yang, Junwei Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to be a human-like helper. Usually, we teach robots in two separate steps: first, we teach their "brain" (the upper body) how to grab things, and then we teach their "legs" (the lower body) how to walk and stay balanced. The problem is that these two parts don't talk to each other well. The brain says, "Grab that cup," and the legs just say, "Okay, I'll walk forward to keep you from falling." They can't do things like kick a ball or step on a pedal because the legs are only allowed to do "balance stuff."

MotionWAM is a new robot brain that fixes this by treating the whole body as one single, coordinated team. Here is how it works, using some simple analogies:

1. The "Movie Director" vs. The "Scriptwriter"

Most robot brains are like scriptwriters. They look at a picture and a sentence (like "pick up the box") and try to guess the next move based on what they've seen before. They don't really understand how physics works or how things move over time.

MotionWAM is different. It's like a Movie Director who has watched thousands of hours of movies. Before it tells the robot what to do, it runs a quick "mental simulation" of what the next few seconds of the movie will look like.

The Trick: Instead of waiting for the whole movie scene to be fully drawn (which takes too long), MotionWAM looks at the sketch of the scene while it's being drawn. It grabs the "vibe" of the future movement from that sketch and immediately tells the robot how to move. This is why it can think fast enough to run in real-time.

2. The "Unified Remote Control"

In old systems, the robot had two remotes: one for the arms and one for the legs. The leg remote was very simple and only knew how to walk straight or turn.

MotionWAM's Innovation: It uses one single remote control for the whole body. When the robot needs to kick a soccer ball, the "kick" command isn't just for the leg; it's a single instruction that tells the arms to balance, the torso to lean, and the foot to swing all at once. This allows the robot to do things like step on a pedal or kick a ball, which the old "two-remote" systems couldn't do because they didn't understand that the legs could be part of the task, not just the balance.

3. The "Three-Stage Training Camp"

Teaching a robot this way is hard, so the researchers used a three-step training camp:

Stage 1 (The Movie Buff): They showed the robot thousands of hours of videos from a human's point of view (like a GoPro on a head). The robot didn't learn to move yet; it just learned how the world looks when you move through it. It learned the "physics of vision."
Stage 2 (The Translator): They taught the robot how to translate those movie skills into actual robot movements. They used data from different types of robot bodies to make sure the robot understood the general idea of "moving," not just one specific body shape.
Stage 3 (The Special Forces): Finally, they gave the robot specific practice tasks (like loading a cart or wiping a board) using a human operator guiding it with VR goggles. This is where the robot learned to apply its movie knowledge to real, tricky jobs.

The Results

When they tested MotionWAM on nine difficult tasks (like kicking a soccer ball, loading a cart, or doing laundry), it was a huge success:

Speed: It thinks fast enough to run in real-time (about 5 times a second), which is necessary for a robot to stay upright without falling over.
Success Rate: It succeeded in 76% of the tasks, while the best previous robot brains only succeeded about 44% of the time.
The "Kick" Factor: The biggest win was on tasks requiring foot interaction. The old robots would try to walk around the ball; MotionWAM actually kicked it.

The Bottom Line

MotionWAM proves that if you give a robot a "movie director" brain that understands how the world moves, and you let it control its whole body with one unified plan, it can do complex, human-like jobs much better than robots that treat walking and grabbing as separate problems.

Limitations: The paper notes that this was tested on one specific robot model (the Unitree G1) and that if the robot loses sight of the object it's holding (because the camera is on its head), it might get confused and stop working. It hasn't been tested on other robot bodies yet.

Technical Summary: MotionWAM

Problem Statement

Current autonomous humanoid loco-manipulation systems predominantly rely on a hierarchical paradigm that decouples the upper and lower bodies. In this setup, a high-level manipulation policy generates fine-grained joint targets for the arms, while a low-level controller tracks coarse base commands (velocity, height, orientation) for the legs. This architecture forces the robot into inconsistent action spaces, restricting the legs to passive balance preservation and preventing task-driven foot interaction (e.g., stepping on a pedal or kicking a ball).

While World Action Models (WAMs) have shown promise in tabletop manipulation by coupling video dynamics priors to policies, existing implementations are too slow for real-time control. They rely on iterative denoising over high-dimensional video-action latents, which creates a computational bottleneck that prevents closed-loop deployment on dynamic, balancing humanoids. The core challenge is to deploy the rich dynamics priors of WAMs in a unified latent space for whole-body humanoid control while maintaining real-time inference speeds.

Methodology: MotionWAM

MotionWAM is an end-to-end World Action Model designed to drive autonomous humanoid loco-manipulation from a single egocentric camera. It replaces the hierarchical upper-lower split with a unified motion latent that predicts whole-body motion tokens covering locomotion, torso movement, height regulation, foot interaction, and hand manipulation simultaneously.

1. Model Architecture

MotionWAM employs a dual-DiT (Diffusion Transformer) framework:

Video DiT: Initialized from Cosmos-Predict2.5-2B, this branch compresses egocentric conditioning frames and future frames into latents. Crucially, instead of waiting for full denoising, the model intercepts the hidden states ( $h^{\tau_f}_t$ ) of the velocity network at a fixed flow timestep near pure noise ( $\tau_f \approx 1$ ). This "one-shot imagination" regime encodes where the scene is heading without the computational cost of iterative denoising.
Motion DiT: This branch consumes the intermediate Video DiT features, along with proprioceptive states and noisy motion tokens, to predict the velocity field for the motion latent.
Unified Latent ( $m_t$ ): The output is a unified motion latent comprising:
- A continuous vector ( $m^{cont}_t$ ) for dexterous hand/gripper commands.
- A discretized token index ( $k_t$ ) derived from a scalar slot ( $\tilde{k}_t$ ), which summarizes locomotion, torso, height, and foot interaction intent. This token is decoded by SONIC, a universal whole-body controller, into joint commands.

2. Three-Stage Training Framework

To adapt the video world model to egocentric dynamics and the specific humanoid embodiment, the authors propose a progressive training recipe:

Stage 1: Egocentric Video Pretraining: The Video DiT is trained alone on ~2,136 hours of egocentric human and humanoid-robot videos. This shifts the world model's distribution toward first-person visual dynamics without being throttled by the scarcity of action-labeled data.
Stage 2: Cross-Embodiment Action Post-Training: The Motion DiT is attached and co-trained on a heterogeneous mixture of Unitree G1 data (varying end-effectors and annotation formats). The video objective is retained as a regularizer to prevent the dynamics prior from being overwritten, grounding the action signal across different embodiments.
Stage 3: Whole-Body Fine-Tuning: The full model is fine-tuned on teleoperated whole-body demonstrations collected on the target Unitree G1. The action output is switched to unified whole-body motion tokens, enabling end-to-end control.

Key Contributions

MotionWAM Architecture: A real-time WAM that conditions policy on intermediate denoising features of a video world model, enabling whole-body control in a single action space.
Unified Motion Latents: The replacement of decoupled upper-lower policies with unified motion tokens allows the lower body to participate actively in tasks (e.g., kicking, stepping) rather than just stabilizing the base.
Three-Stage Learning: A framework that progressively adapts a video world model from generic egocentric dynamics to specific embodiment action grounding, then to task-specific fine-tuning.
Real-Time Whole-Body Control: The first closed-loop, end-to-end WAM-driven policy capable of real-time humanoid loco-manipulation, including task-driven foot behaviors.

Experimental Results

The authors evaluated MotionWAM on nine real-world Unitree G1 tasks requiring waist control, height regulation, squatting locomotion, foot interaction, and body-hand coordination.

Performance vs. Baselines: MotionWAM was compared against five baselines, including state-of-the-art Vision-Language-Action (VLA) models (GR00T-N1.7, $\pi_0.5$ ) and custom ablations. MotionWAM achieved an overall success rate of 76.1%, substantially outperforming the strongest baseline (76.1% vs. 43.9% for GR00T-N1.7), representing a >30% absolute gain.
Task-Specific Gains: The performance gap was most significant in tasks requiring whole-body coordination, such as Kick Soccer (+40%), Load Cart (+40%), and Wipe Board (+45%). VLA-only baselines collapsed on locomotion-heavy tasks, indicating that semantic priors alone are insufficient for closed-loop physics control.
Ablation Studies: Removing Stage 1 (egocentric pretraining) or Stage 2 (cross-embodiment grounding) resulted in significant performance drops (11% and 28% respectively), validating the necessity of the progressive training strategy.
Real-Time Inference: MotionWAM runs at 4.9 Hz on an NVIDIA A100. This is significantly faster than other world-model-based policies (e.g., Cosmos Policy at 0.7 Hz) because it avoids iterative denoising, making it suitable for closed-loop humanoid balance control.

Significance and Claims

The paper claims that MotionWAM demonstrates that video-pretrained WAMs can be extended from tabletop manipulation to coordinated, human-like whole-body humanoid control. By leveraging intermediate denoising features, the model successfully bridges the gap between the rich dynamics priors of large-scale video models and the real-time, safety-critical requirements of balancing humanoids. The work suggests that a unified action space, conditioned on video world dynamics, is a viable path toward autonomous robots that can actively use their entire bodies (including feet) to interact with the environment, rather than being restricted to upper-body manipulation.

Limitations

The authors acknowledge that the Stage 3 fine-tuning has only been validated on the Unitree G1 embodiment, and the transferability of the recipe to other hardware platforms remains unverified. Additionally, the evaluation did not include controlled studies on novel object generalization, as training and test objects shared visual similarities. The system also relies on a single egocentric camera, leading to failure modes when manipulated objects leave the field of view or the viewpoint drifts significantly from the training distribution.