💻 computer science

CLAW: Composable Language-Annotated Whole-body Motion Generation

CLAW is an open-source, interactive web-based pipeline that generates scalable, physically feasible, language-annotated whole-body motion data for the Unitree G1 humanoid robot by composing kinematic planner blocks, simulating them in MuJoCo, and automatically generating diverse natural-language descriptions.

Original authors: Jianuo Cao, Yuxin Chen, Masayoshi Tomizuka

Published 2026-04-15

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianuo Cao, Yuxin Chen, Masayoshi Tomizuka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot dog how to dance, but instead of hiring a choreographer to film a human dancer and then painfully trying to copy those moves onto the robot, you just give the robot a set of Lego blocks.

That is essentially what CLAW does, but for a humanoid robot named the Unitree G1.

Here is the simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Human Video" Bottleneck

Usually, to teach a robot to move, scientists have to:

Hire actors and expensive cameras (Motion Capture).
Film them doing cool stuff.
Manually type out descriptions like "The robot walks forward, then punches."
Try to force the human moves onto the robot's body (which often looks glitchy, like a human trying to walk on stilts).

This is slow, expensive, and hard to scale.

2. The Solution: CLAW (The "Lego Builder")

CLAW is a new tool that lets you build robot movements like you are building a story with Lego bricks.

The Bricks (Motion Primitives): Instead of filming a whole dance, the system has 25 pre-made "bricks." These are simple moves like "Walk," "Run," "Crawl," "Box," or "Do the Zombie Dance."
The Builder (The Interface): You don't need to be a coder. You can use a Keyboard Mode (like playing a video game where you press 'W' to walk and 'A' to turn) or a Timeline Editor (like editing a video on your phone, dragging clips onto a timeline).
The Physics Engine (The Reality Check): When you snap these bricks together, the system doesn't just pretend the robot is moving. It runs a physics simulation (like a super-advanced video game engine). It checks: "If the robot tries to go from a slow walk to a punch, will it fall over?" If yes, it smooths it out so the robot stays balanced.

3. The Magic Feature: The "Auto-Writer"

This is the coolest part. Usually, if you make a video, you have to write the subtitles yourself.

CLAW has a built-in Auto-Writer.

As soon as you build a sequence (e.g., "Walk forward for 3 seconds, then punch"), the system instantly writes multiple different descriptions for it.
It can write it like a robot manual: "Move forward 3s."
It can write it like a news reporter: "The robot marches forward briskly."
It can write it like a storyteller: "The robot, looking determined, strides ahead."

It does this without a human typing a single word, and it does it perfectly every time because it knows exactly what the robot is doing.

4. Why This Matters (The "Infinite Library")

Think of this as an infinite library of robot training data.

Old way: You film 100 hours of humans dancing. It costs a fortune.
CLAW way: You sit at a computer, mix and match your "Lego bricks" (Walk + Run + Punch), and the computer generates 10,000 variations of that move in an hour. It creates a massive dataset of "Robot doing X" paired with "Text describing X."

The Bottom Line

CLAW is a web-based playground where you can mix and match robot moves, watch them happen in a realistic physics simulation, and get a perfect, written description of what happened—all automatically.

It turns the hard job of teaching robots to move into something as easy as dragging and dropping blocks on a screen, while the computer handles the heavy lifting of physics and writing the labels. This helps researchers train smarter, more adaptable robots much faster than ever before.

1. Problem Statement

Training language-conditioned whole-body controllers for humanoid robots (specifically the Unitree G1) requires large-scale datasets pairing motion trajectories with natural-language descriptions. Existing methods face three critical bottlenecks:

Data Scarcity & Cost: Traditional Motion Capture (MoCap) is expensive, logistically difficult, and limits the volume and behavioral diversity of the resulting corpus.
Kinematic Artifacts: Retargeting human MoCap data to robot morphologies often introduces physical infeasibilities (e.g., foot sliding, self-penetration, joint limit violations), degrading policy performance.
Annotation Burden: Manually pairing every recorded clip with natural-language descriptions creates a linear annotation cost that scales poorly with dataset size.
Lack of Control in Generative Models: While text-to-motion diffusion models exist, they operate in a purely kinematic regime. They lack guarantees of physical feasibility and make it difficult to independently control meta-attributes like speed, heading, pelvis height, and gait style, which are crucial for systematic data augmentation.

2. Methodology: The CLAW Pipeline

CLAW is an interactive, web-based pipeline designed to generate scalable, physically grounded, language-annotated motion data. It bridges the gap between low-level robot control and high-level data collection through a composable primitive approach.

A. Core Architecture

The system is a distributed, four-process architecture:

Browser Frontend: The user interface for interaction.
Protocol Bridge: A WebSocket–ZMQ interface handling communication and state aggregation.
C++ Controller: Hosts the kinematic planner and low-level whole-body controller.
MuJoCo Simulator: Executes physics simulation using the Unitree G1 model.

B. Composable Motion Primitives

Instead of generating motion from text, CLAW treats the motion modes of a kinematic planner (based on SONIC) as composable building blocks.

Parameters: Each primitive is parameterized by movement direction, heading, speed, pelvis height, and duration.
Modes: The system supports 25 distinct motion modes across four categories: Locomotion (e.g., Walk, Run), Squat/Ground (e.g., Crawl, Kneel), Boxing, and Styled Walking (e.g., Happy, Zombie).
Seamless Transitions: The planner uses the current robot state as the starting point for mode switching, allowing heterogeneous primitives (e.g., walking $\to$ squat $\to$ crawl) to be chained into continuous trajectories without explicit blending on the pipeline side.

C. Interactive Interfaces

CLAW provides two browser-based modes for data collection:

Keyboard Mode: Real-time, free-form exploration where operators use key bindings (WASD, Q/E, etc.) to control movement, speed, and mode selection. Ideal for rapid prototyping.
Sequence Editor Mode: A timeline-based interface for batch generation. Users drag-and-drop motion clips, configure parameters per segment, and execute reproducible "recipes." This allows for systematic variation of parameters (e.g., same path, different speeds) to expand the training corpus.

D. Automated Language Annotation

A deterministic, template-based annotation engine generates natural-language descriptions simultaneously with motion generation.

Input: It uses the same structured motion parameters (mode, direction, speed, duration) driving the planner.
Output: It produces diverse descriptions in eight stylistic registers (e.g., imperative, natural, narrative, concise) with or without explicit duration.
Mechanism: Diversity is achieved via synonym banks (e.g., run $\to$ sprint, dash, jog) and linear interpolation for speed adverbs, rather than stochastic LLM sampling. This ensures reproducibility (deterministic given a seed) while maintaining lexical variety.

3. Key Contributions

Composable Data-Generation Pipeline: A system that converts parameterized motion primitives into physically simulated, language-annotated trajectories for the Unitree G1. It features a planner-agnostic architecture that decouples data collection from the specific motion-generation backend.
Dual Interactive Interfaces: Keyboard and Sequence Editor modes enable both exploratory and reproducible batch data collection through a unified backend.
Deterministic Annotation Engine: A template-based system that generates multiple stylistic descriptions per trajectory, providing diverse yet consistent supervision for language-conditioned control without manual labeling.
Open-Source Modular System: A four-process architecture (Simulation, Controller, Bridge, Frontend) capable of generating training data at an arbitrary scale.

4. Key Results

Smooth Motion Stitching: The system successfully composes semantically distinct behaviors (e.g., walking into a squat, then crawling) into long-horizon trajectories. The planner handles transitions automatically, with most pairs showing natural continuity. (Note: Extreme posture changes, like crawling to upright boxing, can occasionally produce kinematic artifacts).
Automated Annotation: The engine successfully generates accurate, stage-wise natural-language descriptions for multi-segment sequences (e.g., 17 full-trajectory descriptions and 8 per-segment annotations for a single run) without human intervention.
Scalability: The pipeline scales with compute rather than physical recording sessions. Identical recipes can be replayed with varied parameters to systematically expand the dataset.
No Retargeting Required: Since the planner operates natively on the Unitree G1 skeleton, the resulting joint trajectories are directly in the robot's configuration space, eliminating the need for kinematic retargeting.

5. Significance

CLAW addresses the critical bottleneck of data scarcity in humanoid robot learning. By shifting from MoCap-based or Generative-Model-based approaches to a Physics-Simulation-based Compositional approach, it offers:

Physical Feasibility: All generated data is grounded in physics simulation, ensuring the robot can actually execute the motions.
Controlled Diversity: Researchers can systematically vary specific attributes (speed, height, gait) to create robust training distributions, which is difficult with black-box generative models.
Cost Efficiency: It removes the need for expensive MoCap sessions and manual annotation, enabling the creation of massive, high-quality language-motion datasets essential for training next-generation language-conditioned whole-body controllers.

The system is released as open source, facilitating the community's ability to scale up language-motion paired data for humanoid robotics.