🤖 AI

CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators

CLAP is a novel framework that enables zero-shot physical simulation across diverse robot and human embodiments by training on heterogeneous internet-scale videos, reconciling disparate action spaces through a curriculum-based learning recipe to achieve state-of-the-art performance in few-shot adaptation and generalizable physics understanding.

Original authors: Kechen Liu, Ola Shorinwa

Published 2026-08-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kechen Liu, Ola Shorinwa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled to understand the physical world in the way humans do. While a child learns that a cup will fall if pushed off a table or that a towel can be folded by pulling its corners, robots often require thousands of specific examples to learn even the simplest of these rules. This difficulty stems from the fact that most robotic learning systems are trained on data from a single type of machine. A robot with a long, thin arm learns physics differently than a robot with a gripper or a human hand, creating a barrier where knowledge gained by one cannot easily be shared with another. To overcome this, scientists have begun looking toward video generation models, which are computer programs capable of imagining what happens next in a scene. If a robot could watch a video and predict the future movement of objects, it could plan its actions without needing to physically try them first. However, these predictive models have historically been limited to single robot types, preventing them from learning from the vast, diverse library of human and robot videos available on the internet.

A team of researchers at Princeton University has developed a new framework called CLAP to bridge this gap. Their work demonstrates that a single video model can learn the universal laws of physics by training on a massive mix of videos featuring both humans and many different kinds of robots. The core idea is that while a human hand, a robotic arm, and a gripper all look different, they all obey the same physical rules when moving objects. By teaching a computer to predict the future of a scene regardless of who or what is performing the action, the researchers created a system that understands the underlying mechanics of the world rather than just the specific movements of one machine. This approach allows the model to learn from unlabeled internet videos, where no instructions are provided, as well as from detailed robot data, creating a much richer understanding of how things move and interact.

The researchers faced a significant hurdle because the data they wanted to use was messy and inconsistent. Videos of humans do not come with instructions on how their hands moved, and different robots use different ways to describe their movements. To solve this, the team created a method to translate all these different languages of motion into a common format. They used three main tools: the precise position of a robot's hand, simple text descriptions of the movement, and a hidden, learned representation of action that the computer figures out on its own. The most effective strategy involved a step-by-step learning process. First, the model learned the basic rules of physics by watching unlabeled videos using the hidden action representations. Once it understood the fundamental dynamics, the researchers then refined its knowledge using precise data from robots with labeled movements. This two-stage approach allowed the system to scale up using the vast amount of internet video while still remaining accurate enough to control real robots.

The results of this work show that the new system performs as well as, and often better than, the best existing models that were trained only on a single type of robot. In challenging test environments, the model could predict the future frames of a video with high accuracy, correctly simulating how objects would fall, slide, or be manipulated. Crucially, the system was able to generalize this knowledge to real-world tasks without needing to be retrained for every new robot. When tested on a single-arm robot, the system successfully helped a robot plan its actions to pick up various objects, such as tape, fish, or lobster, outperforming standard robotic policies. Even more impressively, the system could be applied to a two-armed robot and a humanoid robot with a different body structure, despite never being trained on data from those specific machines. By simply adjusting the final layer of the model to match the new robot's movements, the system retained its deep understanding of physics and continued to make accurate predictions.

This research suggests that the path to more capable robots does not require building a unique brain for every new machine. Instead, by training on a diverse mix of data, a single model can learn a general set of physical principles that apply to almost any agent. The researchers found that while relative movements, which describe change rather than position, worked well for text-based instructions, absolute positions were necessary for precise control with robot arms. They also discovered that while human videos were excellent for teaching the model the basics of physics, data from actual robots was essential for translating that knowledge into successful real-world actions. The work establishes a new way to train robots, moving away from isolated, single-purpose systems toward a more unified approach where learning from one embodiment can directly benefit another. While the system is not perfect and can sometimes make errors in its predictions, it represents a significant step forward in teaching machines to understand the physical world through observation and imagination.

Technical Summary: CLAP – Cross-Embodiment Video World Models as Zero-Shot Physical Simulators

Problem Statement

State-of-the-art action-conditioned video models in robotics are currently restricted to single robot embodiments. This limitation prevents them from leveraging the vast, heterogeneous corpus of internet-scale video data (including human demonstrations and diverse robot platforms) that contains rich signals for learning generalizable physical priors. The core challenge in bridging this gap is the heterogeneity of robot morphologies and action spaces, coupled with the fact that large-scale human video datasets lack action labels entirely. Existing approaches either fail to scale across different agents or require agent-specific architectures, thereby foreclosing the potential of cross-embodiment data scaling.

Methodology

The authors introduce CLAP, a framework designed to train cross-embodiment action-conditioned video models that learn universal physical laws governing spatiotemporal dynamics, regardless of the actor's morphology. The methodology addresses the fundamental challenge of reconciling disparate action spaces through three core technical components:

1. Harmonizing Heterogeneous Action Spaces

CLAP unifies diverse action representations into a shared framework using three distinct conditioning mechanisms:

End-Effector (EE) Actions: To handle labeled robot data, CLAP maps joint positions to a unified 7-DoF operational space (3 translational, 3 rotational, 1 gripper) via forward kinematics. Crucially, it utilizes absolute actions normalized to finite bounds $[-1, 1]$ for each robot platform, rather than relative actions, to preserve fine-grained dynamics prediction fidelity.
Language Actions: For natural language conditioning, CLAP projects numerical end-effector actions into text using concise templated motion primitives (e.g., "x=, y=, ..."). To mitigate tokenizer resolution limits, these actions are transformed into relative-action spaces, which improve information density for text-based models.
Latent Actions: To leverage unlabeled internet-scale video data (including human videos), CLAP employs a Latent Action Model (LAM). This model uses a spatiotemporal transformer to extract low-dimensional continuous action tokens ( $d=32$ ) from pairs of video frames $(f_t, f_{t+\Delta t})$ via a VAE objective, effectively learning pseudo-actions without ground-truth labels.

2. Curriculum-Based Cross-Embodiment Learning (CLAP-CURR)

Recognizing that no single action representation is sufficient (Latent actions lack zero-shot deployability; EE actions require labeled data), CLAP introduces a curriculum-based training recipe:

Pretraining: The video model backbone is first trained on massive amounts of unlabeled video data using latent actions to learn foundational physical priors.
Refinement: The latent action head is swapped for an end-effector action head. The pretrained backbone weights are retained, and the new head is jointly trained with the backbone on action-labeled data.
This approach allows the model to scale on unlabeled data while achieving zero-shot deployment capabilities in real-world action spaces without the fidelity degradation typically associated with downstream alignment layers.

3. Architecture and Conditioning

The video generation backbone utilizes a video diffusion paradigm (U-Net denoiser) conditioned on:

History: A horizon of past observations and robot poses to provide temporal velocity information.
Multi-view Prediction: The model is trained to generate future frames from multiple camera views simultaneously by concatenating views in the latent space, enforcing geometric consistency.

Key Contributions

Unified Action Representation: CLAP reconciles disparate action spaces across human and robot morphologies using end-effector poses, natural language, and learned latent actions.
Curriculum Learning Recipe: A novel training strategy that combines the scalability of latent actions (for unlabeled data) with the deployability of end-effector actions, enabling zero-shot transfer to real-world tasks.
Sample-Efficient Adaptation: A paradigm for training high-fidelity single-embodiment video models by few-shot adapting cross-embodiment models to target platforms (both pretrained and novel morphologies like bimanual robots and humanoids).

Experimental Results

The authors evaluate CLAP across diverse datasets (Open X-Embodiment, EgoDex, DROID, Bridge) and against state-of-the-art (SOTA) single-embodiment baselines.

Performance vs. SOTA: On the challenging DROID environment, CLAP approaches or surpasses SOTA single-embodiment baselines (e.g., Ctrl-World) despite having the same model parameters and architecture. On the Bridge environment, CLAP achieves high-accuracy predictions, though it shows minor fidelity degradation compared to a specialized Bridge baseline.
Action Space Analysis:
- Absolute vs. Relative: For end-effector conditioning, absolute action spaces yield superior prediction fidelity compared to relative spaces, as relative spaces suffer from compounding errors. Conversely, for language conditioning, relative action spaces perform better due to the limited resolution of discrete tokenization.
- Latent vs. Labeled: Latent action models (CLAP-LAM) achieve high prediction accuracy on mixed embodiment data (OXE-Mix), often outperforming end-effector models in pure prediction tasks, but require downstream alignment for deployment.
Zero-Shot Generalization: Through inference-time cross-policy planning, CLAP demonstrates zero-shot generalization to real-world manipulation tasks. When used as a world model to score action proposals from policies like $\pi0.5$ and MolmoAct-2, it improves success rates across tasks (e.g., picking tape, lobster, folding towels) by selecting trajectories with higher predicted rewards.
Few-Shot Adaptation: CLAP facilitates data-efficient adaptation to novel morphologies. Few-shot finetuning on bimanual YAM robots and G1 humanoids (with 14D and 26D action spaces, respectively) results in high-fidelity future predictions, outperforming models trained from scratch.

Significance and Claims

The paper claims that CLAP establishes a new paradigm for video world modeling, mirroring the breakthroughs seen in Large Language Models (LLMs) through web-scale training.

Universal Physical Priors: The work demonstrates that universal physical laws can be learned across heterogeneous agents, challenging the notion that agent-specific architectures are necessary for high-fidelity simulation.
Scalability: By utilizing latent actions, CLAP unlocks the potential of internet-scale unlabeled video data for robotics, a resource previously inaccessible to action-conditioned models.
Foundation for Deployment: The curriculum-based approach provides a pathway to deploy these models in the real world without the fidelity loss typically incurred by alignment layers.
Efficiency: The framework proves that cross-embodiment models can serve as superior foundations for single-embodiment tasks, reducing the need for training from scratch or using less-aligned backbones.

The authors conclude that while CLAP achieves high prediction fidelity, it remains prone to hallucinations, and future work should focus on uncertainty quantification and expanding training data to include more diverse bimanual and humanoid demonstrations.