💻 computer science

AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization

The paper introduces AnyWorld, a factorized world modeling framework that synthesizes diverse, cross-embodiment robot experiences from single human egocentric videos by decoupling action, camera, and embodiment factors, thereby enabling controllable data generation that significantly improves manipulation policies on both simulated and real humanoid robots.

Original authors: Cheng Chen, Jerry Bai, Jiacheng Wei, Boyu Chen, Xiaoji Zheng, Fan Wu, Minghao Yang, Tianrun Chen, Ruibo Li, Xiaoyu Yue, Xiaoyang Guo, Yixiao Ge, Guosheng Lin, Fayao Liu

Published 2026-09-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Cheng Chen, Jerry Bai, Jiacheng Wei, Boyu Chen, Xiaoji Zheng, Fan Wu, Minghao Yang, Tianrun Chen, Ruibo Li, Xiaoyu Yue, Xiaoyang Guo, Yixiao Ge, Guosheng Lin, Fayao Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots are learning to do more than just move in straight lines; they are learning to manipulate the world around them, picking up objects, opening doors, and assembling parts. To learn these skills, a robot needs experience. It must see thousands of examples of how a hand reaches for a cup, how a gripper squeezes a sponge, and how a tool slides across a table. For a long time, the only way to get this experience was to have a robot do the task itself, over and over again. This is slow, expensive, and limited by how many robots a lab can afford to build and how many hours they can run without breaking.

A promising alternative has emerged from an unexpected source: human videos. Every day, people record themselves cooking, cleaning, and building things. These videos are full of rich, physical interactions that robots could learn from. However, there is a major problem. A video of a human cooking shows a human hand, a human body, and a human point of view. A robot does not have a human body, and it does not see the world from a human head. If a robot tries to copy a human video directly, it might fail because the human's arm is longer, the robot's camera is in a different spot, or the robot's gripper works differently. The challenge for scientists is to figure out how to take the useful "story" of what is happening in a human video and retell it in a way that makes sense for a robot.

A team of researchers has developed a new system called AnyWorld to solve this problem. Instead of trying to copy a human video exactly, the system breaks the video down into three separate ingredients: the action being performed, the way the camera moves, and the body and scene doing the work. Think of the action as the script of what is happening, the camera movement as the direction of the shot, and the body and scene as the actors and the set. By separating these elements, the researchers can take a single video of a human doing a task and recombine those ingredients to create a brand-new video. In this new video, the "actor" is a robot, the "set" is a robot's environment, and the "camera" is a robot's eye, but the "script" of the action remains the same.

The researchers trained their system using a massive collection of human videos where people were interacting with objects. They taught the model to understand the difference between the motion of the task and the specific body performing it. Once the model learned this, they tested it by feeding it a human video and asking it to generate a version of that same task performed by a robot. They did not need any videos that showed a human and a robot doing the same task side-by-side. The system simply took the human's movements and the camera's path, and then swapped in a robot body and a robot scene. The result was a video that looked like a robot performing the task, complete with the correct perspective and physical constraints of that specific machine.

The team tested this ability to change the body, the camera angle, and the scene. They showed that the system could take a human video and generate a version where a robot named RoboCasa GR1 performed the task, and another version where a different robot named IRON performed it. They also showed that they could keep the robot and the task the same but change the camera angle, creating a view from a different perspective. Crucially, the system preserved the logic of the interaction. If a human in the original video picked up a cup and moved it to a shelf, the generated robot video showed the robot doing the exact same sequence of movements, just adapted to its own body shape and camera position.

To prove that these generated videos were actually useful, the researchers used them to train real robots. They took a standard robot learning system and gave it extra training using the videos created by AnyWorld. They tested this on a simulation of a robot arm and on a real, physical humanoid robot. In the simulation, the robot's success rate at picking up and placing objects improved significantly after training on the generated data. On the real robot, the improvement was even more dramatic. A robot that was only able to grasp a banana successfully in 20 percent of attempts jumped to a 55 percent success rate after being trained on the human-to-robot videos. This showed that the system was not just making pretty pictures; it was creating valid training data that helped the robot learn better.

The researchers also investigated exactly why this worked. They wanted to know if simply telling the robot what action to take was enough, or if seeing the visual scene was also necessary. They ran a test where they gave the robot the correct action commands but kept the visual training data from the original, unmodified robot videos. This approach failed to teach the robot how to follow specific instructions, such as choosing the left banana instead of the right one. However, when they used the full system to generate both the new visual scene and the correct action, the robot learned to follow the instructions reliably. This proved that the visual recomposition was essential. The robot needed to see the world from its own perspective to understand how to apply the action.

The study suggests that this method of breaking down interactions into separate factors allows a single human experience to be reused many times. A human video that was once limited to a human body and a human camera can become a source of training data for many different types of robots, in many different environments. The researchers noted that while the system is powerful, it has limits. It cannot yet capture the feeling of touch or the exact force needed to squeeze a soft object, as these require physical sensors that the video generation does not provide. Additionally, the system relies on being able to track the human movements clearly; if the video is too shaky or the person is hidden behind an object, the system struggles.

Despite these limitations, the work offers a new path forward for robot learning. It suggests that the vast library of human videos on the internet, which has been difficult to use for robots because of the differences in bodies and viewpoints, can now be tapped. By using a model that understands how to separate the action from the actor, scientists can turn human experiences into robot experiences without needing to record every single task with every single robot. This could allow robots to learn complex skills much faster, using the abundant data of human daily life as a foundation for their own capabilities.

Technical Summary: AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization

1. Problem Definition

The central bottleneck in generalizable robot manipulation is the scarcity of large-scale, contact-rich robot experiences. While modern robot learning benefits from large visual-action datasets, collecting such data on physical robots is expensive, hardware-dependent, and difficult to scale across diverse scenes and embodiments. Although human egocentric videos offer abundant physical interactions, they are tied to specific bodies, camera trajectories, and environments, representing only a narrow slice of experience.

The core challenge addressed by this work is zero-shot experience recomposition: converting a single human egocentric interaction video into diverse, robot-native visual experiences without requiring paired human-robot demonstrations. The goal is to treat recorded human interactions as reusable seeds that can be expanded into multiple robot-domain rollouts by varying the acting body, viewpoint, and scene context while preserving the underlying interaction dynamics.

2. Methodology: AnyWorld

AnyWorld is a cross-embodiment world modeling framework that factorizes an interaction into three independent components: Action, Camera, and Embodiment.

2.1 Factorization Strategy

The model decomposes the generation process into three controllable factors:

Action Control ( $A_{1:T}$ ): Represented as a rendered skeleton control video in pixel space. This specifies where motion occurs in the image plane, providing an embodiment-agnostic interface that is easier to share across human and robot bodies than joint commands or specific action vectors.
Camera Control ( $C_{1:T}$ ): Constructed from camera intrinsics and extrinsics, converted into Plücker ray embeddings. This allows the model to separate actor motion from camera motion, enabling viewpoint evolution independent of the action.
Embodiment Context: Specified by an initial frame ( $\tilde{I}^e_0$ ) and a text tag ( $\tau_e$ ). The initial frame defines the target body appearance, scene layout, object configuration, and interaction geometry at the start of the rollout. The text tag identifies the specific embodiment (e.g., human, RoboCasa GR1, IRON).

2.2 Model Architecture

AnyWorld is implemented as a latent diffusion video model (based on DiT blocks).

Input Conditioning:
- The action control video is encoded and concatenated with the noisy video latent along the channel dimension.
- The camera control is passed through a lightweight adapter and added to the patch embedding as an additive geometric signal.
- The embodiment tag and caption are encoded as text embeddings and injected into transformer blocks via cross-attention.
Training Regime:
1. Stage 1 (Pretraining): The model is pretrained on large-scale egocentric human interaction data (EgoDex) to learn strong action-camera conditioned priors.
2. Stage 2 (Fine-tuning): The model undergoes mixed-embodiment fine-tuning using unpaired data from humans, RoboCasa GR1, and IRON robots. Crucially, no paired human-robot clips are used; the model learns to associate different embodiment appearances with compatible interaction dynamics through the shared factorized interface.

2.3 Inference and Recomposition

At inference, the model supports zero-shot recomposition. Given a source human video, the action and camera controls are extracted. The initial frame is edited to specify a new target visual context (changing the body, scene, or both), and the embodiment tag is updated. The model then generates a target-domain rollout ( $\hat{V}^e_{1:T}$ ) that preserves the original interaction structure but adapts to the new embodiment and viewpoint.

For downstream policy training, the system performs Joint Visual–Action Experience Recomposition. It not only re-embodies the visual observation but also uses an action-calibration module to map human wrist trajectories into the target robot's relative end-effector action space, creating paired $(\text{observation}, \text{action})$ data.

3. Key Contributions

Formulation: The paper formulates cross-embodiment experience recomposition as a world-modeling problem, enabling the expansion of a single human interaction into multiple robot-native rollouts without paired demonstrations.
Model Architecture: Introduction of AnyWorld, a factorized video diffusion model that combines action control, camera control, and target embodiment-context conditioning for controllable rollout generation.
Empirical Validation: Demonstration that generated robot-domain rollouts improve downstream Vision-Language-Action (VLA) adaptation in both simulation (RoboCasa GR1) and real-world (IRON) settings. The paper further uses controlled interventions to show that the generated data can correct specific policy failures (e.g., spurious completion priors) and transfer language-grounded spatial target selection.

4. Experimental Results

The authors evaluate AnyWorld from two perspectives: world-modeling controllability and downstream VLA adaptation.

4.1 Controllability and Quality

Controllability: On a benchmark of 60 videos, AnyWorld outperforms baselines (Cosmos-Predict2.5 and WAN Fun-Control) in ActionAlign, CameraAlign, and EmbodAcc (embodiment accuracy). The factorized approach achieves an average score of 0.778, significantly higher than the baselines, demonstrating superior control over action, viewpoint, and body.
Video Quality: Using VBench metrics, AnyWorld maintains strong subject consistency, background consistency, and motion smoothness, achieving an average score of 0.971, which is competitive with or slightly better than baselines.

4.2 Downstream VLA Adaptation

Simulation (RoboCasa GR1): Adding AnyWorld-generated data to the UniT baseline improved success rates on 18 pick-and-place tasks from 49.8% to 54.6%.
Real Robot (IRON): On 20 banana-grasping trials with varying backgrounds, the success rate improved from 20.0% to 55.0%.
Ablation on Data Ratio: A 2:1 mixture of human-to-robot data during fine-tuning yielded the best balance between action diversity and robot-domain grounding.

4.3 Targeted Capability Transfer

The paper demonstrates that recomposed data can target specific policy gaps:

State-Conditional Repair: The baseline policy stalled in "partial-completion" states (one object already placed). Re-scening human demonstrations into these missing states and pairing them with calibrated actions allowed the policy to resume and complete the task.
Counterfactual Grounding: To test language-grounded spatial selection, the authors created dual-target observations with opposite instructions. An "action-only" intervention (changing actions but keeping original one-sided observations) failed to learn stable instruction following. Only joint visual-action recomposition (changing both the visual state and the action) enabled the policy to reliably switch targets based on language.

5. Significance and Claims

The paper claims that AnyWorld serves as a scalable experience engine for robot learning. Its primary significance lies in transforming abundant, unpaired human interaction videos into useful, robot-native training data without the need for costly paired human-robot demonstrations.

The authors emphasize that:

Visual Recomposition is Necessary: The failure of action-only counterfactual interventions proves that simply relabeling actions is insufficient; the robot must be grounded in a robot-native visual state that matches the action and instruction.
Generalization: The framework allows a single model to support controllable recomposition across bodies, viewpoints, and scenes, effectively turning human videos into reusable seeds for diverse robot experiences.
Practical Impact: The generated data directly improves the robustness and capability of real-world robot policies, addressing the data scarcity bottleneck in contact-rich manipulation.

6. Limitations

The authors acknowledge several limitations:

Tactile Feedback: Visual rollouts cannot fully capture tactile feedback or contact forces required for precise control.
Extraction Reliability: The method depends on reliable action-camera extraction; severe occlusion, fast motion, tracking errors, or camera shake may reduce controllability.
Scope: Current evaluation covers only three embodiments (Human, GR1, IRON), leaving broader generalization to other morphologies and long-horizon tasks for future work.