🤖 AI

DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos

DreamDojo is a foundation world model trained on 44,000 hours of egocentric human videos that learns diverse dexterous interactions through continuous latent actions, enabling real-time simulation, precise action controllability, and effective policy planning for generalist robots in open-world, contact-rich environments.

Original authors: Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie

Published 2026-02-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, You Liang Tan, K. R. Zentner, George Kurian, Suneel Indupuru, Pooya Jannaty, Jinwei Gu, Jun Zhang, Jitendra Malik, Pieter Abbeel, Ming-Yu Liu, Yuke Zhu, Joel Jang, Linxi "Jim" Fan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to do everything a human can do: cook, clean, fix things, and play with toys. The problem is that robots are expensive, break easily, and we don't have enough hours of robot footage to teach them all these skills.

DreamDojo is a new "brain" for robots that solves this by learning from human videos instead of robot videos. Think of it as a robot that learns by watching millions of hours of people doing daily tasks on YouTube and TikTok, rather than needing a robot to do every single task itself.

Here is how it works, broken down into simple concepts:

1. The Massive Library (The Data)

Usually, robot training data is like a small library with only a few books on "how to pick up a red block." DreamDojo's library is a massive, infinite library containing 44,000 hours of human videos.

The Scale: It's like comparing a single notebook to the entire Library of Congress.
The Variety: It covers everything from cooking in a kitchen to fixing a car in a garage, involving thousands of different objects and skills.
The Secret Sauce: Since these videos don't have "robot instructions" attached to them, the team invented a special translator called Latent Actions. Imagine watching a video of someone opening a jar. Even though the robot can't see the exact muscle movements, this translator figures out the intent and the physics of "twisting and pulling" just by looking at how the jar and hand move. It turns the video into a universal instruction manual that any robot can understand.

2. The Simulator (The World Model)

Once the robot brain has learned from these videos, it becomes a World Model.

The Analogy: Think of a World Model like a flight simulator for a pilot. Before a pilot flies a real plane, they practice in a simulator. If they make a mistake in the simulator, no one gets hurt.
How DreamDojo Works: You can tell DreamDojo, "Imagine the robot reaches for the cup, but misses." The model then generates a video of what would happen next. It understands physics: if you push a cup, it slides; if you push it too hard, it falls off the table. It can simulate these outcomes instantly, even for objects or environments it has never seen before.

3. The Speed Boost (Distillation)

The original "brain" is very smart but slow, like a genius professor who takes 10 minutes to solve a math problem. For a robot to move in real-time, it needs to think as fast as a human.

The Solution: The team used a process called Distillation. They took the slow, genius professor and trained a fast, young student (the "Student Model") to mimic them.
The Result: The student can now predict the future at 10.81 frames per second. This is fast enough to run in real-time. You can control a virtual robot with a VR controller, and the robot will move and react instantly, just like a real person.

What Can It Do? (Based on the Paper's Claims)

The paper highlights three main ways this technology is used:

Testing Policies (The "Flight Simulator" for Robots):
Before sending a robot into the real world to pack fruit, you can test its "brain" in DreamDojo. The paper shows that if a robot strategy works well in the DreamDojo simulation, it almost certainly works well in the real world (99.5% correlation). This saves time and prevents robots from breaking things while learning.
Planning Ahead (The "Chess Player"):
When a robot has to do a complex task, it can use DreamDojo to "dream" about different outcomes. It can try out five different ways to grab an apple in its mind, see which one leads to the best result, and then execute that specific move. This makes the robot much smarter and more successful at difficult tasks.
Live Teleoperation (The "Virtual Twin"):
Because the model is so fast, a human can wear a VR headset and control a virtual robot in real-time. If the human moves their hand, the virtual robot moves instantly. This allows humans to "be" the robot remotely, which is useful for tasks that are too dangerous or difficult for humans to do directly.

Summary

DreamDojo is a bridge between the messy, diverse real world and the precise world of robots. By learning from the vast amount of human activity online and using a special "translator" to understand actions without needing labels, it creates a robot brain that can imagine the future, test ideas safely, and act in real-time. It turns the robot from a rigid machine that only knows what it was explicitly taught, into a flexible agent that understands how the world works.

Technical Summary: DreamDojo

Problem Statement

The development of generalist robot agents relies heavily on world models capable of simulating future states based on actions. While recent advances in video generation have enabled world models for discrete control domains (e.g., games, driving), they struggle with contact-rich, high-dimensional robot tasks. Existing approaches face three primary limitations:

Data Scarcity and Distribution: Robot data is limited by hardware costs and variability, often failing to cover the infinite variety of real-world environments and objects.
Lack of Stochasticity: Existing datasets predominantly consist of expert demonstrations, lacking the stochasticity in intentions necessary to learn robust action controllability.
Label Scarcity: Fine-grained action labels are scarce in large-scale video datasets. Naively training on passive videos ignores the causality between observations and actions, leading to poor knowledge transfer for action-conditioned simulation.

Consequently, current video world models are often confined to simulating observed setups and fail to respond to counterfactual actions or generalize to unseen objects and environments.

Methodology

DreamDojo is introduced as a foundation world model designed for open-world dexterous robot tasks. The approach consists of three distinct phases:

1. Large-Scale Human Video Pretraining

To overcome data limitations, the authors curate DreamDojo-HV, the largest egocentric human video dataset to date, comprising 44,711 hours of video. This dataset spans diverse daily scenarios (home, retail, transport, etc.) with approximately 6,015 unique skills and 1,135k unique objects.

Continuous Latent Actions: To address the lack of action labels, the authors introduce continuous latent actions as unified proxy actions. A Variational Autoencoder (VAE) with a spatiotemporal Transformer architecture extracts these actions from consecutive video frames in a self-supervised manner. This creates an information bottleneck that disentangles critical motion information from context, allowing for cross-embodiment knowledge transfer (from humans to robots).
Architecture: The model is built upon the Cosmos-Predict2.5 latent video diffusion model.
- Relative Actions: Instead of absolute joint poses, inputs are rebaselined to relative actions to reduce modeling complexity.
- Chunked Injection: Actions are injected as chunks (4 consecutive actions per latent frame) rather than as a global condition to mitigate causality confusion.
- Training Objective: The standard flow matching loss is augmented with a temporal consistency loss to better match ground-truth temporal transitions, enhancing object completeness and action following.

2. Post-Training on Target Robots

To adapt the foundation model to specific robot embodiments (e.g., Fourier GR-1, Unitree G1, AgiBot), a post-training stage is conducted on small-scale target robot datasets.

The action conditioning layer is reset and reinitialized to match the target action space.
The model is fully finetuned, leveraging the strong pretraining to achieve zero-shot generalization to unseen objects and novel environments despite limited target data.

3. Distillation for Real-Time Inference

To enable applications like live teleoperation and online planning, the foundation model is distilled into an autoregressive student model using the Self Forcing paradigm.

Architecture Change: The bidirectional attention mechanism is replaced with causal attention over a sliding window.
Process: A two-stage process (warmup and distillation) aligns the student's distribution with the teacher's ODE solutions, minimizing train-test distribution mismatch.
Result: The distilled model achieves 10.81 FPS at 640×480 resolution with only 4 denoising steps, while maintaining long-horizon consistency.

Key Contributions

DreamDojo-HV Dataset: A massive, diverse dataset of 44k+ hours of egocentric human videos, representing the largest corpus for world model pretraining, with significantly higher diversity in skills and scenes than prior robot datasets.
Foundation World Model: The first world model of its kind that utilizes large-scale human videos and continuous latent actions to achieve zero-shot generalization to unseen objects and environments for dexterous manipulation.
Distillation Pipeline: A method to convert the foundation model into a real-time, autoregressive predictor that maintains context consistency and supports long-horizon interactions without visual degradation.
Downstream Applications: Demonstration of the model's utility in policy evaluation, model-based planning, and live teleoperation.

Experimental Results

The paper evaluates DreamDojo on six out-of-distribution (OOD) benchmarks, including novel objects, environments, and counterfactual actions.

Action Conditioning: Latent action conditioning significantly outperforms action-free pretraining and approaches the performance of ideal settings with ground-truth action labels (e.g., PSNR improvements on EgoDex Eval).
Data Diversity: Increasing data diversity (adding In-lab, EgoDex, and DreamDojo-HV) consistently improves physics modeling and action controllability across all benchmarks.
Generalization: The 14B parameter model (DreamDojo-14B) outperforms the base Cosmos-Predict2.5 and the 2B variant in both physics correctness and action following in human preference evaluations.
Distillation Efficiency: The distilled student model runs 4× faster (10.81 FPS vs. 2.72 FPS) than the teacher while maintaining comparable performance in long-horizon rollouts (600 frames). It also demonstrates superior robustness to occlusions due to its multi-frame context.
Applications:
- Policy Evaluation: DreamDojo shows a strong linear correlation (Pearson $r=0.995$ ) between simulated and real-world success rates for fruit packing tasks.
- Model-Based Planning: Using DreamDojo for test-time planning improved success rates by up to 17% over the best single checkpoint and nearly 2× compared to uniform sampling.
- Live Teleoperation: The model enables real-time teleoperation of a virtual G1 robot using VR controllers.

Significance and Claims

The paper claims that DreamDojo represents a significant step toward general-purpose robot world models. By leveraging large-scale human videos and continuous latent actions, the method successfully bridges the embodiment gap, allowing robots to learn physics and dynamics from human interactions.

The authors emphasize that DreamDojo enables scalable policy evaluation without real-world deployment and facilitates online model-based planning and live teleoperation. The work suggests that simulating open-world, contact-rich tasks is feasible through foundation models trained on diverse human data, paving the way for more capable and adaptable generalist agents.

The paper modestly acknowledges limitations, noting that the model struggles with uncommon, fast actions (e.g., slapping) and that simulated success rates can sometimes be higher than real-world counterparts, indicating a need for better nuance in failure generation. Future work is suggested to explore broader action distributions and multi-view simulation.