🤖 AI

World Models for Learning Dexterous Hand-Object Interactions from Human Videos

The paper introduces DexWM, a world model that learns dexterous hand-object interactions from over 900 hours of human videos by using finger keypoints and a hand consistency loss, achieving superior future-state prediction and zero-shot transfer to unseen robotic tasks compared to prior methods.

Original authors: Raktim Gautam Goswami, Amir Bar, David Fan, Tsung-Yen Yang, Gaoyue Zhou, Prashanth Krishnamurthy, Michael Rabbat, Farshad Khorrami, Yann LeCun

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Raktim Gautam Goswami, Amir Bar, David Fan, Tsung-Yen Yang, Gaoyue Zhou, Prashanth Krishnamurthy, Michael Rabbat, Farshad Khorrami, Yann LeCun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to pick up a delicate egg, pour a glass of water, or tie a shoe. The biggest problem isn't just telling the robot what to do; it's teaching it how the world reacts when it moves its fingers. If you squeeze too hard, the egg breaks. If you move too slowly, the water spills.

This is the challenge of dexterous manipulation: getting a robot to use its fingers with the same subtle, human-like grace we use every day.

The paper introduces a new AI system called DexWM (Dexterous World Model) that solves this by acting like a "crystal ball" for robots. Here is how it works, broken down into simple concepts:

1. The Problem: Robots Don't Have "Intuition"

Most robots today are like clumsy toddlers. They can push a block, but they don't understand that if they push a cup, it might tip over. They usually learn by trying things out millions of times in a simulation, which is slow and expensive. They also struggle to learn from human videos because human hands are complex, and robots often have different "bodies" (embodiments).

2. The Solution: A "Mental Simulator"

Think of DexWM as a robot's internal video game engine. Instead of just reacting to what it sees right now, it constantly runs a simulation in its head: "If I move my finger this way, what will the object do next?"

The Training: Instead of forcing the robot to learn by trial and error (which takes forever), the researchers fed DexWM 900 hours of human videos. They watched people cooking, playing with toys, and handling objects.
The "Secret Sauce": The AI didn't just watch the whole video; it focused specifically on the fingertips. It learned to track exactly where every finger was and how the object moved in response. It's like a coach who doesn't just watch the whole soccer game but zooms in on the player's footwork to understand how the ball moves.

3. How It Learns: The "Hand Consistency" Trick

The researchers found a tricky problem: If you just ask an AI to predict the next video frame, it might get the background right (the table, the wall) but mess up the tiny details of the hand. It might predict a hand holding a cup, but the fingers look like they are floating or passing through the cup.

To fix this, they added a special rule called "Hand Consistency Loss."

Analogy: Imagine you are drawing a picture of a hand holding a ball. If you get the ball right but draw the fingers in the wrong place, you get a red "F" on your homework.
The Result: This rule forced the AI to be obsessed with getting the finger positions perfect. It learned that for the world to make sense, the hand must look physically correct.

4. The Magic: Zero-Shot Transfer

This is the most impressive part. The robot was trained on human videos, but then it was asked to do tasks with a robot arm it had never seen before.

The Analogy: Imagine you learned to drive a car by watching thousands of hours of human drivers on TV. Then, you get into a completely different car (maybe a truck or a futuristic vehicle) and you can drive it perfectly without ever having sat in that specific car before.
The Reality: The researchers took DexWM, which learned from humans, and gave it a tiny bit of "warm-up" data (just 4 hours of random robot movements in a simulation). Then, they asked it to perform complex tasks like grasping, placing, and reaching.
The Result: The robot succeeded 83% of the time in the real world, beating other top AI methods by a huge margin (over 50% better). It didn't need to be retrained on the specific task; it just used its "mental simulator" to plan the perfect path.

Summary

DexWM is like giving a robot a human-like intuition. By watching humans and learning exactly how fingers interact with objects, it built a mental model of physics. When faced with a new task, it doesn't guess; it simulates the future in its head, plans the perfect finger movements, and executes them with surprising grace.

It proves that if you teach a robot to "watch" humans carefully, it can learn to "do" things itself, even if it has a different body than we do.

1. Problem Statement

The core challenge addressed is modeling dexterous hand-object interactions for robotic manipulation. While recent "World Models" (models that predict future environmental states given past states and actions) have advanced, they typically rely on coarse action spaces (e.g., text prompts, navigation commands, or whole-body motion) that fail to capture the fine-grained dexterity required for tasks like in-hand manipulation, grasping, and placing.

Furthermore, learning these models directly from robot data is hindered by the scarcity of large-scale datasets featuring dexterous grippers (multi-fingered hands). Existing robot datasets often use parallel-jaw grippers, which lack the complexity of human hands. The paper aims to bridge this gap by leveraging abundant human video data to learn dexterous dynamics and transfer them to robots.

2. Methodology: DexWM

The authors propose DexWM, a latent-space world model designed to predict future environmental states conditioned on past states and precise dexterous actions.

A. Data Sources

To overcome data scarcity, DexWM is trained on a massive scale of human and non-dexterous robot data:

EgoDex: 829 hours of egocentric (first-person) human videos with rich 3D hand and object annotations.
DROID: ~100 hours of robot manipulation data using parallel-jaw grippers.
RoboCasa (Fine-tuning): ~4 hours of exploratory, non-task-specific robot simulation data used to bridge the sim-to-real gap.

B. State and Action Representations

Latent States ( $S$ ): Instead of raw pixels, the model uses patch-level embeddings from a frozen DINOv2 image encoder. This captures semantic structure (shape, geometry) while ignoring irrelevant details like color, improving generalization.
Action Representation ( $A$ ): To capture dexterity, actions are not represented as high-level text or wrist positions. Instead, they are defined as the difference in 3D hand keypoints between timesteps.
- Based on the MANO model, each hand is represented by 21 keypoints.
- The action vector includes the change in hand keypoints ( $H_{k2} - H_{k1}$ ) and the change in camera pose (translation and orientation).
- For parallel-jaw grippers (DROID), "dummy" keypoints are placed on concentric circles at the end-effector to approximate finger spread.

C. Predictor Architecture

Model: A Conditional Diffusion Transformer (CDiT).
Mechanism: Unlike standard diffusion models that generate images iteratively, DexWM directly regresses future latent states to ensure faster inference.
Conditioning: The action vector (132 dimensions: 44 keypoints $\times$ 3 coordinates + camera motion) is injected into every transformer block via AdaLN (Adaptive Layer Normalization) layers.
Training Strategy: The model is trained with multistep prediction, randomly skipping timesteps to improve generalization to variable frequencies.

D. Hand Consistency Loss (Key Innovation)

The authors observed that predicting the global latent state alone was insufficient for capturing fine-grained hand details, as hands occupy a small portion of the image.

Solution: An auxiliary Hand Consistency (HC) Loss is introduced.
Mechanism: A separate transformer head predicts heatmaps of fingertip and wrist locations from the predicted latent state.
Objective: The model is optimized to minimize the error between the predicted heatmaps and the ground truth heatmaps. This forces the latent state to encode precise hand configurations, significantly improving dexterity modeling.

E. Planning and Control

DexWM is not used as a direct policy (predicting actions from images) but as a state-transition model within a Model Predictive Control (MPC) framework.

Goal: Given a start state and a goal image, the system optimizes a sequence of joint angles ( $\Theta$ ) to minimize the distance between the predicted final state and the goal.
Optimizer: The Cross-Entropy Method (CEM) is used to search the action space.
Execution: The optimized trajectory is executed by a low-level controller. This approach allows for robust planning and zero-shot transfer.

3. Key Contributions

DexWM Architecture: A novel world model that learns dexterous dynamics directly from human videos by using fine-grained 3D hand keypoints as the action space.
Hand Consistency Loss: A critical auxiliary loss that enforces accurate hand configuration prediction, solving the issue of coarse latent representations failing to capture subtle finger movements.
Cross-Embodiment Transfer: Demonstrating that a model trained on human data (EgoDex) and non-dexterous robot data (DROID) can be fine-tuned with minimal robot data (~4 hours) to perform complex tasks on a real-world dexterous robot (Franka Panda + Allegro gripper).
Zero-Shot Planning: Achieving high success rates in grasping, placing, and reaching tasks without task-specific training data, relying solely on the world model's predictive capabilities.

4. Experimental Results

The paper evaluates DexWM across open-loop simulation, action transfer, and real-world robotics.

Open-Loop Prediction:
- DexWM outperforms baselines like NWM (Navigation World Model) and PEVA (Whole-body conditioned video prediction) in predicting future states.
- While PEVA had slightly lower perceptual error (L2), DexWM achieved significantly higher PCK@20 (Percentage of Correct Keypoints), proving it captures hand dynamics more accurately.
- Scaling: Performance improves with model size (DexWM-XL, 456M parameters).
Action Transfer:
- DexWM successfully transferred actions from a reference sequence to a new environment, maintaining fine-grained hand configurations, whereas PEVA failed to replicate specific finger articulations.
Robot Transfer (Simulation & Real-World):
- Simulation: On RoboCasa tasks (Reach, Grasp, Place), DexWM achieved 72% (Reach), 28% (Place), and 58% (Grasp) success rates. This significantly outperformed Diffusion Policy (DP) and a version of DexWM without human pre-training.
- Real-World: Deployed on a Franka Panda with an Allegro gripper, DexWM achieved an 83% success rate (10/12 trials) on grasping tasks zero-shot (no real-world fine-tuning).
- Comparison: Diffusion Policy trained on the same exploratory data failed completely (0% success) on the real robot, highlighting the robustness of the world model approach.

5. Significance and Impact

Data Efficiency: The work proves that human video data is a viable and superior source for training dexterous robot policies, bypassing the need for massive, expensive dexterous robot datasets.
Granularity: It establishes that fine-grained action representations (keypoints) are essential for modeling dexterous interactions, moving beyond the coarse actions used in previous world models.
Robustness: By using a world model for planning (MPC) rather than direct action prediction, the system handles the "sim-to-real" gap and dataset noise more effectively than behavior cloning methods like Diffusion Policy.
Future Directions: This approach paves the way for general-purpose embodied agents capable of complex manipulation tasks by leveraging the vast amount of human interaction data available on the internet.

In summary, DexWM represents a significant step forward in robotic manipulation by combining large-scale human video learning with precise dexterous action modeling, enabling robots to perform complex, unseen tasks with high reliability.