🤖 machine learning

Motus2: A Self-Evolving General World Model for Dexterous Manipulation

Motus2 is a self-evolving general world model for dexterous manipulation that unifies policy, simulation, and evaluation into a closed decision-and-learning loop, leveraging scaled model architectures and progressively expanded multi-modal datasets to enable continuous improvement through both expert demonstrations and self-generated interaction data.

Original authors: Hongzhe Bi, Zihao Zhou, Yihang Tang, Jingrui Pang, Shuhe Huang, Haitian Liu, Runqing Wang, Shuai Huang, Yichen Wang, Yiming Cheng, Ruowen Zhao, Zhenghua Li, Hengkai Tan, Xiaolong Liu, Jinhui Wan, Jiab

Published 2026-09-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongzhe Bi, Zihao Zhou, Yihang Tang, Jingrui Pang, Shuhe Huang, Haitian Liu, Runqing Wang, Shuai Huang, Yichen Wang, Yiming Cheng, Ruowen Zhao, Zhenghua Li, Hengkai Tan, Xiaolong Liu, Jinhui Wan, Jiabao Liu, Min Zhao, Fan Bao, Jun Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To build a robot that can truly handle the messy, unpredictable world of human tasks, scientists have long sought a system that does more than just follow a list of instructions. The goal is to create a machine that can perceive its surroundings, imagine what might happen next, take an action, and then learn from the result to do better the next time. This concept, known as a "world model," acts like an internal simulator for the robot. Instead of just reacting to what it sees right now, the robot runs mental simulations to guess the outcome of its movements before it actually makes them. While earlier versions of these systems could predict the future or suggest a move, they often operated in a straight line: see, guess, act, and stop. They lacked a way to look back at their own predictions, judge whether they were good or bad, and use that judgment to improve their own decision-making skills without needing a human to show them the right way every single time.

A team of researchers has now introduced a system called Motus2, a self-evolving robot brain designed specifically for delicate, complex tasks like using tools or handling objects with human-like hands. This system represents a significant step forward because it closes the loop between thinking and learning. Rather than just being a passive observer or a simple follower, Motus2 acts as its own teacher. It proposes a series of possible moves, simulates what those moves would look like in the future, and then evaluates whether those imagined outcomes would lead to success. If the simulation shows a failure, the robot learns from that mistake. If it shows success, it reinforces that path. This cycle allows the robot to improve its own policy, or its set of rules for acting, by constantly testing and refining its own ideas, even when it has no human to guide it through every step.

The foundation of this system is built on a massive amount of data collected from human hands. The researchers started by teaching the robot to understand how humans interact with the world using video recorded from a first-person perspective, as if the camera were mounted on the person's head. They began with simple, single-lens videos showing a wide variety of tasks, from cooking to organizing, and then moved to more advanced, synchronized stereo videos that provide a sense of depth, much like human binocular vision. This allowed the robot to learn the subtle physics of how hands grasp, lift, and manipulate objects. However, simply watching humans is not enough for a robot with a different body. The researchers then guided the system through a middle stage of training where it learned to translate those human movements into the specific mechanics of its own robotic arms and hands. This process involved showing the robot thousands of hours of human data, followed by specific examples of how humans and robots can work together, effectively bridging the gap between human intuition and robotic execution.

What makes Motus2 unique is how it uses this knowledge. The system is built around a single, unified model that wears three different hats simultaneously. First, it acts as a policy, suggesting what action to take next. Second, it acts as a simulator, predicting what the world will look like after that action is taken. Third, it acts as an evaluator, judging whether that predicted future is good or bad. In traditional systems, these functions are often separate or disconnected, but here they are tightly woven together. When the robot considers a move, it instantly runs a mental simulation to see the consequences. It then asks itself if that consequence is desirable. If the answer is no, it discards that path and tries another. If the answer is yes, it commits to the action. This internal dialogue happens so quickly that the robot can plan several steps ahead, choosing the best path forward before it even moves a muscle.

The researchers tested this system on a fully robotic platform equipped with two arms, two dexterous hands, and sensors that can feel touch, similar to human fingertips. They challenged the robot with a series of difficult tasks, such as placing a ball in a specific spot, screwing a lightbulb into a socket, or attaching an eraser to a pen. In these tests, the robot's ability to learn from its own simulations proved crucial. When the system was allowed to use its internal evaluator to select the best options during a task, its success rate improved significantly. Even more impressively, when the system was allowed to use its own predictions to update its own learning rules, it became even better at the task. The robot did not just get lucky; it genuinely learned to avoid mistakes it had simulated and repeat actions it had simulated to be successful.

A key part of this success was the robot's ability to remember what happened earlier in a task, even if that information was temporarily hidden from view. In many complex jobs, a hand might block the view of an object, or a critical step might happen long before the final result is seen. To handle this, the researchers gave the robot a form of working memory that could hold onto important visual details from the past. They tested different ways of managing this memory, from keeping a short, rolling window of recent events to maintaining a longer, more detailed history. The results showed that having access to this longer history allowed the robot to solve tasks that required remembering a sequence of events over a longer period, proving that the ability to recall the past is just as important as predicting the future.

The system also incorporated a specialized module for touch. While vision is powerful, it cannot always tell if a grip is slipping or if a surface is too soft. The robot was equipped with a lightweight expert system dedicated to processing tactile feedback. This allowed it to refine its movements in real-time, adjusting its grip the moment it felt a slip or a change in pressure. This combination of seeing, feeling, thinking, and learning created a robust system capable of handling the uncertainty of the real world.

The findings suggest that the path to truly capable robots lies not just in collecting more data, but in building systems that can use that data to question and improve themselves. By combining large-scale human interaction data with a self-correcting loop of prediction and evaluation, Motus2 demonstrated that robots can learn to manipulate the physical world with a level of adaptability that was previously out of reach. The researchers found that as they increased the amount of stereo video data used for training, the robot's ability to predict human actions improved in a consistent, predictable way. This scaling suggests that with even more data, these systems could become even more proficient. Ultimately, the work shows that a robot does not need to be programmed with every possible solution to every possible problem. Instead, if it can simulate the future, judge the outcome, and learn from the difference, it can evolve its own skills to meet the challenges of a complex, dexterous world.

Technical Summary: Motus2 – A Self-Evolving General World Model for Dexterous Manipulation

Problem Statement

General embodied agents require a unified system capable of perceiving, predicting, acting, evaluating, and improving. While world models show promise, existing approaches typically append an action output head to a simulator without coupling them into a closed decision-and-learning loop. Current robot foundation models rely heavily on curated, action-supervised datasets, which are expensive to collect at scale. Crucially, imitation learning alone lacks a mechanism to evaluate whether an action is good or bad, nor does it provide a pathway for policy improvement based on the agent's own outcomes. Furthermore, transferring human interaction priors to dexterous robots is challenging due to partial observability (self-occlusion), ambiguous contact events (e.g., fingertip slip), and the need for temporal context beyond short observation windows.

Methodology

Motus2 addresses these challenges by introducing a self-evolving General World Model (GWM) that unifies policy, simulation, and evaluation within a single shared-parameter architecture. The approach is built on two axes of scaling: data scaling (hierarchical egocentric human data) and model scaling (closed-loop policy improvement).

1. Unified Architecture and Control Interfaces

Motus2 utilizes a single video–action backbone (based on a foundation video diffusion model) that exposes three distinct control interfaces through a shared parameter set:

Policy (World–Action Model): Generates executable action chunks conditioned on language, current proprioception, and visual context.
Simulator (Action-Conditioned World Model): Predicts future visual observations given the current state and a proposed action.
Evaluator (Value Model): Assesses the predicted future outcomes to estimate task progress.

These interfaces are realized through a specific factorization of the joint density $p_\theta(A_t, Z_t, Y_t | c_t)$ , where $A_t$ is the action chunk, $Z_t$ is the latent future observation, and $Y_t$ is a discretized task-progress value. The model employs an action-first mask during training and inference: action tokens cannot access future video or value tokens from their own chunk, ensuring causal consistency.

2. Data Scaling and Curriculum

The training curriculum progresses through a hierarchical data pyramid:

Pre-training: Starts with large-scale monocular egocentric data (low to high resolution) to learn broad interaction priors. It then transitions to synchronized stereo egocentric data, which provides implicit depth cues and more accurate 3D hand-pose estimates.
Mid-training (Robot Domain Adaptation): The model is grounded in the robot domain using robot trajectories supplemented with human–robot alignment data.
Supervision Routing: A key innovation is the separation of supervision targets. Curated successful trajectories supervise action learning (imitation). In contrast, failed and suboptimal interactions are routed to the simulator and evaluator branches. This allows the model to learn dynamics and value functions from failures without teaching the policy to reproduce undesirable behaviors.

3. Self-Evolution via Closed-Loop Learning

Motus2 implements a closed decision-and-learning loop:

Value-Guided Supervision: The evaluator is trained with relative-progress targets. Successful segments receive positive rewards proportional to their progress, while failed or task-irrelevant segments receive negative rewards.
Model-Based Reinforcement Learning (MBRL): Using DiffusionNFT, the system converts scalar value estimates from the evaluator into policy updates. High-value candidate actions reinforce the policy distribution, while low-value actions push it away.
Test-Time Planning: The system supports Best-of-N planning, where multiple action candidates are generated, simulated, and scored by the evaluator. The highest-scoring branch is executed, allowing for online selection without modifying the policy weights.

4. Extensions for Long-Horizon and Contact

Working Memory: To address partial observability and long-horizon tasks, Motus2 implements a sliding window context and two extensions: Global Autoregression (retaining all history) and Hybrid Working Memory (retaining anchor frames and compressed persistent tokens).
Tactile Expert: A lightweight module refines action chunks using high-rate tactile feedback immediately before execution and predicts future force signals, enabling contact-aware control without rerunning the full backbone.

Key Contributions

A General World Model for Dexterous Manipulation: Motus2 jointly models executable actions, action-conditioned futures, and task-progress values within a single shared-parameter model. It uniquely integrates a policy, simulator, and evaluator, supported by a tactile expert for contact-sensitive execution.
Egocentric Data Scaling and Transfer: The paper demonstrates a scaling trend where pretraining on monocular data, followed by stereo data, and finally robot-domain adaptation, significantly improves downstream performance. It establishes that stereo human data scales effectively for dexterous manipulation.
Value-Guided Closed-Loop Self-Evolution: The framework separates imitation from other interaction evidence, using failures for dynamics and value learning. It successfully couples model-based planning (Best-of-N) and policy optimization (MBRL) within a unified architecture to improve the policy beyond supervised imitation.

Experimental Results

Experiments were conducted on a fully biomimetic platform with stereo vision, dual arms, and dexterous hands, covering tasks like placing objects, screwing bulbs, and multi-finger grasping.

Performance Gains: Compared to a baseline supervised fine-tuning (SFT) model initialized with a general video model (WAN-SFT), Motus2 (Midtrain-SFT) achieved a macro-average success rate of 84%, a significant increase from the 51% achieved by pretraining-only models and 0% for the baseline.
Scaling Laws: Experiments on stereo egocentric data (2k to 20k hours) confirmed a log-linear scaling law, where validation error decreases monotonically as data volume increases, extending previous findings from monocular to stereo observations.
Policy Improvement: Adding Best-of-N planning improved success rates by 2.5 points. Adding MBRL improved rates by 7.5 points. Combining both yielded a 10-point gain over the base Motus2 model, demonstrating that both inference-time selection and policy distribution updates contribute to performance.
Context and Tactile: The study confirmed that long-horizon context mechanisms (hybrid memory) and tactile feedback are critical for tasks involving occlusion and contact-rich manipulation.

Significance

The paper posits that Motus2 provides a general path toward self-evolving dexterous manipulation. By unifying prediction, action, and evaluation in a single model and leveraging heterogeneous interaction data (including failures), the system moves beyond static imitation learning. It demonstrates that a general world model can not only simulate the future but also use that simulation to iteratively improve its own policy, addressing the core limitations of current robot foundation models that lack mechanisms for self-improvement. The work suggests that scaling both data (from monocular to stereo to robot-aligned) and model capabilities (via closed-loop learning) is a viable strategy for building capable general embodied agents.