💻 computer science

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

GE-Act 2.0 is a novel world-action model trained from scratch on manipulation data that integrates a control-oriented autoencoder, a single-step visual planner, and an inverse dynamics model with knowledge-aligned selective optimization, achieving significant zero-shot success across diverse tasks and embodiments through extensive scaling and cross-embodiment transfer.

Original authors: AgiBot Research Team, Renhang Liu, Wenzhi Zhao, Zhuo Yang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Guanghui Ren, Youlun Peng, Rongjun Jin, Nan Wang, Sukai Wang, Xindong He, Jinyuan Feng, Ziyu Xion

Published 2026-09-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: AgiBot Research Team, Renhang Liu, Wenzhi Zhao, Zhuo Yang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Guanghui Ren, Youlun Peng, Rongjun Jin, Nan Wang, Sukai Wang, Xindong He, Jinyuan Feng, Ziyu Xiong, Linqing Zhong, Yifei Wei, Feng Han, Long Zhang, Da Huang, Nanshu Zhao, Chenghao Yin, Mo Wu, Zhaodong Yan, Kongtao Hu, Yuxiang Yan, Aogelijiang Niyazi, Yu Fang, Jia Zeng, Lizhu Meng, Daizhen Lv, Haoyu Cao, Zhiwen Hou, Lianjin Ye, Yuehan Niu, Zhikai Cai, Xuan Hu, Hui Min, Xiongfeng Cai, Yue Liao, Jing Wu, Soujanya Poria, Ye Li, Sanping Zhou, Maoqing Yao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of the factory floor, repeating the same precise motion thousands of times without error. But step them into a kitchen, a living room, or a cluttered workshop, and they often freeze. The challenge is not just moving a hand; it is understanding how the world changes when that hand touches something. To navigate this, a new generation of artificial intelligence is learning to imagine the future. Instead of simply reacting to what it sees right now, these systems try to predict what will happen next if they take a specific action. This approach, known as a world-action model, allows a robot to simulate a sequence of events in its mind before it moves a muscle, checking if the outcome matches its goal. For years, researchers have tried to build these systems by taking existing video generators—programs trained to create fake movies—and forcing them to understand robot movements. However, this method often leaves the robot with a vague understanding of physics, as the video generator was never truly designed to control a physical body.

A team from AgiBot has now introduced a different approach, one that builds the robot's imagination from the ground up using only real-world interaction data. They created a system called GE-Act 2.0, which learns to predict the future and decide on actions simultaneously, but with a crucial twist: every part of the system is trained from scratch on data collected from robots and humans manipulating objects. The researchers did not rely on pre-made video models. Instead, they taught the system a compressed language of movement, a way to see the world that strips away unnecessary detail to focus on what matters for control. This allowed the robot to learn from a vast and varied library of data, including hours of successful demonstrations, failed attempts, and even videos of humans working without any recorded instructions. By combining these different types of learning, the system learned to generalize, meaning it could apply what it learned in one situation to a completely new one it had never seen before.

The core of this achievement lies in how the researchers handled the messy reality of the physical world. When a robot tries to learn, it often faces a confusing problem: the same instruction can be completed in many different ways. A robot might be told to "pick up the red cup," and it could approach from the left, the right, or grab it by the handle or the rim. If the training data only shows one way, but the robot's imagination predicts another, the two parts of the system can get out of sync. The researchers identified this mismatch as a "validity gap," where the robot's prediction of the future does not align with the action it is supposed to take. To fix this, they developed a selection process that acts like a filter. Before the robot learns from a predicted future, it checks if that future is compatible with the action it needs to perform. It generates several possible futures, tests them against the required action, and only learns from the ones that make sense together. This ensures the robot does not blur its understanding of how different actions lead to different outcomes.

The results of this training are striking, particularly when tested on tasks the robot has never practiced. The researchers evaluated the system on one hundred distinct manipulation tasks, ranging from stacking blocks and pouring liquids to folding towels and inserting plugs. These tests were conducted on real robots in environments with different lighting, backgrounds, and object arrangements than those used during training. When the system was trained on a small dataset of 300 hours, it succeeded in only 17.1% of the tasks on one robot model. However, as the researchers scaled up the training data to 30,000 hours, the success rate climbed steadily to 44.1%. This improvement happened without any specific fine-tuning for the individual tasks; the robot simply applied the general skills it had learned to the new challenges. Even more surprisingly, the system showed strong abilities on a second, different robot model that made up less than 2% of the training data, suggesting that the skills learned from the larger dataset transferred effectively to a new physical form.

The system's ability to follow instructions was also tested under difficult conditions. In many trials, the robot was asked to perform an action that conflicted with what it might have expected based on the scene or a previous habit. For instance, if a robot was in the middle of a movement, it could still stop and follow a new, explicit command to change its path. In over 90% of these trials, the robot correctly identified the specific object, color, shape, or position mentioned in the instruction, even when those details were subtle or the context was confusing. The researchers found a strong link between the variety of skills the robot learned during training and its ability to succeed on new tasks. The more diverse the training data, the more likely the robot was to handle a novel situation. This suggests that the path to more capable robots lies not just in better algorithms, but in the sheer volume and variety of the data they consume.

This work marks a significant step toward robots that can learn from the same rich, unstructured data that humans use to understand the world. By building a system that predicts the future and plans actions in a unified way, and by teaching it to select the right predictions from many possibilities, the researchers have created a model that is robust and adaptable. The findings suggest that with enough diverse experience, a robot can develop a deep, intuitive understanding of how objects behave and how to interact with them, moving beyond rigid programming toward a more flexible form of intelligence. The success of this approach on real hardware, without needing to retrain for every new job, points to a future where robots can be deployed in dynamic, unpredictable environments and learn to thrive there.

Technical Summary: GE-Act 2.0

Problem Statement

World-action models (WAMs) aim to predict future scene states to guide robotic actions, enabling learning from both action-free video and action-labeled interactions. However, existing WAMs typically inherit pretrained video generators and focus on coupling them with action models, leaving the pretraining and scaling of the WAM as a unified system underexplored. This approach creates two primary unresolved challenges:

Representation Mismatch: Standard video autoencoders are optimized for reconstruction fidelity rather than control, often resulting in latent spaces that are too dense for efficient control loops or lack the specific information needed for action recovery.
Data Utilization and Alignment: Most prior WAMs cannot effectively leverage diverse data sources (e.g., action-free videos for visual generation and instruction-free trajectories for inverse dynamics) separately before connecting them. Furthermore, a "validity gap" exists where a stochastically generated future state may depict a behavior (outcome mode) different from the recorded action used as a label, leading to mismatched supervision that erodes action diversity.

Methodology: GE-Act 2.0

The authors introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model where every trainable component is initialized from scratch on manipulation data. The architecture employs a decoupled two-stage design consisting of three core components:

1. Control-Oriented Autoencoder (CoAE)

To address the representation mismatch, GE-Act 2.0 introduces a CoAE that learns an aggressively compressed latent space (64× spatial downsampling, 24 tokens per frame) specifically for manipulation.

Training: It combines pixel reconstruction with multi-teacher feature alignment against frozen visual encoders (SigLIP 2, V-JEPA 2.1, and DINOv3).
Goal: This ensures the latent space retains semantic, spatiotemporal, and dense visual features necessary for both visual prediction and action recovery, while minimizing token overhead.

2. Single-Step Visual Planner (SVP)

The SVP serves as the generative world model, predicting a complete future visual state in a single differentiable pass.

Architecture: It utilizes a conditional MeanFlow formulation with a multimodal understanding module (a frozen VLM, Qwen3.5) to ground instructions in the current observation.
Multi-Scale Prediction: It predicts both dense near-term frames (for control dynamics) and sparse far-horizon frames (for task intent).
Significance: By generating the full future in one step, the SVP allows the visual planner and the Inverse Dynamics Model (IDM) to be pretrained separately on complementary data (action-free video for SVP; instruction-free trajectories for IDM) before being connected via end-to-end co-training. To our knowledge, this SVP is the first native single-step robot world model pretrained from random initialization on manipulation video.

3. Inverse Dynamics Model (IDM) and KASO

The IDM translates visual transitions into action sequences. A critical innovation is Knowledge-Aligned Selective Optimization (KASO), designed to resolve the "validity gap."

The Validity Gap: In manipulation, multiple valid behaviors (outcome modes) can satisfy a single instruction. If the SVP generates a future corresponding to Mode A, but the recorded action label corresponds to Mode B, the IDM receives conflicting supervision.
KASO Mechanism: During co-training, the SVP samples $N$ candidate futures. The active IDM probes these candidates at a high-noise point in the action flow path to estimate their compatibility with the recorded action. Only the top- $k$ candidates deemed behaviorally compatible (low energy discrepancy) are used for the joint action loss update.
Benefit: This preserves action diversity and prevents the model from collapsing to a mean action or blurring distinct modes, facilitating better downstream exploration (e.g., via reinforcement learning).

Key Contributions

From-Scratch Pretraining: GE-Act 2.0 initializes all trainable generative and action components from scratch on manipulation data, rather than relying on pretrained video generators.
Decoupled Pretraining Strategy: The introduction of the SVP enables separate pretraining of visual generation (on action-free video) and inverse dynamics (on instruction-free trajectories), followed by end-to-end co-training.
KASO for Validity Gap: The proposal of KASO to select action-compatible futures during training, effectively mitigating the supervision mismatch inherent in multimodal manipulation tasks.
Systematic Scaling: A demonstration of how scaling co-training data from 300 to 30,000 hours improves zero-shot Out-of-Distribution (OOD) performance across diverse tasks and embodiments.

Results

The authors evaluated GE-Act 2.0 directly on 100 atomic tasks across 20 skill groups using two robot embodiments (G1-OP and G2-90D) without any per-task fine-tuning (zero-shot OOD).

Scaling Performance: Increasing co-training data from 300 to 30,000 hours raised the zero-shot success rate on G1-OP from 17.1% to 44.1% and on G2-90D from 13.4% to 31.1%.
Cross-Embodiment Transfer: Despite G2-90D comprising less than 2% of the co-training data, its performance improved by 17.7 percentage points, indicating effective transfer from the broader corpus.
Skill Coverage: Success gains were observed in 19/20 skill groups for G1-OP and 18/20 for G2-90D. Skill-specific coverage strongly correlated with zero-shot OOD success (Pearson $r=0.80$ ).
Grounding and Instruction Following: The model successfully grounded object, color, shape, and position references in at least 90% of trials. Qualitative tests showed it could follow explicit instructions even when they conflicted with committed behaviors or conventional scene associations.
Ablation Studies: Controlled real-robot experiments confirmed that KASO significantly outperforms standard End-to-End (E2E) and E2E with pretraining losses (E2E+PT) in terms of pick success, particularly in complex multi-object scenarios.

Significance

The paper claims that GE-Act 2.0 demonstrates that pretraining and scaling world-action models from scratch is a viable and effective paradigm for robotic manipulation. By decoupling visual planning and inverse dynamics pretraining and addressing the validity gap through KASO, the model achieves broad zero-shot generalization without task-specific adaptation. The results suggest that a single world-action model can learn from heterogeneous data sources (action-free video, failed attempts, and fully labeled demonstrations) to acquire a wide repertoire of manipulation skills, with performance scaling predictably with data volume. The work highlights the importance of action-compatible supervision in maintaining the diversity of learned behaviors, a critical factor for future reinforcement learning and exploration strategies.