💻 computer science

Riemann-1.0: An Embodied World Action Model for Physical AI

Riemann-1.0 is a unified causal autoregressive World Action Model that integrates multi-view observations, robot states, and actions into a single framework, leveraging a progressive pretraining strategy on over 200,000 hours of diverse embodied data to achieve state-of-the-art performance in both simulation and real-world long-horizon manipulation tasks.

Original authors: Haofeng Sun, Jiangbo Pei, Fei Kang, Zexiang Liu, Yaokun Li, Boyi Jiang, Hua Xue, Cindy Zhou, Wei Li, Yichen Wei, Mengyin An, Fanliang Zhao, Biao Jiang, Zile Wang, Yang Liu, Yangguang Li

Published 2026-08-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haofeng Sun, Jiangbo Pei, Fei Kang, Zexiang Liu, Yaokun Li, Boyi Jiang, Hua Xue, Cindy Zhou, Wei Li, Yichen Wei, Mengyin An, Fanliang Zhao, Biao Jiang, Zile Wang, Yang Liu, Yangguang Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Artificial intelligence has long been a master of the digital realm, reading text, recognizing images, and solving puzzles within the confines of a computer screen. But for machines to truly join us in the physical world, they must do more than just understand what they see; they must learn to act within it. This is the challenge of embodied intelligence: teaching robots to perceive their surroundings, reason about cause and effect, and execute physical movements with the same fluidity a human uses to pick up a cup or fold a shirt. For a machine to succeed, it needs a model of the world that understands not only how objects look but how they move and change when touched. It must learn that pushing a block causes it to slide, and that lifting a lid reveals what is inside. Until now, creating a single system that can both plan these physical actions and predict the visual consequences of those actions has been a difficult hurdle, often requiring separate systems for thinking and for moving.

A team of researchers has introduced a new system called Riemann-1.0, designed to bridge this gap by treating a robot's actions and the resulting changes in the world as a single, continuous story. Instead of building separate tools to decide what to do and tools to imagine what will happen next, this model learns to do both at once. It operates on a simple but powerful principle: in the real world, an action always happens before the result is seen. If a robot reaches for a spoon, the movement occurs first, and the visual change—the spoon moving—follows. Riemann-1.0 is built to respect this order, learning from a massive collection of video and movement data to predict exactly what a robot should do next and what the world will look like after it does.

To teach this system, the researchers gathered a vast library of experience, totaling more than 200,000 hours of interaction. This collection was not just a pile of robot videos; it was a carefully curated mix of three different types of learning material. First, they included thousands of hours of videos recorded from a human's point of view, showing people performing everyday tasks like cooking or cleaning. These videos provided a broad understanding of how objects interact and how tasks unfold over time, even though they lacked the specific mechanical data of a robot. Second, they added demonstrations from handheld robotic grippers and wearable devices, which serve as a bridge, showing how human hand movements translate into machine-like controls. Finally, they included precise recordings of actual robot movements, which provided the exact, executable instructions needed to teach the machine how to move its own limbs. By combining these sources, the researchers created a unified dataset that allowed the model to learn from the broad wisdom of human experience while grounding that knowledge in the precise mechanics of robot control.

The training process for Riemann-1.0 was structured like a school curriculum, moving from general observation to specific execution. In the first stage, the model studied the vast library of human videos. Since these videos did not have robot movement data, the system used a helper tool to estimate the invisible "actions" that must have caused the visual changes seen on screen. This allowed the model to learn the basic dynamics of how the world moves without needing perfect robot data. In the second stage, the model was introduced to the mixed data of human hands and robot grippers, learning to align its understanding of movement with real, physical controls. Finally, in the third stage, the model focused exclusively on high-quality recordings of robots performing tasks. This final phase sharpened its ability to generate precise, executable commands, ensuring that the actions it planned were not just visually plausible but physically possible for a real machine to perform.

The results of this approach were striking. When tested on a wide variety of tasks, both in computer simulations and on real robots, Riemann-1.0 outperformed previous methods by a significant margin. In a simulation designed to test long, complex tasks involving many steps, the model succeeded 62.6% of the time, a notable improvement over the best existing systems. In another simulation focused on two-armed robots, it achieved a success rate of 94.3%. Perhaps most impressively, when deployed on real-world robots to perform tasks like stacking colored blocks, folding clothes, organizing a cluttered desk, or arranging kitchen items, the system completed the tasks successfully 85% of the time. It also showed a remarkable ability to adapt to new situations it had not seen before, such as placing a Rubik's cube into a box or putting a towel into a basin, succeeding in 85% of these novel trials.

Beyond simply acting as a robot controller, Riemann-1.0 can also function as a simulator. Because it understands the causal link between an action and the visual result, researchers can ask it to imagine what would happen if a robot performed a specific movement. The model can generate a video of the future, showing exactly how objects would move and where they would end up, based on the action provided. This dual capability means the system can serve as both the brain that decides what to do and the imagination that checks if the plan will work before the robot ever moves. This ability to predict the future of the physical world, grounded in the reality of how actions cause changes, represents a significant step forward in making artificial intelligence a reliable partner in the physical world. The work suggests that by unifying the learning of how to act with the learning of how the world responds, machines can develop a more robust and generalizable understanding of the tasks we perform every day.

Technical Summary: Riemann-1.0

Problem Statement

The development of Physical AI requires "Embodied Brain Models" capable of unifying environmental perception, world dynamics modeling, executable action generation, and long-horizon reasoning. While World Action Models (WAMs) have emerged as a promising paradigm for jointly modeling robot actions and world dynamics, existing approaches face two fundamental limitations in scaling embodied experience:

Data Heterogeneity: Embodied experience is inherently multimodal and embodiment-dependent. Existing pretraining strategies are often tightly coupled with specific robot trajectories, failing to effectively leverage large-scale, weakly supervised human data (e.g., egocentric videos) or bridge the gap between human demonstrations and robot control spaces.
Modeling Paradigm Limitations: Current WAM formulations—such as joint generation, video-first prediction, or decoupled action-video modeling—fail to jointly model observations, robot states, and actions within a unified causal process. Consequently, they cannot simultaneously support efficient robot policy learning and causally consistent, action-conditioned world simulation within a single model.

Methodology

1. Unified Embodied Data Infrastructure

To address data heterogeneity, the authors constructed a corpus of over 200,000 hours of embodied experience spanning three complementary sources:

Egocentric Human Videos (200K+ hours): Providing scalable interaction knowledge and long-horizon task structures.
Handheld-Gripper/Exoskeleton Demonstrations (12K+ hours): Bridging human interactions with robot-compatible control spaces.
Heterogeneous Robot Trajectories (20K+ hours): Providing precise, embodiment-specific executable control supervision.

A Unified Embodied Data Engine processes these sources into a standardized action-level trajectory representation. Key processing steps include:

Visual Preprocessing: Fisheye correction and orientation normalization.
Semantic Annotation: Hierarchical VLM-based decomposition of videos into task and action-level clips.
3D Hand Reconstruction: Using MANO-based reconstruction and camera pose estimation (VGGT-Ω) to derive continuous 3D hand trajectories from video-only data, creating pseudo-action supervision.
Semantic-Aware Balancing: Sampling across semantic dimensions (scenes, skills, objects) to handle long-tail distributions rather than simply balancing raw data volume.

2. Riemann-1.0: Fully Causal Autoregressive World Action Model

Riemann-1.0 formulates robot interaction as a unified fully causal autoregressive sequence over multi-view visual observations ( $z_t$ ), robot states ( $s_t$ ), and embodiment-specific actions ( $a_t$ ).

Architecture: The model utilizes a Transformer backbone with a Flow Matching objective. It employs a Wan VAE for visual latent encoding and 3D Patch Embeddings.
Causal Factorization: The joint probability is factorized as:
$p(a_{1:T}, z_{1:T} | z_0, s_0, c) = \prod_{t=1}^T p(a_t | z_{<t}, s_{<t}, a_{<t}, c) \cdot p(z_t | z_{<t}, s_{<t}, a_{\leq t}, c)$
Crucially, actions are predicted before the corresponding visual latent is generated. This ordering mirrors real-world causality where actions precede visual consequences.
Dual Functionality:
- As a Policy: The model predicts actions online; the predicted visual latent is replaced by the actual environment observation for the next step.
- As a Simulator: The model receives a candidate action trajectory and recursively generates future visual latents (and decoded video) conditioned on those actions, enabling action-conditioned world simulation.
Embodiment Handling: The model shares a unified backbone but uses embodiment-specific linear projections and prediction heads to handle varying action/state dimensions and coordinate systems.

3. Progressive Embodied Pretraining

The training follows a three-stage curriculum to progressively transfer knowledge from weak supervision to executable control:

Stage 1: LAM-Action Bootstrap: The model is trained on unlabeled human videos using pseudo-actions generated by a frozen Latent Action Model (LAM). The LAM infers latent codes representing visual transitions between frames. The action loss weight ( $\lambda$ ) is low (0.1), focusing on learning broad visual dynamics.
Stage 2: Trajectory-Grounded Alignment: The model is fine-tuned on a mixed corpus of UMI demonstrations, robot trajectories, and human videos with 3D hand annotations. This stage aligns the latent action space with real executable actions across heterogeneous embodiments. The action weight increases to $\lambda=0.5$ .
Stage 3: Robot-Policy Enhancement: The model is trained exclusively on high-quality robot trajectories. The action weight is set to $\lambda=0.9$ , sharpening the model's executable action distribution and calibrating it for real-world actuator conventions and contact-rich execution.

Key Contributions

Fully Causal Autoregressive World Action Model: A novel formulation that jointly models multi-view observations, states, and actions in a single causal sequence, enabling a single model to function as both an executable robot policy and an action-conditioned visual world simulator.
Progressive Embodied Pretraining Framework: A three-stage training strategy that effectively unifies heterogeneous embodied experience (human videos, handheld demos, robot trajectories) under a shared objective, scaling learning from weak supervision to precise control.
Unified Data Infrastructure: A comprehensive pipeline transforming 200K+ hours of heterogeneous data into a standardized, action-level trajectory representation, including 3D hand reconstruction for video-only data.

Experimental Results

Riemann-1.0 was evaluated on both simulation benchmarks and long-horizon real-world manipulation tasks, achieving state-of-the-art (SOTA) performance:

Simulation Benchmarks:
- RoboTwin 2.0: 94.3% Success Rate (SR), outperforming the previous best (94.1%) by 0.2%.
- LIBERO: 99.0% SR, outperforming the previous best (98.9%) by 0.1%.
- RoboCasa-365: 62.6% SR on the long-horizon compositional benchmark, outperforming the previous best (54.2%) by 8.4 percentage points.
Real-World Manipulation:
- Evaluated on four tasks: Ordered Cube Stacking, Clothes Folding, Desk Organization, and Kitchen Organization.
- Achieved an average 85.0% SR and 94.4% Progress Success Rate (PSR).
- Outperformed the strongest open-source baseline (G0.5) by 15 percentage points in SR.
- Demonstrated strong Compositional Generalization (65.0% SR on unseen task instructions with seen objects) and Out-of-Distribution (OOD) performance (85.0% SR on unseen objects and scenes).
Visual Simulation: The model successfully generated multi-embodiment, action-conditioned visual rollouts, preserving scene structure and exhibiting action-consistent motion across diverse robot geometries and viewpoints.

Significance and Claims

The paper claims that Riemann-1.0 demonstrates that unified World Action Modeling combined with Progressive Embodied Pretraining effectively transforms large-scale embodied experience into generalizable robot manipulation capabilities.

The authors argue that by unifying the learning of visual dynamics and executable control within a single causal framework, and by leveraging a diverse spectrum of data sources through a progressive curriculum, the model overcomes the limitations of previous approaches that either decouple action and vision or rely solely on robot-specific data. The results suggest a scalable path for Physical AI where models can continuously evolve by scaling embodied experience rather than being constrained by the scarcity of labeled robot trajectories.