💻 computer science

Action Images: End-to-End Policy Learning via Multiview Video Generation

This paper introduces "Action Images," a unified world action model that formulates robot policy learning as multiview video generation by translating 7-DoF actions into pixel-grounded action videos, enabling the video backbone to function as a zero-shot policy without separate control modules while achieving superior performance in both zero-shot task success and video-action generation quality.

Original authors: Haoyu Zhen, Zixian Gao, Qiao Sun, Yilin Zhao, Yuncong Yang, Yilun Du, Tsun-Hsuan Wang, Yi-Ling Qiao, Chuang Gan

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoyu Zhen, Zixian Gao, Qiao Sun, Yilin Zhao, Yuncong Yang, Yilun Du, Tsun-Hsuan Wang, Yi-Ling Qiao, Chuang Gan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to make a sandwich.

The Old Way (The "Translator" Problem):
Traditionally, when we teach robots, we have two separate parts. First, a "Vision System" looks at the world and says, "I see a bread slice." Then, a separate "Brain" (the policy) has to take that visual information, translate it into a secret code of numbers, and tell the robot's arm exactly how to move.

The problem is like trying to teach a person to drive by only showing them a map and then asking them to guess the steering wheel movements. If the map is slightly different (a new room, a different lighting), the "Brain" gets confused because it was never taught to see the steering wheel move; it only learned the abstract numbers.

The New Way (Action Images):
This paper, titled "Action Images," proposes a brilliant shortcut. Instead of translating the robot's movement into secret numbers, the authors say: "Let's just draw the movement."

They turn the robot's 7-joint arm movement into a video of glowing dots that look just like a normal video.

The Creative Analogy: The "Ghostly Puppeteer"

Imagine you are watching a movie of a robot arm moving. Now, imagine that on top of that movie, there is a ghostly, glowing puppet made of light that perfectly mimics the robot's arm.

Red Light: Shows where the robot's hand is.
Green Light: Shows which way the hand is pointing.
Blue Light: Shows if the robot's "fingers" are open or closed.

In this paper, the robot doesn't just watch the movie; it learns to generate the movie of the glowing puppet itself.

How It Works (The Magic Trick)

One Big Brain, Not Two:
Instead of having a "Vision Brain" and a "Movement Brain," this model uses one giant video generator (like the AI that makes those viral videos of cats dancing).
- Normally, these AIs are trained to predict: "If I see a ball rolling left, what will the next frame look like?"
- This paper teaches the AI: "If I see a ball rolling left, AND I see a glowing red dot moving with it, what will the next frame look like?"
The "Pixel-Grounded" Secret:
Because the movement is drawn as pixels (dots on a screen) rather than abstract numbers, the AI doesn't need to learn a new language. It already knows how to move pixels because it was trained on millions of videos.
- Analogy: It's like teaching a child to draw a car by having them trace the outline of a real car, rather than giving them a math formula for how wheels turn. The child learns the shape of the movement directly.
The "Zero-Shot" Superpower:
"Zero-shot" means the robot can do a task it has never seen before without any extra training.
- Because the AI learned to move the "glowing dots" in many different camera angles (multiview), it understands the 3D shape of the movement.
- If you show it a new room or a new object, it doesn't panic. It just says, "Okay, I need to move the red dot from here to there," and it figures out the path naturally, just like it would in a video game.

The Decoder: Reading the Map Back

Once the AI generates the video of the glowing dots, how does the robot actually move?
The paper includes a simple "decoder" that looks at the glowing dots in the video and says:

"The red dot is at coordinate X."
"The blue dot is at coordinate Y."
"The blue intensity is low, so the gripper is closed."

It converts the drawing back into the robot's instructions. It's like looking at a treasure map with an "X" and knowing exactly where to dig.

Why This is a Big Deal

No More "Translation" Errors: By keeping the movement in the same "language" (pixels) as the vision, the robot doesn't lose information when translating between seeing and acting.
It's a Swiss Army Knife: The same model can do three things at once:
1. Predict the future: "If I do this, what will the video look like?"
2. Plan the action: "I want this video to happen, what action do I need?"
3. Label the action: "I see this video, what action was performed?"
Generalization: It works better in new environments because it learned the physics of movement visually, not just the math of specific tasks.

In a Nutshell

The authors realized that robotics is just a video game. Instead of building a complex controller to play the game, they taught the AI to generate the gameplay video itself. By turning robot actions into a visual "movie" of glowing dots, they let the AI use its natural video-generation superpowers to control a real robot, making it smarter, more adaptable, and able to learn new tricks instantly.

1. Problem Statement

Current World Action Models (WAMs) aim to predict future observations to learn robot policies. However, a significant gap exists between video generalization (predicting plausible future frames) and policy generalization (deciding how to act in unseen environments).

The Bottleneck: Existing approaches typically rely on separate action modules (policy heads) or non-pixel-grounded action representations (e.g., low-dimensional latent tokens). This forces the model to learn an indirect mapping from video features to control, often causing transfer failures across viewpoints and embodiments.
The Challenge: How to represent robot actions in a way that is native to video generation models, allowing the video backbone itself to act as a zero-shot policy without specialized decoders.

2. Methodology: Action Images

The core innovation is Action Images, a framework that formulates policy learning as multiview video generation. Instead of treating actions as abstract vectors, the method translates 7-DoF robot controls into pixel-grounded, interpretable action images.

A. Action Representation (Action as Images)

The authors convert a 7-DoF action vector $a_t = [p_t, \theta_t, g_t]$ (position, orientation, gripper openness) into a visual representation:

Semantic 3D Points: The action is decomposed into three semantic 3D points:
- Position Point: The end-effector position ( $p_t$ ).
- Normal Point: Derived from the gripper's normal vector.
- Up Point: Derived from the gripper's "up" direction.
Multi-View Projection: These 3D points are projected into 2D image space for multiple camera views using camera intrinsics/extrinsics.
Gaussian Rendering: The projected points are rendered as RGB Gaussian heatmaps:
- Red Channel: Encodes the position point.
- Green Channel: Encodes the normal point.
- Blue Channel: Encodes the up point and the gripper openness (encoded in low-response background regions).
Unified Stream: These "action videos" are temporally concatenated with "observation videos" (RGB frames), creating a unified video-space representation where observation and action share the same modality.

B. Decoding (Action Images to Control)

To execute the policy, the generated action images are decoded back into continuous 7-DoF control:

Gripper State: Extracted directly from the blue channel's low-response regions.
3D Reconstruction: A geometric decoding process is used:
- Select a 2D centroid from the main view heatmap.
- Cast a ray from the camera through this point.
- Sample candidate 3D points along the ray and project them to a side view.
- Select the 3D point that best matches the side-view heatmap response (resolving depth ambiguity).
Vector Recovery: The reconstructed 3D points are converted back into the original position, orientation, and gripper state.

C. Unified World-Action Model Training

The system fine-tunes a large pre-trained video generator (Wan 2.2) to jointly model observations and actions.

Tokenization: Observation and action video clips are encoded into latent space and concatenated temporally.
Multi-Task Masking Strategy: The model is trained under a single objective with diverse masking patterns to support multiple capabilities:
1. Joint Generation: Mask both video and action tokens (predict both).
2. Action-Conditioned Video: Mask video tokens, keep action visible (predict future visuals given actions).
3. Video-to-Action Labeling: Mask action tokens, keep video visible (infer actions from video).
4. Video-Only Generation: Mask action tokens for data without action labels.
Camera Conditioning: Camera poses (Plücker embeddings) are injected to maintain geometric consistency across views.

3. Key Contributions

Pixel-Grounded Action Representation: Proposes translating 7-DoF control into multi-view action images, making action prediction a visual tracking problem rather than an abstract token decoding task.
Zero-Shot Policy without Policy Heads: Demonstrates that a unified video backbone can serve as a direct policy model, eliminating the need for separate action modules and leveraging pre-trained video knowledge directly for control.
Unified Generative Framework: A single model capable of video-action joint generation, action-conditioned video generation, and action labeling, all sharing the same representation.
Improved Generalization: By grounding actions in pixels and using multiple views, the model achieves superior transfer across viewpoints, environments, and robot embodiments compared to prior art.

4. Experimental Results

The model was evaluated on RLBench (simulation) and Real-World (xArm robot) datasets, including unseen objects and environments.

Zero-Shot Policy Performance:
- Outperformed strong baselines (MV-Policy, $\pi_0.5$ , MolmoAct, TesserAct, Cosmos-Policy) significantly.
- RLBench: Achieved 60% success on "Pick Cup" and 50% on "Reach Target" (vs. 0-35% for others).
- Real-World: Achieved 45% success on "Pick Unseen Toy" and 40% on "Place Cup," demonstrating robustness to unseen objects and environments.
Generation Quality:
- Video Metrics: Outperformed baselines in PSNR, SSIM, FVD, and LPIPS.
- Action Accuracy: Achieved lower 2D and 3D trajectory errors compared to TesserAct and Cosmos-Policy.
Additional Capabilities:
- Action-Conditioned Video: Generated higher quality future videos given action inputs compared to Tora.
- Video-to-Action Labeling: Significantly outperformed point-tracking baselines (TAPIR, CoTracker) in inferring action trajectories from video.

5. Significance

This work bridges the gap between generative video models and robotic control. By treating actions as visual data ("Action Images"), the authors enable the direct application of massive pre-trained video knowledge to robotics.

Paradigm Shift: Moves away from "Video Model + Policy Head" to "Video Model is the Policy."
Generalization: The pixel-grounded, multiview approach naturally handles viewpoint shifts and occlusions, addressing a major weakness in current world models.
Versatility: The unified framework simplifies the robotics stack, allowing a single model to predict the future, generate actions, and label data, suggesting a promising path toward generalist robot agents.

Limitations: The current system operates in an open-loop setting (single forward pass without online replanning). Future work aims to integrate this into closed-loop control and accelerate inference via distillation.