💻 computer science

RepWAM: World Action Modeling with Representation Visual-Action Tokenizers

RepWAM introduces a representation-centric world action model that utilizes a novel visual-action tokenizer to learn aligned semantic latent spaces, enabling superior instruction-following and closed-loop robot manipulation performance compared to traditional reconstruction-oriented approaches.

Original authors: Junke Wang, Qihang Zhang, Shuai Yang, Yiming Luo, Yujun Shen, Zuxuan Wu, Yu-Gang Jiang, Yinghao Xu

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junke Wang, Qihang Zhang, Shuai Yang, Yiming Luo, Yujun Shen, Zuxuan Wu, Yu-Gang Jiang, Yinghao Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do chores, like picking up fruit or opening a drawer. To do this, the robot needs a "brain" that can understand what it sees and figure out what movements to make next. This paper introduces a new kind of brain called RepWAM.

Here is the simple breakdown of how it works, using everyday analogies:

The Problem: The "Pixel-Perfect" vs. "Meaningful" Gap

Most current robot brains are built on top of video generators (like AI that makes fake videos). These generators are obsessed with pixel perfection.

The Old Way: Imagine a student trying to learn how to drive by staring at a photo of a car. They memorize the exact color of the paint, the texture of the road, and the shadows. But when they get behind the wheel, they don't understand how the steering wheel turns the car or why the car moves. They are stuck on the "look" of the world, not the "logic" of it.
The Issue: Existing robot models try to predict the future by guessing exactly what every pixel will look like. This is too much detail and misses the important stuff: What object is moving? Is the drawer opening? Is the fruit being picked?

The Solution: A "Semantic" Translator

The authors created RepWAM, which uses a special translator called a Visual-Action Tokenizer. Think of this as a translator that converts the robot's raw camera feed into a language of "meaning" rather than "pixels."

The Visual Translator (The "What"):
Instead of just copying the image, this part of the system looks at the video and asks, "What is the main story here?" It compresses the video into semantic tokens.
- Analogy: Instead of describing a picture of a cat by listing the color of every fur strand, it says, "Cat, sitting on a mat, looking left." It keeps the important identity and relationships but drops the unnecessary noise.
The Action Translator (The "How"):
This is the clever part. The system doesn't just learn to see; it learns to see changes. It creates Latent Action Tokens.
- Analogy: Imagine a flipbook. The old way tried to draw every single frame perfectly. RepWAM learns the "arrow" that turns page 1 into page 2. It learns that "pushing the drawer" is a specific transition that moves the state from "closed" to "open." It treats actions as the bridge between two meaningful pictures.

How They Work Together

RepWAM puts these two translators into a single room.

The Training: The robot watches videos and learns to predict: "If I see this (meaningful picture) and I do this (meaningful action), the next picture will look like that."
The Result: Because the robot is thinking in terms of "objects and movements" rather than "pixels and colors," it gets much better at following instructions like "Pick up the fruit" or "Push the drawer."

The Results: Does it Work?

The authors tested this on real robots and in simulations:

Real World: On a dual-arm robot, RepWAM successfully picked up fruit, pushed drawers, and inserted tubes into racks much better than previous models.
Simulation: In a complex video game-like simulation (RoboTwin 2.0), it scored very high on difficult tasks, even beating models that were pre-trained on massive video datasets.
The Key Finding: The paper shows that you don't need to memorize the exact pixels to be a good robot. You need to understand the semantic story of the scene. By aligning what the robot sees with what it does in this "meaningful" space, the robot becomes much more reliable.

In a Nutshell

Previous robot models were like artists trying to copy a painting stroke-by-stroke to understand how a scene works. RepWAM is like a director who understands the plot, the characters, and the actions. By focusing on the story of the movement rather than the texture of the pixels, the robot learns to act more intelligently and follow instructions more accurately.

Technical Summary: RepWAM – World Action Modeling with Representation

1. Problem Statement

World Action Models (WAMs) aim to bridge the gap between passive future prediction and embodied robot control by learning a single latent space that forecasts visual dynamics and infers the robot actions required to realize them. However, current WAMs face two critical limitations in their representation design:

Visual Representation Deficiencies: Existing WAMs typically inherit video tokenizers from pretrained video generation models (e.g., WAN, VQ-VAE). These tokenizers are optimized for pixel reconstruction, dedicating latent capacity to low-level appearance (background textures) rather than task-relevant semantics (object identity, spatial relations, interaction cues). Consequently, instruction-conditioned world modeling is poorly grounded, as language goals must be inferred from a latent space dominated by appearance variation.
Action-Visual Decoupling: Visual latents and motor commands reside in disjoint spaces. This forces an Inverse Dynamics Model (IDM) to bridge a significant modality gap at every step, preventing the action expert from effectively grounding motor commands in scene content.

The authors posit that the fidelity of WAMs is bottlenecked by these latent spaces and ask: What kind of representation does a world action model actually need?

2. Methodology: RepWAM

The paper proposes RepWAM, a representation-centric WAM built upon a Representation Visual-Action Tokenizer (RepViTok). The framework consists of three core components:

A. Representation Visual-Action Tokenizer (RepViTok)

RepViTok aligns visual latents with a frozen visual foundation model and induces latent action tokens as transitions within this shared semantic space.

Visual Tokenization: A Vision Transformer (ViT) autoencoder processes visual observations. Unlike standard tokenizers, it is supervised by a dual objective:
1. Reconstruction Loss: Combines $L_1$ , perceptual, and adversarial losses to ensure pixel fidelity.
2. Feature Alignment Loss: Pulls video latents toward a frozen visual foundation model (e.g., Perception Encoder) via a linear projection. This ensures the latent space captures high-level semantics (object identity, interactions) rather than just low-level appearance.
Latent Action Tokenization: Building on the aligned visual tokens, a Latent Action Tokenizer (LAT) learns action tokens that represent transitions between visual states. It couples an Inverse Dynamics Model (IDM) and a Forward Dynamics Model (FDM):
- The IDM compresses the transition between consecutive latent frames ( $z_t, z_{t+1}$ ) into a compact latent action $\ell_t$ .
- The FDM realizes $\ell_t$ as a transport map (routing visual content) and a residual, reconstructing the next visual latent.
- This design ensures latent actions are defined as transformations between semantic visual states, making them transferable across embodiments.

B. Causal World Action Model

RepWAM pretrains a causal diffusion transformer to jointly model future visual states and latent actions under language conditioning.

Architecture: The model uses a block-causal mask where visual and action tokens share attention weights but utilize modality-specific feed-forward networks (FFNs).
Training Objective: The model is trained using a conditional flow-matching objective. It samples a chunk of visual latents and paired latent actions, interpolating them with Gaussian noise. The network regresses the target velocity to predict both the next visual state and the corresponding latent action simultaneously.
Adaptation: After pretraining on large-scale video-action data, the model is adapted to real robot trajectories for closed-loop manipulation.

3. Key Contributions

Semantic Visual-Action Tokenization: The introduction of RepViTok, which aligns visual latents with a foundation model and learns latent actions as transitions within this semantic space, addressing the "shallow" and "decoupled" nature of existing WAM representations.
Joint Modeling Framework: A unified architecture that pretrains a causal diffusion transformer to jointly forecast future visual states and latent actions, followed by adaptation to real-world robot control.
Empirical Validation: Comprehensive evaluation on real-world dual-arm manipulation tasks and the RoboTwin 2.0 simulation benchmark, demonstrating that semantic alignment improves both visual dynamics prediction and closed-loop execution.

4. Experimental Results

The authors evaluated RepWAM (1.3B and 5B parameter scales) against baselines including $\pi_0.5$ , Motus, and Lingbot-VA.

Real-World Manipulation (Franka Dual-Arm):
- On tasks like "Pick the fruit," "Push the drawer," and "Insert the tube," RepWAM-5B achieved success rates of 60%, 80%, and 60% respectively.
- These results outperformed $\pi_0.5$ (by 10–50 percentage points) and Lingbot-VA (by 10–20 percentage points), particularly on long-horizon and fine-grained tasks where coherent visual dynamics are crucial.
Simulation Benchmarks (RoboTwin 2.0):
- RepWAM-5B achieved 89.3% on Easy tasks and 88.4% on Hard tasks, outperforming the 50-task average of $\pi_0.5$ and Motus.
- Notably, RepWAM was trained from scratch without inheriting pretrained video generation weights, yet it remained competitive with models like Lingbot-VA that utilize such pretraining.
Ablation Studies:
- Tokenizer Design: Replacing the standard WAN2.2 VAE with RepViTok improved Open-Loop Score (OLS) and real-robot success rates, confirming that semantic alignment is critical for instruction-following.
- Latent Action Training: A "Two Stages" approach (pretraining on latent actions, then adapting to robot control) significantly outperformed "Joint Prediction" or training without latent actions, validating the utility of latent actions as an intermediate representation.
- Classifier-Free Guidance (CFG): Unlike video generation models, RepWAM achieved optimal performance with a CFG scale of 1.0 (no extrapolation), suggesting the semantic space is robust enough to reduce reliance on CFG.

5. Significance and Claims

The paper claims that representation visual-action tokenization is a promising foundation for world action models. By grounding visual and action latents in a shared semantic representation, RepWAM demonstrates that:

Explicit future generation is not strictly necessary for policy learning if the representation captures task-relevant semantics.
Semantic alignment allows WAMs to achieve strong generalization and closed-loop performance even without inheriting pretrained video generation backbones.
This approach moves toward generalist robot policies by creating a representation space where action is understood as a transformation of semantic states rather than a disjoint motor command.

The authors conclude that the design of the latent space is fundamental to the capability of embodied agents, and RepWAM offers a step toward more interpretable, transferable, and effective robot manipulation. Future work is suggested to scale pretraining beyond robotics-specific videos to include large-scale internet and egocentric human videos.