💻 computer science

Vid2WAM: Distilling Video Diffusion Priors into World Action Models

Vid2WAM is an offline distillation framework that enhances robot policy learning by transferring visual diffusion priors from large video foundation models into compact World Action Models, thereby improving generalization and data efficiency without relying on extensive expert demonstrations.

Original authors: Chenhao Qiu, Ruixiang Wang, Runyi Zhao, Sixu Lin, Songen Gu, Shufeng Nan, Guiliang Liu, Kui Jia, Yanwei Fu, Simo Wu

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenhao Qiu, Ruixiang Wang, Runyi Zhao, Sixu Lin, Songen Gu, Shufeng Nan, Guiliang Liu, Kui Jia, Yanwei Fu, Simo Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to cook a meal. Traditionally, you'd have to hold the robot's hand and physically guide it through every single step—chopping the onions, stirring the pot, flipping the pancake—recording thousands of hours of "expert" demonstrations just to get it to make one perfect omelet. This is the old way of teaching robots: show them exactly what to do, over and over, until they memorize it. But what if the robot could learn by imagining? What if, instead of just watching a human cook, the robot could close its eyes, visualize the entire cooking process in its mind, and then figure out the muscle movements needed to make that vision a reality? This is the exciting frontier of "World Action Models." These are special robot brains that don't just react to the present moment; they predict the future. They ask, "If I do this, what will the world look like in five seconds?" By learning to predict the future, they understand the cause-and-effect of their actions much better than robots that just copy-and-paste human moves. However, there's a catch: these smart "future-predicting" robots usually still need a massive library of real human recordings to learn how to imagine correctly.

Enter Vid2WAM, a new method that acts like a magical shortcut for robot training. The researchers asked a bold question: "Do we really need a human to show us the future, or can we just ask a super-smart video AI to dream it up for us?" They took a giant, pre-trained video model (think of it as an AI that has watched millions of hours of movies and knows how objects move, fall, and interact) and used it as a "teacher." This teacher doesn't need to see the specific robot; it just needs a picture of the starting scene and a sentence like "make a sandwich." The teacher then generates a perfect, imaginary video of what happens next.

Here is where the magic happens. The researchers didn't just use these imaginary videos to show the robot what to do; they used them to teach the robot how to think. They built a system that takes the teacher's imaginary future and breaks it down into two lessons. First, it teaches the robot's "future-brain" to predict the visual changes (the bread getting toasted, the cheese melting). Second, it uses a clever "reverse-engineering" tool (called an Inverse Dynamics Model) to guess what robot arm movements would be needed to create that imaginary video. This creates a set of "pseudo-actions"—fake but highly educated guesses at what the robot should do.

To make sure the robot doesn't get confused by these fake guesses, the team added a special "noise-canceling" filter. They realized that while the teacher's video is great, the "reverse-engineering" tool might make small mistakes. So, they built a system that learns the general rules of movement from real human data, but adds a small, custom "correction layer" for the fake data. This way, the robot learns the solid basics from real humans and the creative, generalizable skills from the AI's imagination, without letting the imaginary errors mess up its real-world performance.

The results are impressive. In simulations and real-world tests with robot arms, this new method allowed the robots to learn new tasks much faster and with far fewer real human demonstrations. When faced with a completely new task they had never seen before—like picking up a specific type of tissue or handling a new object—the Vid2WAM robots succeeded significantly more often than previous methods. They could generalize better, meaning they didn't just memorize a specific path; they understood the idea of the task. Best of all, once the robot learned this way, it didn't need the giant video teacher or the reverse-engineering tool anymore. It became a compact, fast, and efficient robot that could just look at a scene and act, carrying the "wisdom" of the video AI inside its own brain. The paper suggests that by using these powerful video models as offline teachers, we can teach robots to be more creative and adaptable without needing to film millions of hours of expensive human demonstrations.

Technical Summary: Vid2WAM

Problem Statement

World Action Models (WAMs) enhance robot policy learning by jointly modeling future visual dynamics and actions, providing dense spatiotemporal supervision beyond static semantic labels. However, current WAMs face a critical scalability bottleneck: their future supervision is strictly dependent on costly, task-specific expert demonstrations. This reliance limits generalization to distributions covered by recorded trajectories and hinders adaptation to novel tasks where such data is unavailable. While video foundation models possess rich priors regarding object motion and long-horizon dynamics, existing methods often treat generated rollouts merely as sources for pseudo-action labels via Inverse Dynamics Models (IDMs). This approach creates an information bottleneck, discarding the rich predictive knowledge of the video generation process and subjecting the policy to errors from imperfect action recovery.

Methodology: Vid2WAM

The authors propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. The framework operates in three stages:

1. Teacher Adaptation and Rollout Generation

A large, pre-trained video diffusion model (the "teacher") is fine-tuned on a limited set of embodiment-specific demonstration videos. This adaptation specializes the teacher to the robot's physical constraints and appearance while preserving its general motion prediction capabilities. Given an initial observation and language instruction, the frozen teacher generates task-conditioned future visual rollouts ( $\hat{v}_{1:T}$ ).

2. Dual-Channel Supervision

Vid2WAM distills the teacher's knowledge into the student through two complementary channels:

Future Visual Dynamics: The generated video rollouts are decoded to pixels and re-encoded into the student's latent space ( $\hat{z}_{1:T}$ ). These latents provide direct supervision for the student's future prediction branch, allowing the model to learn the temporal evolution of the scene without relying on real future frames.
Embodiment-Specific Actions: An Inverse Dynamics Model (IDM), trained on action-labeled trajectories, recovers pseudo-actions ( $\hat{a}_{1:H}$ ) and state sequences from the generated rollouts. These serve as supervision for the student's action prediction branch.

3. Source-Aware Residual Adaptation

To address the noise inherent in synthetic pseudo-actions (stemming from video generation artifacts and IDM errors), Vid2WAM introduces source-aware residual action adapters.

The student shares a common video and action backbone for both real demonstrations and pseudo-data.
Lightweight residual adapters are placed before and after the shared action backbone, with separate parameters for the "real" and "pseudo" domains.
These adapters learn source-specific corrections around the shared representation. During inference, only the real-domain adapter is retained, effectively filtering out the noise associated with the pseudo-action distribution while preserving the shared knowledge.

Inference

At deployment, the video teacher, IDM, future prediction head, and pseudo-source adapters are discarded. The system relies solely on the compact WAM student, which takes a current observation and language instruction to directly generate executable action chunks, ensuring low-latency inference.

Key Contributions

Rethinking Supervision Sources: The paper challenges the assumption that future supervision for WAMs must originate from target-task expert trajectories. It formulates a distillation framework using generated futures from video foundation models as direct supervision.
Dual-Channel, Source-Aware Transfer: The authors introduce a strategy that jointly distills future visual dynamics and IDM-inferred actions. Crucially, they employ residual adapters to mitigate interference from noisy pseudo-actions, allowing the model to leverage synthetic data without compromising robustness.
Efficient Deployment: The framework demonstrates that high-quality future priors can be distilled offline, enabling the student model to achieve strong generalization and data efficiency while maintaining the low-latency inference required for real-time robot control.

Experimental Results

The authors evaluate Vid2WAM on simulation benchmarks (RoboTwin 2.0, LIBERO, LIBERO-Plus) and a real-world bimanual platform.

Novel Task Generalization: In the "novel regime" (where no expert trajectories exist for target tasks), Vid2WAM significantly outperforms baselines. On RoboTwin 2.0, it improved success rates on novel tasks by 9.7 and 12.5 percentage points over Fast-WAM under clean and randomized conditions, respectively. On LIBERO, it achieved the highest average success rate among all methods.
Data Efficiency: Under a low-data regime (limited expert demonstrations), Vid2WAM consistently outperformed pretrained VLA models (e.g., $\pi0.5$ , Motus) and other WAM baselines, demonstrating superior ability to learn from scarce real data augmented by synthetic supervision.
Robustness: Vid2WAM showed improved robustness against perturbations in camera views, robot initial states, language, lighting, and background noise, particularly in the LIBERO-Plus benchmarks.
Real-World Performance: On a real-world dual-arm robot, Vid2WAM successfully executed novel tasks (e.g., picking a test tube, taking tissue) where it had no real action-labeled trajectories, relying instead on teacher-generated supervision.
Inference Latency: The distilled student model maintains inference latency comparable to Fast-WAM (~212ms on RTX 4090) and is substantially faster than online teacher policies or larger models like Motus.

Significance and Claims

The paper claims that Vid2WAM establishes a scalable paradigm for transferring the predictive capabilities of large multimodal foundation models into efficient robot policies. By decoupling future supervision from the need for exhaustive target-task demonstrations, the method enables robots to adapt to novel tasks and environments with limited real-world data. The authors emphasize that their approach does not require online video generation during deployment, making it practical for real-world robotic systems. The results suggest that large pretrained models can serve as effective offline knowledge sources, bridging the gap between general visual understanding and specific robotic control.