OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining
This paper introduces OpenWAM, an open and modular research stack that systematically investigates World-Action Model pretraining through controlled experiments to derive key design principles, culminating in the release of the high-performing OpenWAM- model trained on 6,400 hours of diverse embodied data.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Intelligence is not just about seeing the world; it is about knowing how to change it. For decades, scientists have built computer systems that can recognize objects in a photograph or describe a scene in a sentence. More recently, they have created models that can imagine how a scene might evolve over time, predicting the next frame of a video based on the one before it. These systems learn the physics of the world—how a cup falls, how a door swings—by watching vast amounts of video. However, there is a gap between watching the world and acting within it. A robot needs more than just a prediction of what will happen; it needs to know which specific movements will make that prediction come true. This is the challenge of embodied learning: bridging the gap between understanding a visual future and generating the physical actions to reach it.
A team of researchers has introduced a new approach called OpenWAM to solve this problem. Instead of building a single, rigid robot brain that is difficult to change or understand, they created an open, modular system that treats the design of a robot controller like a set of interchangeable parts. They broke down the complex task of teaching a robot to act into three distinct questions: what knowledge should the robot inherit from watching videos, how should the robot's understanding of the world and its ability to move work together, and how can this learning be scaled across different types of robots and environments? By answering these questions through controlled experiments, they distilled a set of principles that led to the creation of OpenWAM-α, a powerful new model that can learn to manipulate objects in both simulated and real-world settings.
The researchers began by asking what kind of "world knowledge" a robot should start with. They tested whether the robot should learn from a massive video generator that has seen millions of hours of footage, or if it should rely on a simpler visual encoder. They found that the best results came from using a large, pre-trained video generator as a foundation. This model already understands how objects move and interact, providing a rich head start. However, simply attaching a robot arm to this video model was not enough. The researchers discovered that the robot also needed a compact, efficient way to represent what it sees. They tested various methods to compress visual information into a smaller, more manageable form, finding that a specific type of compression that kept the essential details of the world while discarding the noise worked best. This allowed the robot to focus on the important parts of a scene without getting overwhelmed by data.
Next, the team investigated how the robot's "brain" should be structured to connect its understanding of the world with its ability to move. They compared different architectural designs, ranging from models where the video and action parts were completely separate to those where they were deeply intertwined. Their experiments showed that the most effective design was a dual-system approach. In this setup, one part of the model focuses on predicting the future visual state of the world, while a dedicated part focuses on generating the sequence of movements. Crucially, these two parts were allowed to talk to each other constantly. The action generator could look at the predicted future to decide what to do, and the world predictor could use the planned actions to refine its vision of what would happen next. This constant, two-way conversation allowed the model to learn a true synergy between seeing and acting, rather than just learning them side by side.
The researchers also explored how to train these models using different types of data. They had access to two main sources: videos of humans performing tasks from a first-person perspective, which offered a wide variety of visual scenes but no robot action data, and recordings of actual robots performing tasks, which provided precise movement instructions but covered fewer types of scenes. They tested whether it was better to train on robot data alone, human data alone, or a mix of both. They found that combining the two sources in a single training session was the most powerful strategy. The human videos taught the model about the broad diversity of the world, while the robot data grounded that knowledge in physical reality. This combination allowed the model to generalize much better to new, unseen situations than if it had been trained on just one type of data.
Using these principles, the team built OpenWAM-α, a model trained on over 500 million frames of video and robot data. They tested this model across eight different simulation benchmarks and on real robots with single arms, two arms, and even dexterous hands. The results were consistent and impressive. In simulations, the model performed at the top level across a wide range of tasks, from stacking blocks to manipulating complex objects. When moved to the real world, it maintained this high performance, successfully completing tasks on physical robots it had never seen before. The model showed a particular strength in adapting to new environments, suggesting that the combination of video-based world knowledge and action-based grounding created a robust foundation for general intelligence.
One of the most significant findings was how this approach compared to other popular methods. Some systems rely heavily on language and visual understanding but do not explicitly model the future, while others focus purely on the mechanics of movement. The researchers found that their video-based approach excelled at fitting the training data and performing well in familiar settings, while other methods sometimes generalized better to completely new, chaotic environments. However, they concluded that neither approach was perfect on its own. The key to future progress lies in combining the strengths of both: using the rich visual and physical priors of video generation to guide action, while ensuring the system is trained on diverse, high-quality data that covers the full range of real-world challenges.
The OpenWAM project does more than just present a new robot controller; it provides a complete toolkit for the scientific community. By releasing the infrastructure, the training data, and the evaluation protocols, the researchers have turned the development of world-action models into a transparent, reproducible science. This allows other scientists to test specific ideas, swap out components, and understand exactly why certain designs work better than others. The work suggests that the path to more capable robots is not through building larger, more opaque systems, but through carefully understanding how different pieces of knowledge and learning interact. By treating the robot's mind as a collection of composable parts, the researchers have opened the door to a new era of systematic exploration in artificial intelligence, where every design choice can be tested, measured, and improved upon.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.