VisualPatchWorld: Code World Models as Latent Structured Representations for Planning
VisualPatchWorld introduces a novel approach to constructing world models by representing dynamics as executable code, which is selected via qualitative probing and fitted from data to enable interpretable, editable, and highly effective planning that outperforms existing code-based baselines.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot how to navigate a messy room, pick up a cup, and pour a drink without spilling. To do this, the robot needs a "world model"—a mental map of how the world works. Think of this like a video game engine running inside the robot's brain. If the robot wants to know what happens if it pushes a chair, it can't just guess; it needs to simulate the push in its head first.
For a long time, scientists have tried to build these mental maps in two very different ways. The first way is like training a super-smart but mysterious student. You show the robot thousands of videos of things happening, and it learns to predict the future by finding patterns in a hidden, mathematical space. It's great at guessing, but if it makes a mistake, no one knows why—it's like a black box where the rules are invisible. The second way is like building a detailed, manual blueprint. Engineers write out every single law of physics (gravity, friction, collisions) in code. This is very clear and easy to fix, but it's incredibly hard to write a blueprint for every possible room or object in the world. The big question is: Can we get the best of both worlds? Can we teach a robot to learn the rules of the world from watching videos, but end up with a clear, readable set of instructions instead of a mysterious black box?
This is exactly what the paper "VisualPatchWorld" tackles. The researchers, Jiaxin Bai and Jiaxuan Xiong, propose a new method that treats the world's rules as code. Instead of letting a robot guess the future in a hidden math space, they teach it to write a simple, executable program that describes how objects move. Imagine asking a robot to watch a video of a ball rolling down a ramp and then asking it to write a short Python script that says, "If the ball is on a slope, add speed to its movement."
The team's approach works in two clever steps. First, the robot runs a few quick, "active probes"—like a detective testing a theory by poking a suspect. It tries a few different types of movement rules (e.g., "Does the object slide like it's on ice, or does it stick like it's on sand?") to figure out the qualitative shape of the rule. Once it picks the right "sketch" of the rule, the second step is to fill in the specific numbers (like exactly how fast it slides) by watching recorded videos of the robot moving. The result is a piece of code that the robot can run like a mini-simulator. It can look at the code, understand it, and use it to plan its next moves.
The results are quite promising. When tested on four different tasks—like navigating a maze, reaching for an object, pushing things, and stacking cubes—their method, called VisualPatchWorld (VPW), achieved a 69.0% success rate in planning. This is a significant jump, beating the previous best code-based methods by 23.5 percentage points. The biggest improvements happened in tasks where knowing the type of movement was crucial. For example, in a task where a robot arm had to reach for something, the old methods only succeeded 8% of the time, but VPW jumped to 72%. In a task involving pushing a block, previous code methods barely worked at all (near zero success), while VPW managed 22%.
However, the paper is careful to note that this isn't a perfect solution for everything yet. While the robot's self-written code was almost as good as a perfect physics engine for navigation and grasping, it still struggled a bit with complex pushing tasks where things collide and bounce. To fix this, the authors suggest a "hybrid" approach: let the robot's code do the heavy lifting of planning, but then double-check the top few best plans with a perfect physics engine before the robot actually moves. This small check closes most of the remaining gap.
Ultimately, this paper suggests that we don't have to choose between a mysterious AI that guesses and a rigid engineer who writes every rule by hand. By teaching robots to learn the structure of the world first and then fill in the details, we can create world models that are not only powerful enough to plan complex actions but also clear enough for humans to read, understand, and fix if they go wrong. It's a step toward robots that don't just "know" how to move, but can actually explain the rules of the game they are playing.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.