← Latest papers
💻 computer science

ZZ-sight: Unifying Vision-Language-Action and Latent World Modeling

ZZ-sight is a unified robot foundation model that integrates vision-language-action control with latent world modeling by predicting future spatial tokens and actions directly within the VLM's native token space, achieving state-of-the-art performance on diverse benchmarks without relying on pixel reconstruction or external encoders.

Original author: Junjie Ye, Yiqi Zhao, Muchen Xu, Celina Shiyu Wang, Hao Dong, Ruohai Ge, Rong Xue, Benjamin J Kolligs, Aditya Dutt, Sasanka Nagavalli, Nassim A. Samad, Gaurav S. Sukhatme, Jyotirmoy V. Deshmukh, Yue W
Published 2026-11-19
📖 6 min read🧠 Deep dive

Original author: Junjie Ye, Yiqi Zhao, Muchen Xu, Celina Shiyu Wang, Hao Dong, Ruohai Ge, Rong Xue, Benjamin J Kolligs, Aditya Dutt, Sasanka Nagavalli, Nassim A. Samad, Gaurav S. Sukhatme, Jyotirmoy V. Deshmukh, Yue Wang, Yuliang Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled to move through the world with the fluid intuition of a human. While they can be programmed to repeat a specific motion perfectly, they often fail when the environment changes slightly or when a task requires a sequence of steps that must adapt in real time. To bridge this gap, scientists have developed systems that combine vision, language, and action. These systems allow a robot to understand a command like "put the cup on the table" and then figure out how to move its arm to do it. However, a significant hurdle remains: most of these systems only look at the present moment. They see the current scene and decide on the next move, but they do not truly anticipate what will happen next. Without this ability to foresee the immediate future, robots can stumble when a task requires coordinating complex movements, such as walking while carrying an object or navigating around obstacles that shift as they move.

A new approach called Z-sight, developed by researchers at Maven Robotics and the University of Southern California, attempts to solve this by giving the robot a form of "foresight." Instead of just reacting to what it sees right now, the system learns to predict what the visual world will look like a fraction of a second into the future, and it uses that prediction to guide its actions. This is not about generating a full video of the future, which would be computationally expensive and slow. Instead, the system predicts the essential visual patterns that the robot's brain uses to understand the scene. By learning to imagine the next moment while simultaneously deciding on the next move, the robot can plan its path more smoothly, handling complex tasks that previously caused other systems to freeze or fail.

The researchers built this system by training it on a massive collection of data, including thousands of hours of videos showing humans performing everyday tasks and recordings from various robots. The training process was divided into stages. First, the system learned to understand human movements and language from videos of people doing things like opening doors or folding clothes. Then, it was exposed to data from different types of robots, learning to translate those human-like observations into the specific mechanics of robotic movement. Finally, the system was fine-tuned on a specific robot, a humanoid machine called the Unitree G1, to master tasks like pushing a chair under a table or throwing trash into a bin. Throughout this process, the system was taught to do two things at once: predict the future visual state of the world and generate the physical actions needed to reach that state.

The core innovation lies in how the system handles these predictions. Many previous attempts to give robots foresight relied on generating full video frames of the future or using separate, frozen systems to guess what would happen. These methods were either too slow or disconnected from the robot's own way of seeing the world. Z-sight avoids these pitfalls by predicting the future directly in the same "visual language" the robot uses to understand the present. It does not try to recreate a pixel-perfect video; instead, it predicts the abstract visual tokens that represent the scene. This allows the robot to see the future as a set of possibilities that are tightly integrated with its current understanding, enabling it to adjust its actions on the fly as the predicted future unfolds.

The results of this approach were tested in both computer simulations and real-world experiments. In a simulated environment designed to test whole-body coordination, the system achieved an 81.9 percent success rate on a variety of complex tasks, outperforming the strongest existing methods by a significant margin. When tested on the physical Unitree G1 robot in a real room, it succeeded in 73.3 percent of trials across six different tasks, which included moving objects, navigating around furniture, and performing multi-step chores. In comparison, the next best system succeeded in only 53.3 percent of these real-world attempts. The researchers found that the system's ability to anticipate the future was crucial for tasks that required a sequence of movements, such as walking to a chair, pushing it, and then placing an object on top. Without this foresight, the robot would often complete the first part of the task but then stall when trying to transition to the next step.

To ensure that the improvements came from the foresight mechanism itself and not just from having a larger or more complex model, the researchers conducted several controlled experiments. They tested versions of the system that could not predict the future and found that these versions performed significantly worse, even when they had the same amount of computing power. They also tested whether the system relied on the specific content of its predictions or just the fact that it was making a prediction. When they replaced the predicted future with random noise or with the current static image, the robot's performance collapsed, indicating that it was genuinely using the specific details of its future prediction to guide its movements. Furthermore, the system managed to make these complex predictions and decisions incredibly fast, processing a chunk of 30 steps in just 110 milliseconds, which is fast enough to control a robot in real time without lag.

The study also explored how the system behaves when things go wrong. By intentionally disrupting the prediction stream during testing, the researchers observed that the robot could adapt. If the system was told that its prediction was unreliable, it would rely more heavily on what it could see right now, rather than guessing what might happen. This flexibility suggests that the system has learned a robust way to balance planning with reacting. The researchers noted that while the system is highly effective, it currently predicts the future based only on what it sees, without explicitly testing different action choices before committing to one. This leaves open the possibility for future versions to compare different potential outcomes before acting, further enhancing its ability to plan.

Ultimately, Z-sight demonstrates that a robot does not need to generate full videos of the future to understand what comes next. By learning to predict the future in the same abstract visual space it uses to understand the present, the system achieves a level of coordination and adaptability that was previously difficult to attain. This approach allows the robot to move with a sense of continuity, bridging the gap between seeing the world and acting within it. The success of this method on both simulated and real humanoid robots suggests that this way of combining perception, prediction, and action could become a standard foundation for more capable and autonomous machines in the future.

Drowning in papers in your field?

Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.

Try Digest →