🤖 AI

AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction

AcrossVAM1.0 is a lightweight, text-assisted video action model that improves robot video prediction by factorizing future frames into object-centric particle dynamics and dense appearance, demonstrating that explicit particle modeling reduces trajectory error despite current limitations in language grounding and perceptual quality.

Original authors: Yafei Zhang, Nan Wu

Published 2026-08-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yafei Zhang, Nan Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To understand how a robot might learn to see the future, we must first understand how it sees the present. In the world of physical artificial intelligence, a robot does not merely process a static image; it must anticipate how that image will change when it interacts with the world. This is the challenge of video prediction: teaching a machine to look at a scene and guess what will happen in the next few moments. For a human, this is a natural instinct. If you see a cup on the edge of a table, you know it will fall if pushed. For a robot, however, this requires a complex internal simulation. It must separate the parts of the scene that will move from the parts that will stay still, and it must understand that a simple command like "pick up the blue cube" can lead to many different visual outcomes depending on the robot's arm and the environment. The goal is not just to create a pretty picture of the future, but to create a reliable map of motion that a robot can use to plan its actions safely and effectively.

Researchers at Across Physical AI and the Chinese Academy of Sciences have taken a new approach to this problem with a system they call AcrossVAM1.0. Instead of trying to predict every single pixel of a video frame all at once, which often leads to blurry or confused results, they broke the task down into two distinct jobs. They realized that in a typical robot video, most of the image is actually static. The table, the wall, and the floor do not move; only the robot's arm, its gripper, and the object it is holding change position. The team designed a model that treats these moving parts as distinct, semantic particles. Imagine the robot's arm and gripper not as a continuous stream of pixels, but as a few specific, tracked objects with their own positions, sizes, and speeds. The model focuses its computational power on predicting exactly how these few moving parts will travel through space, while treating the rest of the scene as a stable background.

The system works by first looking at four frames of video and a written instruction, such as "pick the blue cube." It uses a pre-trained, frozen vision system to identify the robot, its arm, and its gripper, turning them into simple data points that describe where they are and how big they are. A small, efficient brain, containing only 0.28 million trainable parameters, then takes these data points and calculates where they will be in the next five moments. This part of the model is strictly focused on motion and is guided by the text instruction, ensuring that the predicted movement actually corresponds to the command given. Once the model has figured out the path of the moving parts, a second, separate process fills in the details. It takes the last known frame of the video and uses it to restore the fine textures and colors of the static background, ensuring that the wall and the table look sharp and real. Finally, a smart blending mechanism combines the predicted motion of the robot with the high-quality static background, creating a complete video of the future.

The results of this approach show a clear trade-off between motion and image quality. When the researchers tested the model on a benchmark of real robot movements, the particle-based prediction significantly improved the accuracy of the motion itself. The error in the path of the robot's arm dropped by 21.0 percent compared to a simple method that just assumes nothing will move. The model successfully predicted the trajectory of the arm and gripper with much greater precision than previous methods that tried to guess the whole image at once. However, the model is not perfect. While it excels at moving the robot parts correctly, it still struggles slightly with the overall visual quality of the final image. In tests measuring how similar the predicted video looks to the real video, the new model scored slightly lower than the simple method that just copies the last frame, particularly in how it handles the subtle textures of the scene. The researchers found that while the model can follow the text instruction to move the robot, the connection between the language and the specific path taken is still weak; changing the instruction only slightly altered the predicted path in most cases.

This work highlights a fundamental insight about teaching robots to see: it is often better to understand the movement of specific objects than to try to predict every pixel of a scene simultaneously. By separating the job of predicting motion from the job of restoring image detail, the researchers created a system that is lightweight and interpretable. They can look at the internal "particles" and see exactly what the robot thinks is moving and where it thinks it is going. While the system is not yet ready to replace all existing methods, as it still needs to improve its ability to generate photorealistic images and strictly obey complex language commands, it offers a promising new direction. It suggests that the future of robot vision may lie in simple, structured models that understand the physics of moving parts, rather than massive, complex systems that try to memorize every detail of a scene. The path forward involves refining how these two streams of information—the moving parts and the static background—are combined, and finding ways to make the robot's understanding of language more robust and reliable.

Technical Summary: AcrossVAM1.0

Problem Statement

Predicting robot videos presents a unique challenge: the majority of pixels in a scene are static, while the critical information lies in the motion of small articulated parts (e.g., robot arms and grippers). Monolithic pixel-based models often fail to learn the necessary motion dynamics because they can achieve competitive global image metrics simply by copying the last observed frame. Furthermore, existing world models typically entangle motion reasoning, object correspondence, and high-frequency texture synthesis within a single dense latent space, making it difficult to isolate and improve specific components. Additionally, conditioning video models on natural language does not guarantee that the predicted trajectory actually depends on the instruction; models may rely on scene priors or dataset biases instead.

Methodology

AcrossVAM1.0 proposes a text-assisted, object-centric video action model that factorizes future prediction into two distinct streams: semantic particle dynamics and causal appearance synthesis.

1. Semantic Particle Codec

The model utilizes a frozen SAM3-DLP codec to decompose context frames into semantic representations. Instead of processing the full image, it extracts:

Three semantic particles: Representing the whole robot, the arm, and the gripper.
A background latent: Capturing static scene elements.
Particle State: Each particle $z_t$ encodes position ( $p$ ), scale ( $s$ ), depth ( $d$ ), presence ( $q$ ), and appearance ( $a$ ).
Identity Alignment: For unordered proposals, the model employs Hungarian matching based on position and appearance similarity to ensure temporal correspondence across frames.

2. Text-Conditioned Dynamics

A lightweight Spatio-Temporal Transformer (0.28M trainable parameters) predicts the future states of these particles.

Input: Concatenated geometry, velocities, depth, presence, and learned embeddings.
Language Conditioning: A frozen OpenCLIP encoder processes the natural language instruction. Its embeddings modulate the Transformer's internal layers via FiLM (Feature-wise Linear Modulation). This design ensures that language influences only the dynamics core, making its effect measurable at the trajectory level.
Training: The model is trained using a combination of one-step transition loss and autoregressive rollout loss.

3. Causal Motion–Appearance Decoder

To reconstruct high-fidelity video without leaking future information, the decoder operates in two parallel streams:

Motion Stream: A frozen renderer maps predicted particles and the background latent into a structural RGB proposal and an alpha map. This captures the correct geometry and motion but lacks fine texture.
Appearance Stream: A dense encoder extracts features exclusively from the last observed context frame (never future frames). A conditional residual prior predicts future details based on these context features and the particle motion.
Fusion: A learned confidence mask blends the generated motion/detail with the original context frame (persistence). This allows the model to preserve static regions while synthesizing moving parts.

Key Contributions

Semantically Anchored Particle World Model: The authors introduce a model that reduces complex robot video dynamics to a compact trajectory problem. The dynamics core is extremely lightweight (0.28M parameters) and exposes part-level motion for diagnosis.
Causal Motion–Appearance Decoder: A novel architecture that combines predicted particle geometry with context-only dense appearance. This explicitly controls the trade-off between sharp static regions and moving-object fidelity, preventing future appearance from leaking into the prediction.
Rigorous Evaluation Framework: The paper evaluates the model against multiple baselines, including persistence, shuffled-language controls, random labels, oracle particles, and per-robot splits. This decomposition isolates the reliability of motion modeling from the limitations of language grounding and appearance delivery.

Experimental Results

The model was evaluated on the VRS (Video Robot Segmentation) benchmark, constructed from diverse real-robot trajectories (Franka, WidowX, Mobile ALOHA, etc.).

Motion Prediction: Particle dynamics reduced trajectory error by 21.0% over a persistence baseline.
Image Quality: Across three delivery-mask seeds, AcrossVAM1.0 improved global PSNR from 19.97 to 20.573 and SSIM from 0.796 to 0.8004.
Motion-Region Quality: Raw particle generation improved motion-region PSNR from 11.89 to 13.23. The final delivered model maintained a motion-region PSNR of 12.87, significantly outperforming persistence (11.89).
Limitations:
- LPIPS: The model did not beat persistence in perceptual similarity (0.1304 vs. 0.122), indicating that static region fidelity is still imperfect.
- Language Grounding: Correct-versus-shuffled language changed trajectory error by only 2.8–3.1%, suggesting the model is "text-assisted" rather than fully "instruction-controlled."
- Generalization: Performance varied across robot platforms, with significant gains on Franka but degradation on Mobile ALOHA in leave-one-robot-out experiments.

Significance and Claims

The paper posits that explicit particle dynamics serve as a promising low-dimensional interface for robot video prediction, successfully separating motion reasoning from texture synthesis. By freezing perception and language encoders, the authors demonstrate that a small, trainable dynamics core can effectively model robot motion.

However, the authors are modest about the current state of the system. They explicitly state that robust language grounding and appearance delivery remain the principal open challenges. The model does not yet achieve general-purpose action control, and the gap to the "pixel-selection oracle" (which chooses the best pixel between generation and persistence) highlights that deciding where to synthesize remains a bottleneck. The work provides a foundation for interpretable video action models that do not hide failure modes behind aggregate image metrics, but it acknowledges that significant work is needed to achieve reliable cross-robot transfer and strong instruction dependence.