🤖 AI

FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models

FoMoVLA is a novel framework that enhances Vision-Language-Action models by jointly learning future feature foresight and sparse 2D point tracking to bridge the gap between goal prediction and continuous motion guidance, achieving state-of-the-art performance and strong zero-shot generalization on multiple robotic benchmarks.

Original authors: Wei Li, Peijin Jia, Yuan Ma, Xuefeng Jiang, Titong Jiang, Sheng Sun, Yujian Li, Xin Wen, Han Hong, Zhikang Liu, Bailin Li, Kun Zhan

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Wei Li, Peijin Jia, Yuan Ma, Xuefeng Jiang, Titong Jiang, Sheng Sun, Yujian Li, Xin Wen, Han Hong, Zhikang Liu, Bailin Li, Kun Zhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to make a sandwich. You show it a picture of the ingredients and say, "Make me a turkey club." A standard robot brain might look at the picture, guess the next move, grab the bread, and hope for the best. It's like playing a game of "hot or cold" where you only react to what's happening right this second. But real life is a movie, not a slideshow. To do complex tasks, you need to imagine the future: "If I grab the turkey now, the bread will slide, and I'll need to catch it." This is the challenge of Vision-Language-Action (VLA) models. These are the smart brains trying to turn what a robot sees and hears into physical moves. The big question scientists are asking is: How do we teach a robot to not just react, but to foresee? How do we make it understand not just where an object needs to go, but how it gets there?

Enter FoMoVLA, a new framework that acts like a "crystal ball" for robot brains. The researchers argue that existing methods are missing a crucial piece of the puzzle. Some methods try to predict the entire future picture pixel-by-pixel, which is like trying to predict every single raindrop in a storm—it's too much data and too slow. Others just guess the final destination, like saying "the sandwich will be on the plate," but forgetting the messy journey of getting there. FoMoVLA fixes this by teaching the robot two complementary skills at once: Visual Foresight (imagining the final state of the scene) and Motion Guidance (tracking the specific path objects take to get there). Think of it as teaching a robot to look at a destination on a map (the foresight) while simultaneously watching a GPS dot trace the winding road to get there (the motion). By combining these, the robot learns a much clearer, more physical understanding of how to move, leading to better performance on tricky tasks like stacking blocks or picking up objects, all without slowing down the robot when it's actually working.

The Problem: Robots That Can't See the Future

Most robot brains today are incredibly reactive. They see a cup, they grab it. They see a door, they push it. But they struggle with the "what if" questions. If a robot tries to move a heavy box, it needs to know that the box might tip over before it tips over. Current methods try to solve this by either predicting the entire future video frame-by-frame (which is computationally heavy and full of useless static details like the background wall) or by just predicting the final goal state (which tells the robot where to go but not how to get there).

The authors of this paper suggest that these two approaches are actually best friends that need to be introduced. You need the "goal" (the final picture) and the "path" (the movement). If you only have the goal, the robot doesn't know the physics of the journey. If you only have the path, the robot might get lost without a destination.

The Solution: FoMoVLA's Two-Pronged Approach

FoMoVLA (Foresight and Motion VLA) is a training framework that teaches a robot to do both simultaneously. It doesn't change the robot's brain permanently; instead, it adds a special "training mode" where the robot learns two extra things alongside its main job of moving.

The Crystal Ball (Future Feature Prediction): The robot is taught to imagine what the scene will look like after it finishes its task. Instead of trying to redraw the whole future picture (which is hard), it learns to predict the "vibe" or the key features of the future state. It's like looking at a puzzle and predicting what the final picture will look like without drawing every single piece.
The GPS Tracker (Sparse Point Tracking): Simultaneously, the robot learns to track specific dots on objects as they move. Imagine putting a tiny sticker on a cup and watching it slide across the table. The robot learns to predict exactly where that sticker will be in the next few seconds. This teaches the robot the "how" of the movement—the continuous flow of action.

The Secret Sauce: Connecting the Dots

The real magic happens in how these two skills talk to each other. In many previous attempts, robots learned these skills separately, like two students studying in different rooms. FoMoVLA forces them to collaborate using a special module called FCCA (Future-Conditioned Cross-Attention).

Here's how it works: The "Crystal Ball" (the future prediction) tells the "GPS Tracker" (the motion prediction) what the destination looks like. The tracker then uses that information to figure out the best path to get there. It's like a driver (the tracker) who knows the destination (the foresight) can navigate much better than a driver who is just guessing where to turn. The paper shows that when these two are linked, the robot's predictions become much more accurate and consistent.

The Results: Smarter Robots, No Extra Cost

The researchers tested FoMoVLA on several challenging robot tasks, including the LIBERO suite (a collection of tasks like stacking blocks and moving objects) and the RoboCasa dataset (simulating a home environment with a humanoid robot).

Performance: FoMoVLA achieved state-of-the-art results, beating other top-tier robot models. For example, on the "Long" horizon tasks in LIBERO (which require planning several steps ahead), FoMoVLA reached a success rate of 97.6%, significantly outperforming previous methods.
Zero-Shot Generalization: Even when the robot was tested on completely new situations it had never seen before (like different lighting or camera angles), it still performed incredibly well, achieving an 80.5% success rate on the LIBERO-Plus benchmark.
Efficiency: The best part? All this extra "thinking" happens only during training. When the robot is actually working in the real world, it doesn't need to run the crystal ball or the GPS tracker. It just uses the knowledge it absorbed. This means the robot runs just as fast as before, with almost no extra memory or time cost (adding only about 9.4 ms of latency).

What the Paper Rules Out

The authors are careful to point out what doesn't work well. They found that simply predicting the future pixels (the whole image) is too messy and redundant. They also found that if you teach the robot to predict the future and track the motion separately without letting them talk to each other, the results are only slightly better than doing nothing. The synergy—the conversation between the "what" and the "how"—is what makes the difference.

The Bottom Line

FoMoVLA suggests that to build truly capable robots, we need to teach them to imagine the future and understand the motion path simultaneously. By combining a "goal-oriented" view with a "motion-oriented" view, robots can learn to manipulate the world more naturally and reliably. While the current system still relies on 2D tracking and doesn't fully understand 3D geometry yet, it represents a significant step forward in making robots that don't just react, but truly anticipate.

Technical Summary: FoMoVLA

Problem Statement

Vision-Language-Action (VLA) models have demonstrated strong performance in visuomotor policy learning but remain fundamentally reactive. They map current observations and language instructions directly to actions without explicit forward prediction of world dynamics. While recent research has attempted to enhance VLA foresight through future-predictive supervision, existing approaches suffer from two primary limitations:

Redundancy and Overhead: Dense pixel-level prediction of future frames preserves rich scene information but captures large amounts of static content irrelevant to control, leading to high modeling costs.
Lack of Motion Guidance: Compact keyframe-level representations improve efficiency but typically only characterize the target state (where to go) without capturing the continuous motion path (how to get there) required to reach it.

The authors argue that actionable foresight must encode not only the future state but also the continuous temporal motion dynamics leading toward it.

Methodology: FoMoVLA

FoMoVLA is a training-only framework that augments a foundation VLA model with explicit spatio-temporal supervision by jointly learning future feature prediction and sparse 2D point tracking. The architecture consists of three complementary branches:

1. Point Tracking Branch (Motion Guidance)

This branch grounds image token representations in explicit spatial motion.

Mechanism: The model predicts 2D point trajectories spanning the entire action chunk. To ensure the model understands task-relevant motion, the input sequence is reordered so text tokens precede image tokens, allowing image tokens to attend to the full language instruction.
Supervision: A frozen, off-the-shelf point tracker (CoTracker-v3) acts as a teacher, generating sparse ground-truth displacement targets and visibility labels for a grid of points.
Output: Lightweight projection heads predict point displacements and visibility, optimized via displacement regression, visibility classification, and trajectory smoothness losses.

2. Future Feature Prediction Branch (Goal State)

This branch provides a compact representation of the final goal state.

Mechanism: The model appends $K$ learnable <Foresight> tokens to the input sequence. These tokens are trained to encode the visual features of the final frame ( $t+T$ ) of the action chunk.
Supervision: An Exponential Moving Average (EMA) teacher provides stable target visual features. A lightweight MAE decoder reconstructs the target features from the compact $K$ -token bottleneck.
Objective: A cosine-similarity loss ensures the tokens capture global scene-level changes rather than fine-grained temporal details.

3. Future-Conditioned Cross-Attention (FCCA)

To ensure consistency between the anticipated state and the motion path, the two objectives are coupled.

Mechanism: A lightweight cross-attention module conditions the motion prediction (point tracking) on the foresight representation.
Design: The module uses multi-head attention where the vision features are queries, and the foresight tokens are keys and values. The output projection is zero-initialized, allowing the module to act as an identity mapping at the start of training and gradually learn to inject future information into spatial tokens.
Effect: This anchors local trajectory estimation to a global foresight prior, ensuring the predicted motion is consistent with the predicted future state.

Training and Inference

Training: The total loss combines the action prediction loss (Flow Matching) with the foresight and tracking auxiliary losses. The teachers (point tracker and EMA) are frozen and receive no gradients.
Inference: All auxiliary branches (tracking heads, MAE decoder, FCCA) are discarded. Only the $K$ <Foresight> tokens remain in the input sequence. This results in zero parameter overhead and negligible latency increase (9.4 ms) during deployment.

Key Contributions

Novel Framework: Introduction of FoMoVLA, which augments latent VLA representations with explicit spatio-temporal supervision by jointly learning compact future feature prediction and sparse 2D point tracking.
FCCA Module: A future-conditioned cross-attention mechanism that couples foresight tokens with temporal point tracking, ensuring consistency between anticipated visual states and geometric motion dynamics.
State-of-the-Art Performance: The method achieves state-of-the-art results on the LIBERO and RoboCasa GR-1 Tabletop benchmarks, with strong zero-shot generalization on LIBERO-Plus.

Experimental Results

The paper evaluates FoMoVLA across three benchmarks:

LIBERO: The full model (with FCCA) achieves an average success rate of 98.8% across four task suites (Spatial, Object, Goal, Long), outperforming competitive baselines including GR00T-N1.5, $\pi_0$ , and X-VLA. The ablation study confirms that the FCCA module provides significant gains, particularly in long-horizon tasks (LIBERO-Long), where independent multi-task training fails to achieve synergy.
RoboCasa GR-1 Tabletop: FoMoVLA achieves a 56.9% success rate, a 9.1% improvement over the StarVLA-GR00T backbone, demonstrating effectiveness on a humanoid platform with egocentric observations.
LIBERO-Plus (Zero-Shot): The model achieves 80.5% overall success on perturbed data (viewpoint, robot state, language, lighting, etc.) without fine-tuning, significantly outperforming the base StarVLA (+6.4%) and matching the performance of Abot-M0.

Significance and Claims

The authors claim that FoMoVLA successfully bridges the gap between visual foresight and motion guidance. By treating future feature prediction and sparse point tracking as complementary objectives, the model learns to encode both the goal state and the continuous motion path toward it.

Key claims regarding significance include:

Spatio-Temporal Grounding: The framework enables the policy to learn actions that are both future-oriented and physically constrained, moving beyond reactive mappings.
Efficiency: The approach introduces explicit spatio-temporal supervision without increasing inference cost, as all auxiliary components are training-only.
Generalization: The method demonstrates strong zero-shot generalization, suggesting that explicit spatio-temporal supervision enhances the VLM's ability to represent visual structure and dynamics, which is identified as a primary bottleneck in embodied control.

The paper acknowledges limitations, noting that current tracking supervision models view-dependent motion in 2D image space and does not capture 3D geometry. Future work is suggested to complement 2D tracking with 3D motion prediction.