🤖 AI

Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

This paper demonstrates that while unified vision-language models struggle with forward dynamics prediction, they can be effectively bootstrapped to achieve state-of-the-art performance in action-centric image editing by leveraging the significantly easier task of inverse dynamics prediction to generate synthetic training data and guide inference-time search.

Original authors: Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B. Cohen, Edoardo M. Ponti

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B. Cohen, Edoardo M. Ponti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot that can look at pictures and read text. You ask it a simple question: "If I see a man holding a book, and then I tell him to put the book down, what will the picture look like next?"

This paper asks: Can our current smart robots actually predict the future?

The authors found that while these robots are great at reading and looking, they are terrible at guessing what happens next in a physical world. If you ask them to predict the next scene, they often just guess randomly or copy the old picture without changing anything. They haven't really learned the "physics" of how actions change the world.

However, the researchers discovered a clever trick. They found that it is much easier to teach the robot the reverse: "Here is a picture of a man holding a book, and here is a picture of him putting it down. What action happened in between?"

The robot is surprisingly good at guessing the action (the story) when shown the before and after pictures. The authors call this Inverse Dynamics (working backward). Predicting the future picture from the action is Forward Dynamics (working forward), and that's the hard part.

The Solution: Using the "Reverse" to Teach the "Forward"

Since the robot is good at working backward but bad at working forward, the authors used the robot's "backward" skill to teach it how to "move forward." They used two main strategies, which they call bootstrapping (using a small step to pull yourself up to a bigger step).

Strategy 1: The "Ghost Writer" (Weak Supervision)

Imagine you have a huge library of video clips, but no one wrote down what happened in them.

The Ghost Writer: They took their "backward-skilled" robot and asked it to watch these unlabelled videos. The robot looked at the start and end of a clip and wrote a sentence describing the action (e.g., "The man kicked the ball").
The Student: Now, they had a massive amount of new data: Start Picture + Robot's Sentence = End Picture.
The Lesson: They used this new, self-made data to train the robot to predict the future. It's like hiring a ghost writer to create a textbook for a student, so the student can learn to predict the future without needing a human to write every single example.

To make sure the robot didn't just copy the old picture, they added a special rule: "Pay extra attention to the parts of the picture that actually changed." If a ball moves, focus on the ball, not the static background.

Strategy 2: The "Judge" (Inference-Time Verification)

Imagine you are taking a test, and you can write down three different answers for the future picture.

The Generator: The robot tries to draw three different "future" pictures based on the instruction.
The Judge: Before you pick the final answer, you ask the "backward-skilled" robot (the Judge) to look at the three options. It asks: "If I see the start picture and this future picture, does the action 'pick up the book' make sense?"
The Selection: The robot picks the future picture that the Judge says makes the most sense. It's like having a teacher check your homework answers before you hand them in to pick the best one.

The Results

The authors tested this on a "World Model" benchmark (a set of challenges to see if a robot understands how the world works).

Before: The robots were bad at predicting the future, often failing to change the image at all.
After: Using these two tricks, their robots became better at predicting future images than even the most advanced, specialized image-editing tools currently available.
The Proof: When humans looked at the results, they preferred the robot's predictions over the specialized tools. The robot was better at following instructions like "move the book" or "make the dog jump" without messing up the rest of the picture.

The Bottom Line

The paper shows that we don't need to build a brand-new, super-complex robot from scratch to understand the world. We can take a general-purpose robot that is already good at understanding stories (actions) and use that strength to teach it how to predict the future.

Important Note: The authors are very clear that this is currently limited to single steps (predicting the very next picture after one action). They are not claiming this robot can plan a whole day or control a robot arm in a factory yet. They have just taken the first step toward building a robot that can truly simulate how the world changes.

Technical Summary: Bootstrapping World Models from Inverse Dynamics in Unified VLMs

Problem Statement

The central question addressed in this work is whether unified Vision-Language Models (VLMs) possess the inherent capability to perform Forward Dynamics Prediction (FDP). FDP is defined as predicting the future state of an environment (in image form) given a previous observation and an action (expressed as a language instruction).

The authors investigate whether off-the-shelf VLMs can serve as "world models" capable of simulating physical transitions. Their initial empirical evaluation reveals a critical limitation: existing unified VLMs do not exhibit a reliable preference for ground-truth action-observation trajectories over adversarially manipulated ones. When tasked with predicting the next frame, these models fail to differentiate between physically plausible transitions and random or copied observations, indicating that the implicit world models within current VLMs are not well-grounded in real-world forward dynamics.

Conversely, the authors identify a significant asymmetry in multimodal grounding: Inverse Dynamics Prediction (IDP)—predicting the action that occurred between two observed frames—is substantially easier for VLMs to learn than FDP. While VLMs struggle with FDP in a zero-shot setting, they demonstrate a capacity to learn IDP with relatively little fine-tuning.

Methodology

To leverage the ease of learning IDP to bootstrap FDP, the authors propose two distinct strategies to enhance a unified VLM's forward dynamics capabilities:

1. Weakly Supervised Learning from Synthetic Trajectories

This strategy utilizes a fine-tuned IDP model to generate training data for FDP.

Data Synthesis: The authors extract motion key-frame pairs from 45 hours of unlabelled real-world videos (sourced from Moments-in-Time, Kinetics-700, and UCF-101).
Annotation: A VLM fine-tuned for IDP is used to predict the textual action ( $\hat{a}$ ) occurring between these frame pairs ( $o_s, o_t$ ).
Training: These synthesized triplets $(o_s, \hat{a}, o_t)$ are combined with ground-truth data from the AURORA dataset to perform weakly supervised fine-tuning of the VLM as a Forward Dynamics Model (FDM).
Recognition-Weighted Loss: To prevent the model from degenerating into simply copying the source image, the authors introduce a novel loss-weighting mechanism. A recognition model estimates the visual difference between source and target frames, assigning higher weights to image tokens in regions that change significantly. This forces the FDM to focus on semantically meaningful regions indicative of the action's consequences.

2. Inference-Time Verification

This strategy operates at test time without additional training.

Candidate Generation: The FDM generates $N$ candidate future observations for a given source and action.
Scoring: The IDP model acts as a verifier, scoring each candidate by calculating the likelihood of the action given the source and the candidate future frame ( $p_{IDM}(a | o_s, o_t^{(i)})$ ).
Selection: The candidate with the highest IDP score is selected as the final prediction. This effectively guides the search space at inference time, leveraging the IDP's superior ability to recognize valid transitions.

Key Contributions

Empirical Characterization of VLMs: The paper provides a systematic evaluation showing that current unified VLMs lack a consistent preference for real-world forward dynamics trajectories compared to adversarial negatives, whereas they show a much stronger, learnable preference for inverse dynamics.
Bootstrapping Framework: The authors propose and validate two strategies (synthetic trajectory generation and inference-time verification) to bootstrap FDP capabilities from IDP within a single unified architecture.
Loss-Weighting Mechanism: The introduction of a recognition-weighted training objective that dynamically emphasizes changing image regions, mitigating the tendency of generative models to copy source inputs.
Comprehensive Evaluation: The work evaluates these strategies across five subsets of the AURORA-BENCH (MagicBrush, Action-Genome, Something-Something, WhatsUp, Kubric) and compares them against state-of-the-art specialized image editing models.

Results

The authors evaluated their approach using two families of unified VLMs: Chameleon-7B and Liquid-8B.

Performance vs. Specialized Models: The best-performing model (C-FDM, Chameleon fine-tuned with synthetic data and loss weighting) achieved performance competitive with, and often superior to, state-of-the-art specialized image editing models (e.g., PixInstruct, GoT, SmartEdit).
- According to GPT-4o-as-a-judge, the proposed method improved upon specialized baselines by a margin of 7% to 13% on average across AURORA-BENCH subsets.
- In human evaluations, C-FDM achieved the best average score across all subsets, particularly excelling in action-centric datasets (Action-Genome, Something-Something, Kubric) where specialized image editing models often struggled.
Effectiveness of Strategies:
- Synthetic Trajectories: Removing the synthetic data led to performance drops, particularly on datasets requiring complex real-world action understanding, confirming the value of weak supervision.
- Loss Weighting: Removing the recognition-weighted loss resulted in consistent degradation, highlighting its role in preventing copy-paste behaviors and improving instruction following.
- Inference-Time Verification: This training-free strategy improved FDP performance to a level comparable to trajectory synthesis, demonstrating that IDP can effectively guide search at inference time.
Generalization: The models trained with FDP objectives showed improved performance on spatial reasoning benchmarks (SpatialMQA, EmbodiedSpatial-Bench), suggesting that learning forward dynamics enhances spatial understanding beyond simple image editing.

Significance and Claims

The paper claims to offer "promising early evidence" that unified VLMs can be successfully endowed with forward dynamics prediction capabilities. The significance lies in demonstrating that:

Asymmetry is Key: The difficulty gap between learning IDP and FDP can be exploited to bootstrap world modeling abilities in general-purpose models without requiring massive, specialized world model datasets.
General-Purpose Viability: A general-purpose VLM, when augmented with these bootstrapping strategies, can outperform specialized models in action-centric image editing tasks, suggesting a path toward developing long-horizon world models for embodied agents.
Modest Scope: The authors explicitly limit their scope to single-step action-observation trajectories. They acknowledge that while the results are promising, fine-grained control and subtle instruction understanding remain challenging, and the models are not yet ready for complex, multi-step planning without further development.

The work positions itself not as a final solution for world modeling, but as a foundational step proving that the "world model" capabilities required for embodied AI can be cultivated within existing unified multimodal architectures through the strategic use of inverse dynamics.