💻 computer science

DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models

DreamPlan is a novel framework that enhances Vision-Language Model planners for robotic manipulation by training an action-conditioned video world model on exploratory data and then fine-tuning the planner via Odds Ratio Policy Optimization within this virtual environment, thereby achieving high success rates without costly real-world interactions.

Original authors: Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi, Vitor Guizilini, Jiageng Mao, Yue Wang

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi, Vitor Guizilini, Jiageng Mao, Yue Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Smart but Clumsy" Robot

Imagine you have a robot with a brain as smart as a genius librarian. It has read every book on how to fold a shirt, tie a rope, or move a stuffed animal. This is a Vision-Language Model (VLM). It knows the words and the concepts perfectly.

But here's the catch: It has never actually touched anything.

If you ask this robot to fold a wet, slippery shirt, it might say, "Okay, I will grab the left corner and pull it to the right." But because it doesn't understand physics (like how fabric stretches, wrinkles, or gets stuck), it might pull too hard, tear the shirt, or just make a bigger mess. In the real world, especially with soft, squishy things like ropes and clothes, knowing the plan isn't enough; you need to feel the physics.

The Old Way: Learning by Trial and Error (The "Burn the House Down" Method)

Usually, to teach a robot these physical skills, we use Reinforcement Learning (RL). This is like teaching a dog by letting it try, fail, get a treat, and try again.

The Problem: If we let a real robot try to fold a shirt 10,000 times to learn, it will take years, break the robot, and wear out the clothes. It's too expensive, too slow, and too dangerous.
The Simulation Problem: We could try to simulate this in a computer game. But making a computer game where a piece of cloth acts exactly like real cloth is incredibly hard. The "game physics" usually look fake, so the robot learns the wrong lessons.

The New Solution: DreamPlan (The "Lucid Dream" Method)

The authors of this paper created DreamPlan. Think of it as teaching the robot to learn inside a lucid dream where the physics are perfect, but it never has to wake up and touch the real world.

Here is how DreamPlan works in three simple steps:

Step 1: The "Clumsy" Practice Session

First, they let the "genius librarian" robot (the VLM) try the task in the real world just a few times.

What happens? It fails a lot. It pulls the rope the wrong way or drops the toy.
The Magic: Even though it fails, the robot records what happened. "I pulled here, and the rope went there." This data is messy and full of mistakes, but it contains the secret recipe of how the real world actually reacts.

Step 2: Building the "Dream Machine" (The World Model)

Next, they take all that messy failure data and train a special AI called a Video World Model.

The Analogy: Imagine you are an artist. You watch a video of someone trying to fold a shirt and failing. You then paint a picture of what would happen if they did it correctly.
The Innovation: This "Dream Machine" is an AI that can predict the future. You tell it, "If the robot grabs the rope here and pulls that way," and it instantly generates a video of the rope moving exactly as it would in real life. It learns the physics of the real world from the messy data, without needing a perfect physics simulator.

Step 3: The "Daydream" Training (Reinforcement Learning)

Now comes the cool part. Instead of letting the robot touch the real world again, they let it daydream.

The robot thinks of 10 different ways to fold the shirt.
The "Dream Machine" instantly simulates all 10 ways in a video.
The system looks at the videos and says, "Hey, idea #3 looks like it will work! Idea #1 looks like it will fail."
The robot learns from this feedback: "Okay, I should do #3."

This happens entirely in the computer's "imagination." It's like a chess player playing 1,000 games in their head against a perfect opponent, rather than playing one game on a real board.

Why is this a Big Deal?

It's Safe and Cheap: The robot doesn't break anything or wear out clothes because it's learning in a dream.
It's Fast: Generating a video in a dream takes seconds. Breaking a real robot arm takes hours.
It Works on "Squishy" Things: Most robots are good at moving hard boxes. They suck at moving soft things (like ropes or blankets). DreamPlan specifically teaches the robot how soft things behave.

The Result

The paper tested this on three hard tasks:

Straightening a tangled rope.
Folding a piece of cloth.
Repositioning a soft stuffed toy.

The Outcome: The robot started as a "genius but clumsy" planner. After a short period of "daydreaming" using DreamPlan, it became a master of physical manipulation. It succeeded at tasks where it previously failed almost 100% of the time, all without needing thousands of hours of real-world practice.

Summary Analogy

Imagine you want to learn to surf.

Old Way: You jump in the ocean, get hit by waves, swallow water, and try again until you get it right. (Dangerous, slow).
Simulation Way: You play a video game. But the water feels like jelly, so you learn to surf on jelly, and when you get to the real ocean, you sink. (Inaccurate).
DreamPlan Way: You watch a video of a pro surfer failing and succeeding. Then, you close your eyes and vividly imagine yourself surfing perfectly, feeling the water, the wind, and the balance. Your brain learns the feeling of the physics so well that when you finally jump in the real ocean, you can stand up on the first try.

DreamPlan gives robots that same "vivid imagination" to master the physical world.

1. Problem Statement

Robotic manipulation, particularly involving deformable objects (e.g., cloth, ropes, soft toys), requires sophisticated commonsense reasoning and a deep understanding of complex, nonlinear physical dynamics.

The Gap: Large-scale Vision-Language Models (VLMs) possess strong semantic reasoning and can act as zero-shot planners. However, they lack physical grounding. They cannot reliably predict how deformable objects will respond to contact, force, or gravity, leading to compounding errors and low success rates in real-world deployment.
The Bottleneck: While Reinforcement Learning (RL) can adapt planners to specific dynamics, directly fine-tuning massive VLMs via real-world robot interaction is prohibitively expensive, unsafe, and sample-inefficient.
Limitations of Existing Solutions:
- Simulators: High-fidelity simulators for deformable objects are difficult to construct and suffer from a severe "sim-to-real" gap.
- Existing World Models: Current action-conditioned video models are often limited to rigid-body interactions or require curated successful demonstrations, failing to capture the diverse, sub-optimal dynamics of deformable objects.

2. Methodology: The DreamPlan Framework

DreamPlan is an offline reinforcement fine-tuning framework that aligns VLM planners with real-world physics using virtual rollouts generated by a learned video world model. The process consists of three main stages:

A. Zero-Shot Data Collection & Exploration

A pre-trained VLM planner is deployed in the real world to collect exploratory interaction data.
Although the initial planner has a low success rate, the resulting dataset contains dense, rich signals regarding action-outcome causality, including failed and sub-optimal attempts.
Action Representation: The planner uses a discrete, keypoint-based action space (selecting a grasp point and a target placement point) to reduce spatial ambiguity.

B. Learning an Action-Conditioned Video World Model

Objective: Train a world model ( $W_\phi$ ) that predicts future video frames given an initial state, goal, and action sequence.
Architecture: Built upon a pretrained video diffusion backbone (CogVideoX-5B).
Key Innovation (ControlNet-style Conditioning): To effectively condition the model on low-level robotic actions, the system renders the kinematic configuration of the robot arms corresponding to the commanded motions. These rendered action videos are injected into the diffusion model via a ControlNet architecture.
Object-Only Focus: To improve modeling quality, the model is trained to predict cropped, object-only videos on a white background. This forces the network to focus on deformation dynamics rather than irrelevant background clutter or lighting changes.
Data Efficiency: The model is successfully trained on a relatively small dataset (~2,000 trajectories collected in ~4 hours) dominated by sub-optimal data.

C. Reinforcement Fine-Tuning via ORPO

Virtual Rollouts: Instead of real-world interaction, the VLM planner is fine-tuned entirely within the "imagination" of the world model.
Best-of-K Strategy: To avoid the computational cost of repeated video generation during optimization:
1. For a given state, the planner samples a batch of $K$ candidate actions.
2. The world model predicts the outcome for each candidate.
3. The action with the highest predicted task-level objective is selected as the positive sample; others are negative samples.
Optimization (ORPO): The planner is fine-tuned using Odds Ratio Policy Optimization (ORPO). This preference-based method maximizes the log-likelihood ratio of the preferred action over the disfavored ones, eliminating the need for a separate value function and enabling stable, memory-efficient training.

3. Key Contributions

DreamPlan Framework: A novel, highly efficient framework that adapts zero-shot VLM planners to complex physical tasks via offline RL within video-generated virtual rollouts.
Efficient Adaptation Strategy: Integrates Best-of-K sampling with ORPO. This decouples the computationally expensive video diffusion inference from the policy optimization loop, allowing for physics-informed supervision without repeated world-model queries during training.
Deformable Object Modeling: Demonstrates that sub-optimal exploratory data is sufficient to train a robust action-conditioned world model capable of simulating complex deformable dynamics (cloth, rope, soft toys).
Object-Only Video Prediction: Proves that training world models on cropped object videos significantly improves the accuracy of deformation prediction compared to full-scene generation.

4. Experimental Results

The framework was evaluated on three challenging real-world tasks: Rope Straightening, Cloth Folding, and Soft Toy Repositioning.

Performance Gains:
- DreamPlan significantly outperforms zero-shot baselines across all tasks.
- Success Rate: Achieved a 15%–40% absolute improvement over zero-shot baselines.
- Quantitative Scores: The fine-tuned Qwen3-VL-8B achieved an average score of 0.60, nearly doubling the performance of the strongest zero-shot baseline (0.35).
- Specific Task Wins: On Rope Straightening, DreamPlan (0.60) doubled the performance of GPT-4o (0.30). On Toy Repositioning, it outperformed the much larger Qwen3-VL-32B (0.85 vs. 0.70).
World Model Reliability: Qualitative comparisons showed that DreamPlan's world model generates physically plausible deformations, whereas baseline image-to-video models (e.g., CogVideoX, Wan2.2) failed to follow specific actions or produced unrealistic physics.
Efficiency:
- Inference Speed: DreamPlan requires only ~1.12 seconds per decision.
- Comparison: A sampling-based baseline (using the world model as an explicit verifier for every candidate) took ~926 seconds for just 4 samples, with only marginal performance gains. DreamPlan achieves superior performance with orders-of-magnitude lower computational cost.

5. Significance

Bridging the Reality Gap: DreamPlan successfully bridges the gap between semantic reasoning (VLMs) and physical grounding (world models) without requiring massive-scale real-world data collection.
Scalability: By leveraging "imagination" (virtual rollouts) and efficient preference optimization (ORPO), the method makes RL fine-tuning of massive VLMs feasible for complex, contact-rich tasks that were previously considered too dangerous or expensive to train on.
Generalizability: The approach demonstrates that even sub-optimal, exploratory data collected by zero-shot agents is sufficient to train high-fidelity world models for deformable manipulation, offering a new paradigm for robotic learning in unstructured environments.