💻 computer science

ViVa: A Video-Generative Value Model for Robot Reinforcement Learning

The paper introduces ViVa, a video-generative value model that leverages the spatiotemporal priors of a pretrained video generator to jointly predict future robot states and scalar values, thereby enabling more reliable value estimation and improved performance in real-world long-horizon robotic tasks.

Original authors: Jindi Lv, Hao Li, Jie Li, Yifei Nie, Fankun Kong, Yang Wang, Xiaofeng Wang, Zheng Zhu, Chaojun Ni, Qiuping Deng, Hengtao Li, Jiancheng Lv, Guan Huang

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jindi Lv, Hao Li, Jie Li, Yifei Nie, Fankun Kong, Yang Wang, Xiaofeng Wang, Zheng Zhu, Chaojun Ni, Qiuping Deng, Hengtao Li, Jiancheng Lv, Guan Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to fold a shirt or pack a box. You want the robot to learn by doing, but there's a catch: the robot can't see the future.

In the real world, a robot might make a small mistake (like tilting a box too early), but it won't know it messed up until 10 minutes later when the box falls apart. This "delayed feedback" makes learning incredibly hard. To fix this, robots use something called a Value Model. Think of this as an internal "scorecard" or a "gut feeling" that tells the robot, "You are doing well," or "You are heading for a disaster," right in the moment.

For a long time, these scorecards were built using Vision-Language Models (VLMs). These are like super-smart photo analysts. They look at a single picture and say, "That looks like a successful shirt fold." But here's the problem: They are stuck in the present. They are great at describing a static photo, but they are terrible at imagining how a scene moves and changes over time. They might think a robot is doing well just because the picture looks nice, even if the robot is about to drop the shirt in the next second.

Enter ViVa: The Robot's "Crystal Ball"

The paper introduces ViVa (Video-Generative Value Model). Instead of just looking at a photo, ViVa is built on a Video Generator.

Here is the best way to understand the difference:

The Old Way (VLM): Imagine a teacher who only looks at a single snapshot of a student taking a test. If the student is holding a pen and looking at the paper, the teacher says, "Great job!" even if the student is writing gibberish.
The ViVa Way: Imagine a teacher who can fast-forward the movie of the student's actions. Before the student even finishes the sentence, the teacher sees the future: "Oh no, if they keep writing like that, they'll run out of time and fail." The teacher then immediately lowers the student's score.

ViVa does exactly this. It takes the robot's current view and its own body position (proprioception), and it simulates the next few seconds of video in its head.

It Imagines the Future: "If I move my arm this way, what will my hand look like in 2 seconds?"
It Checks the Outcome: "Does that future look like a successful task, or a mess?"
It Assigns a Score: Based on that imagined future, it gives a score (a "Value") to the current moment.

Why This Changes Everything

The authors tested this on three tricky real-world tasks: folding a shirt, packing a box, and organizing toilet paper.

The "Photo Analyst" (Old Model): When the robot made a mistake, like misaligning a box corner, the old model didn't care. It kept giving high scores because the picture still looked "okay." It was blind to the impending disaster.
ViVa (The Crystal Ball): As soon as the robot started to misalign the box, ViVa's internal simulation showed the box falling over. It immediately dropped the score, screaming, "Stop! You're going to fail!" This allowed the robot to correct its course instantly.

The Results

Because ViVa could "see" the future consequences of its actions, it learned much faster and became much better at the tasks:

Success Rate: It went from packing boxes successfully 58% of the time (with the old model) to 73%.
Speed: It packed more boxes per hour because it made fewer mistakes and didn't have to restart as often.
Generalization: When they gave the robot a new object it had never seen before (like a pair of pants instead of a shirt), the old model got confused. ViVa, however, understood the physics of folding and did a great job, because it was predicting how the fabric would move, not just matching patterns.

The Bottom Line

ViVa is like giving a robot a time machine.

Instead of just reacting to what it sees right now, it uses the power of video generation to dream about the future. By asking, "What happens if I do this?" before it actually does it, the robot learns to avoid mistakes before they happen. It turns robot learning from a game of "guess and check" into a game of "plan and execute."

1. Problem Statement

Context: Vision-Language-Action (VLA) models have advanced robot manipulation through large-scale pretraining. However, deploying these robots in the real world is hindered by partial observability and delayed feedback. In long-horizon tasks, the consequences of an action may not be immediately apparent, making it difficult for agents to learn from sparse rewards.

The Challenge: Reinforcement Learning (RL) relies on value functions to estimate task progress and guide policy improvement. Existing approaches often repurpose Vision-Language Models (VLMs) for value estimation. However, VLMs are trained on static image-text pairs and lack explicit modeling of temporal dynamics. Consequently, they struggle to capture how physical interactions evolve over time, leading to unreliable value signals that fail to distinguish between beneficial behaviors and execution errors in dynamic environments.

Core Insight: Value estimation is fundamentally a problem of anticipating the future. Unlike discriminative models trained on static data, video generative models are explicitly optimized to capture spatiotemporal evolution. Therefore, they offer a more natural foundation for value learning by grounding estimates in anticipated embodiment dynamics.

2. Methodology: ViVa

The authors propose ViVa (Video-Generative Value Model), a novel framework that repurposes a pretrained video generator to serve as a value function for robotic RL.

Architecture

Base Model: Built upon Wan2.2, a pretrained video Diffusion Transformer (DiT).
Input Modalities: The model takes current multi-view RGB images and robot proprioception (joint angles/poses) as input.
Latent Injection: Instead of modifying the core architecture, ViVa uses latent injection to map different modalities into the video generator's latent space:
- Images: Encoded via a spatiotemporal VAE.
- Proprioception: Mapped via repeat-padding to match latent dimensions.
- Scalar Value: Mapped via broadcast operations.
Joint Prediction: The model is trained to jointly predict:
1. Future Proprioception: The robot's state (e.g., end-effector pose) at a future time step $t+K$ .
2. Scalar Value: A normalized value representing the expected return (task progress) from the current state.

Training Objective

Flow Matching: The model uses a flow matching formulation to denoise target latents (future proprioception and value) conditioned on the current observation (clean prefix).
Reward Definition: To handle sparse rewards, the authors define a step-wise reward that encodes both temporal progress and the final outcome (success/failure).
- Successful episodes yield a value in $[0, 1)$ .
- Failed episodes are shifted to $[1, 2)$ via a terminal penalty, ensuring a clear margin between success and failure.
Loss Function: A weighted combination of losses for predicting future proprioception and the value latent. The inclusion of proprioception prediction forces the model to internalize the robot's physical dynamics, which is crucial for tasks requiring precise coordination.

Integration

ViVa is integrated into the RECAP (Reinforcement Learning with Experience and Corrections via Advantage-conditioned Policies) pipeline, replacing the original VLM-based value function to compute advantages for policy refinement.

3. Key Contributions

Paradigm Shift: Identifies that value estimation in robotic RL is inherently a future anticipation problem. It argues that video generative models, trained on spatiotemporal evolution, are superior to static VLMs for this task.
ViVa Model: Introduces a video-generative value model that intrinsically couples value with foresight by jointly predicting future embodiment dynamics and the current state value.
Robust Generalization: Demonstrates that leveraging spatiotemporal priors from video corpora allows the model to generalize to novel objects and unseen scenarios where static VLMs fail.
Empirical Validation: Achieves substantial improvements in real-world tasks, specifically in box assembly, outperforming prior VLA and VLM-based value methods.

4. Experimental Results

Tasks

Experiments were conducted on three real-world manipulation tasks:

Shirt Folding: High deformability, dual-arm coordination.
Box Packaging/Assembly: Long-horizon, multi-stage assembly.
Toilet Paper Organization: Precise tearing, disposal, and sealing.

Qualitative Analysis

Error Detection: Unlike VLM-based values that often increase monotonically or fluctuate erratically, ViVa produces sharp value drops precisely when execution errors occur (e.g., corner misalignment, box overhang).
Progress Tracking: ViVa maintains a smooth, steadily increasing value trajectory that accurately reflects gradual task progress (e.g., folding steps), whereas VLM baselines often fail to register intermediate milestones.
Out-of-Domain Generalization: When tested on folding pants (an unseen object), ViVa successfully tracked key milestones with a smooth trajectory. In contrast, the VLM baseline failed to detect milestones and exhibited counter-intuitive value drops.

Quantitative Results (Box Assembly)

Success Rate:
- Imitation Learning ( $\pi_0.5$ ): 42%
- Imitation Learning (Gigabrain-0): 53%
- RECAP (VLM Value): 58%
- RECAP (ViVa): 73%
Throughput: ViVa achieved 14 successful tasks per hour, compared to 11 for the VLM baseline and 8 for the base imitation policy.
Efficiency: ViVa reduced training time by 1.5x (4 GPU-days vs. 6 for VLM) and inference latency by nearly 50% (0.18s vs. 0.32s).

Ablation Studies

Video Backbone: Replacing the VLM backbone with a video generator alone (without proprioception) significantly improved value stability, proving the importance of spatiotemporal priors.
Future Proprioception: Removing the future proprioception prediction objective made the model insensitive to subtle physical errors (e.g., uneven force, asynchronous lifting), confirming that predicting embodiment dynamics is essential for accurate value estimation.
Prediction Horizon ( $K$ ): A horizon of $K=50$ provided the optimal balance between temporal context and stability. Short horizons ( $K=25$ ) were too sensitive to noise, while long horizons ( $K=75$ ) introduced too much uncertainty.

5. Significance

ViVa represents a significant step forward in embodied AI by bridging the gap between video generation and reinforcement learning.

Reliability: It solves the critical issue of "value hallucination" in long-horizon tasks by grounding value estimates in physical dynamics rather than static visual patterns.
Scalability: By repurposing pretrained video generators, ViVa offers a scalable path to value learning without requiring massive amounts of new robot-specific data.
Real-World Impact: The demonstrated improvements in success rates and throughput on complex assembly tasks highlight the practical viability of video-generative value models for deploying robust robots in unstructured environments.

In summary, ViVa demonstrates that anticipating the future (via video generation) is the key to evaluating the present (via value estimation), providing a more reliable signal for robot policy improvement than static visual understanding alone.