Technical Summary: Large Video Planner Enables Generalizable Robot Control

Problem Statement

General-purpose robots require planning algorithms that can generalize across diverse tasks and novel environments. While recent advancements in Vision-Language-Action (VLA) models have extended Multimodal Large Language Models (MLLMs) to include action outputs, these systems face a fundamental data scarcity problem. Unlike the web-scale text and image data underpinning MLLMs, robot action data is extremely limited. Consequently, existing VLAs rely on an asymmetric transfer paradigm where broad pre-trained knowledge is fine-tuned on narrow robot datasets. This approach often results in poor generalization when robots encounter new tasks or unseen situations, as the models lack sufficient exposure to the temporal dynamics of physical interactions.

Methodology

The authors propose an alternative paradigm: using large-scale video pretraining as the primary modality for building robot foundation models. Unlike static images or language, videos naturally encode spatio-temporal sequences of states and actions, providing a richer representation of continuous physical behavior.

The proposed system, Large Video Planner (LVP), operates through a two-stage pipeline:

1. The Video Foundation Model (LVP)

LVP is a 14-billion parameter video diffusion model designed specifically for generative planning.

• Architecture: It utilizes a temporally causal 3D Variational Autoencoder (VAE) to compress video clips into a latent space. A Diffusion Transformer (DiT) is then trained in this latent space using a Flow Matching objective.
• Training Strategy: The model employs Diffusion Forcing, a technique that applies independent noise levels to different frames within a sequence. This allows the model to flexibly condition on varying amounts of history (from zero frames to multiple previous frames) during both training and inference.
• Guidance Mechanisms: To ensure physical consistency and adherence to instructions, the model integrates History Guidance. This is a variant of Classifier-Free Guidance (CFG) that conditions the generation on provided context frames (e.g., the initial robot observation) alongside text instructions, significantly improving motion fidelity compared to text-only guidance.
• Training Stages: The model undergoes two stages:
  1. Continual Pretraining: Trained on a massive dataset to learn rich dynamics and instruction following.
  2. Low Camera Motion Finetuning: A specialized stage using a subset of data with low optical flow to suppress unwanted camera drift, ensuring the generated plans are stable for robotic execution.

2. LVP-1M Dataset

To support this training, the authors curated LVP-1M, an open, internet-scale dataset of 1.4 million video clips.

• Sources: The dataset aggregates data from four robotics teleoperation datasets (Bridge, Droid, Language Table, AgiBot-World) and four human-centric activity datasets (Ego4D, Epic-Kitchens, Something-Something, and a filtered subset of Panda-70M).
• Processing: Clips are rigorously filtered for quality, visible embodiment (hands or grippers), and lack of rapid camera motion. Crucially, the authors perform temporal alignment, resampling robot videos to match human speeds to ensure temporal consistency across different morphologies.
• Captioning: The dataset is enriched with action-centric captions generated by LLMs (Gemini), ranging from short task descriptions to detailed action narratives, to enhance instruction-following capabilities.

3. Action Extraction and Retargeting

The system translates generated video plans into executable robot actions:

• Motion Estimation: For human hand videos, the system uses HaMeR for per-frame hand pose estimation and MegaSAM for 4D scene reconstruction to ensure metric scale and temporal smoothness.
• Retargeting: The estimated human hand trajectories are retargeted to robot end-effectors. For dexterous hands, Dex-Retargeting solves an optimization problem to map human joint configurations to robot joints. For parallel-jaw grippers, the system uses GraspNet to predict grasp poses based on the scene geometry and detected grasping intent.
• Execution: The retargeted trajectories are transformed into the robot's control frame, and inverse kinematics (using cuRobo) are solved for the arm, while finger trajectories drive the hand controller directly.

Key Contributions

1. Large Video Planner (LVP): A 14B parameter video foundation model designed for embodied planning that generates zero-shot video plans from a single image and text instruction.
2. LVP-1M Dataset: A curated, open dataset of 1.4M clips combining human and robot interactions, processed for action-centric learning and temporal alignment.
3. Rigorous Evaluation Protocol: A novel evaluation framework using third-party selected "in-the-wild" tasks and real-robot experiments to assess task-level generalization, moving beyond standard object-level benchmarks.

Results

The paper evaluates LVP through two primary experiments:

Video Motion Planning Evaluation

Using a test set of 100 diverse, third-party selected tasks (e.g., tearing tape, opening a gate, flushing a toilet), LVP significantly outperformed baseline video generation models (Wan 2.1, Cosmos-Predict 2, Hunyuan).

• Task Completion: LVP achieved a 59.3% success rate at "Level 3" (Task Complete: correct contact, end state, and plausible motion), compared to 39.3% for the best baseline (Wan 2.1).
• Physical Consistency: LVP demonstrated superior ability to generate physically coherent trajectories, avoiding the spatial and semantic inconsistencies common in baselines.

Real-Robot Execution

The system was deployed on two platforms: a Franka Arm with a parallel-jaw gripper and a Unitree G1 humanoid with an Inspire dexterous hand.

• Generalization: LVP successfully executed zero-shot tasks in unseen environments, including opening doors, wiping tables, scooping coffee beans, and tearing tape.
• Comparison: In direct comparison with state-of-the-art VLA baselines (π0 and OpenVLA), LVP showed stronger generalization on novel tasks. While baselines performed well on familiar pick-and-place tasks, they struggled with complex, out-of-distribution tasks where LVP succeeded.
• Dexterous Manipulation: The system successfully transferred complex human hand articulations to the dexterous robot hand, a capability where existing baselines (which do not support multi-finger hands) could not be evaluated.

Significance and Claims

The paper claims that using video as a primary modality offers a "directly grounded source of transfer" for robotics. By leveraging the abundant temporal dynamics present in internet-scale video data, LVP bridges the gap between large-scale pretraining and embodied decision-making.

The authors argue that this approach enables robust instruction following and strong generalization to unseen tasks and environments, addressing a central challenge in embodied intelligence. The work demonstrates that a video foundation model can serve as a visual motion planner in pixel space, generating plans that can be retargeted to diverse robot morphologies for real-world execution. The authors release the model, dataset, and code to support open, reproducible research in video-based robot learning.

Note: The authors acknowledge limitations, including the inference time required for video generation (several minutes on a single A100 GPU), potential errors in the open-source reconstruction and retargeting modules, and the current reliance on open-loop execution which may limit performance on highly dexterous tasks requiring closed-loop feedback.