Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning

1. Problem Statement

Video generation models (e.g., Sora, Veo) have demonstrated impressive visual coherence but struggle with tasks requiring spatial reasoning, multi-step planning, and logical consistency.

• Limitation of Supervised Fine-Tuning (SFT): While SFT on demonstration videos teaches models to solve specific tasks, it leads to overfitting and poor generalization. Models trained on simple mazes fail on complex 3D layouts or irregular textures, suggesting they memorize patterns rather than learning robust reasoning.
• Limitation of Standard RL: Applying Reinforcement Learning (RL) to video generation is hindered by reward design. Multimodal reward models (Vision-Language Models or VLMs) are prone to reward hacking, where the generator produces visually convincing but logically incorrect videos that fool the judge. Furthermore, sparse binary rewards (success/failure) provide insufficient gradient signals for stable training.

2. Methodology

The authors propose Wan-R1, a framework that adapts Group Relative Policy Optimization (GRPO) to flow-based video generation models, guided by verifiable reward functions.

A. Algorithm: Flow-GRPO

The paper adapts GRPO (originally for text) to flow-matching video models:

• Stochastic Sampling: To enable exploration required for RL, the deterministic Ordinary Differential Equation (ODE) sampling of flow models is converted into a Stochastic Differential Equation (SDE). This introduces controlled noise during the generation process.
• Group Relative Advantage: Instead of a separate value network, the model generates a group of $G$ videos for a single prompt. The advantage for each video is calculated by normalizing its reward relative to the group mean and standard deviation.
• Denoising Reduction: To make online RL computationally feasible, the authors use fewer denoising steps ($S_{train}=30$) during training data collection while retaining full steps ($S_{infer}=50$) for evaluation.

B. Verifiable Reward Design

The core innovation is replacing learned reward models with objective, verifiable metrics to prevent reward hacking.

1. For Structured Games (Maze Solving):

   • Trajectory Extraction: The agent's path is extracted from the generated video using object tracking.
   • Multi-Component Reward:
     • Exact Match ($R_{EM}$): Binary reward if the path matches the ground truth perfectly.
     • Precision ($R_{PR}$): Rewards the proportion of consecutively correct steps (provides a dense learning signal).
     • Fidelity ($R_{MF}$): Ensures the maze structure (walls/background) remains consistent across frames, penalizing "cheating" where walls disappear.
   • Combined: $R = \alpha R_{EM} + \beta R_{PR} + \gamma R_{MF}$.

2. For Robotic Navigation (Target-Bench):

   • Since ground truth is a reference video rather than discrete actions, the reward is computed at the embedding level.
   • Frame Embedding: Uses frozen vision encoders (DINOv2/CLIP) to compare generated frames with reference frames.
   • Components:
     • Cosine Similarity ($R_{cos}$): Visual similarity between generated and reference frames.
     • Temporal Consistency ($R_{temp}$): Measures the consistency of motion dynamics.
     • Endpoint Fidelity ($R_{end}$): Ensures the final state matches the goal.

3. Key Contributions

1. Flow-GRPO Adaptation: Successfully adapted GRPO to flow-matching image-to-video models with denoising step reduction, making online RL for video reasoning computationally practical.
2. Systematic Reward Analysis: Demonstrated that VLM-based reward models fail catastrophically in video reasoning due to reward hacking (hallucinating quality). Introduced verifiable rewards based on trajectory extraction and embedding similarity as a robust alternative.
3. Embedding-Level Reward for Robotics: Proposed a novel reward mechanism for robotic reasoning that compares video rollouts via frame similarity and temporal consistency, enabling RL without discrete action supervision.
4. Comprehensive Scaling Insights: Provided practical guidelines on KL regularization, cold-start requirements (SFT is necessary before RL), and test-time scaling.

4. Experimental Results

The model was evaluated on VR-Bench (maze solving) and Target-Bench (robotic navigation).

• Generalization Performance:

  • Exact Match (EM): Wan-R1 improved EM by 29.1% over the SFT baseline on complex 3D mazes and by 51.4% on trap-avoidance tasks.
  • Difficulty Generalization: When trained only on "Easy" mazes, Wan-R1 maintained high performance on "Hard" mazes (e.g., 57.2% Precision on Hard Irregular Mazes vs. 5.8% for SFT), whereas SFT models degraded significantly.
  • Texture Generalization: The model showed strong invariance to unseen visual textures, focusing on structural features rather than superficial patterns.

• Robotic Navigation (Target-Bench):

  • Wan-R1 achieved the best overall score, reducing Average Displacement Error (ADE) and Final Displacement Error (FDE) by roughly 50% compared to the base model.
  • It significantly lowered the miss rate and improved approach consistency.

• Ablation Studies:

  • Reward Models: Using Qwen2.5-VL as a reward model resulted in the model learning to generate "plausible-looking" but incorrect paths (reward hacking).
  • Sparse vs. Dense: Sparse rewards (Success Rate only) led to suboptimal paths. The dense precision reward ($R_{PR}$) was critical for stable learning.
  • Cold Start: Direct RL on the base model failed; SFT is required as a bootstrap to provide a foundational policy.

5. Significance

This work establishes verifiable reward design as a critical enabler for robust video reasoning. It proves that:

1. RL can bridge the generalization gap in video generation, moving models from pattern memorization to true spatial reasoning.
2. Learned reward models are unreliable for complex visual tasks due to the high-dimensional nature of video outputs; objective, task-grounded metrics are superior.
3. The proposed framework is applicable beyond games, extending to real-world robotic navigation, suggesting a pathway for training world models that can plan and reason in physical environments.

The code is set to be publicly available, offering a blueprint for applying RL to video generation tasks where logical correctness is paramount.