Reward-Forcing: Autoregressive Video Generation with Reward Feedback

1. Problem Statement

Current state-of-the-art video generation relies heavily on bidirectional diffusion models (e.g., Sora, Wan2.1), which generate frames by considering both past and future context. While these models produce high-quality, temporally consistent videos, they are inherently unsuitable for real-time or streaming applications because they require minutes to generate a single 5-second clip.

To address latency, recent research has attempted to convert bidirectional models into autoregressive (AR) variants (generating frame-by-frame or chunk-by-chunk). However, existing AR methods face two major limitations:

1. Heavy Reliance on Distillation: They typically require complex, heterogeneous distillation from a powerful bidirectional "teacher" model to learn the generation process.
2. Performance Ceiling: The quality of the resulting AR model is often bounded by the teacher's performance and struggles to maintain temporal coherence without future context, often lagging behind their bidirectional counterparts.

2. Methodology: Reward-Forcing

The authors propose Reward-Forcing, a novel framework that converts bidirectional video diffusion models into autoregressive generators using reward signals rather than relying solely on heavy distillation. The pipeline consists of three main stages:

A. ODE-Based Initialization (Motion Learning)

Instead of training from scratch or using complex distillation immediately, the method first leverages the observation that diffusion models learn global motion structures before refining texture details.

• Process: The student model is initialized by training on a small set of ODE (Ordinary Differential Equation) trajectories generated by a pre-trained bidirectional teacher.
• Goal: This stage teaches the autoregressive model to generate consistent motion dynamics without needing to learn fine-grained textures yet.

B. Reward-Guided Optimization (Texture & Quality Refinement)

Once the model learns motion, the authors introduce a reward-guided optimization stage to enhance visual fidelity and texture.

• Reward Model: They utilize ImageReward (a human preference-based reward model) to evaluate generated frames.
• Training Objective: The model is optimized to maximize the reward score.
  • Key Insight: The authors found that applying reward supervision to random frames degrades motion quality because image-based reward models focus on static texture and alignment, ignoring temporal continuity.
  • Solution: They apply the reward signal only to the last frame of the generated video sequence. This preserves motion coherence (learned in the ODE stage) while refining the final visual quality.
• Differentiability: The reward model is integrated in a differentiable manner to directly guide the diffusion model's gradient updates.

C. Data-Free Training

A significant advantage of this approach is that it is data-free regarding real-world video datasets during the AR training phase. The model learns entirely from:

1. Synthetic ODE trajectories from the teacher.
2. Feedback from the reward model.
   This eliminates the need for large-scale video datasets or complex self-rollout mechanisms used in prior work.

3. Key Contributions

1. Breaking the Distillation Bottleneck: The paper demonstrates that existing AR methods are often limited by the quality of the teacher model and the complexity of heterogeneous distillation. The proposed method achieves comparable or superior results without extensive distillation stages.
2. Motion-Texture Decoupling: The authors empirically validate that motion is learned before texture in diffusion processes. They propose a two-stage framework (ODE for motion, Reward for texture) that aligns with this phenomenon.
3. Reward-Only Supervision Strategy: They show that using reward signals to guide the final frame is sufficient to achieve high-fidelity generation, avoiding the conflict between distribution matching (distillation) and reward optimization.
4. Efficient Pipeline: The method removes the need for a "critic" model or a second distillation stage, making the training process significantly more lightweight and efficient.

4. Experimental Results

The method was evaluated on VBench, a comprehensive video generation benchmark, and compared against state-of-the-art bidirectional models (Wan2.1, LTX-Video) and autoregressive baselines (CausVid, Self Forcing, NOVA).

• Quantitative Performance:
  • Total Score: The proposed method achieved 84.92 on VBench, outperforming the second-best autoregressive method (Self Forcing, 84.31) and closely matching the bidirectional teacher (Wan2.1, 84.26).
  • Quality Score: Achieved 85.91, surpassing Self Forcing (85.25).
  • Efficiency: The method maintains low latency (0.69s) and high throughput (17.0 FPS), comparable to other chunk-wise autoregressive models.
• Ablation Studies:
  • Reward vs. Distillation: Combining reward loss with distillation loss resulted in lower performance (82.55) than using reward alone. The authors attribute this to a conflict where the reward signal pushes the model away from the teacher's distribution, destabilizing the distillation.
  • Frame Selection: Supervising random frames reduced motion quality by >10%, confirming the necessity of supervising only the last frame to preserve temporal dynamics.
  • Teacher Size: Using a larger teacher (14B) did not significantly improve the 1.3B student's performance, suggesting the student model's capacity is the bottleneck, not the teacher's expressiveness.

5. Significance and Future Work

Significance:

• Real-Time Viability: This work provides a scalable path toward high-quality, real-time video generation by removing the computational overhead of bidirectional inference and complex distillation pipelines.
• Paradigm Shift: It challenges the prevailing notion that AR video generation must rely on heavy teacher-student distillation. Instead, it suggests that reward feedback can be a more direct and effective guide for refining generative policies.
• Simplicity: The "data-free" nature of the training (relying on ODE trajectories and rewards) simplifies the deployment pipeline for specialized video generation tasks.

Future Directions:

• Video-Specific Rewards: Moving beyond image-based rewards (like ImageReward) to develop reward models that explicitly understand temporal continuity and motion dynamics.
• Eliminating ODE Initialization: Exploring whether reward-driven training alone (without the initial ODE phase) can suffice for video generation, inspired by recent success in image synthesis.
• Stability: Improving the stability of reward-based training for large-scale models to further reduce inconsistencies in generated videos.

In conclusion, Reward-Forcing presents a compelling alternative to traditional distillation-based autoregressive video generation, achieving state-of-the-art quality with a simpler, more efficient training paradigm.