Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback

1. Problem Statement

Current state-of-the-art (SoTA) text-to-video models, despite their ability to generate photorealistic visuals, struggle significantly with dynamic object interactions. They frequently suffer from:

• Physics Violations: Objects move unrealistically or violate physical laws (e.g., gravity, collision).
• Static Scenes: The model ignores motion prompts, generating static images or videos where objects do not move as described.
• Hallucinations: Objects appear or disappear unnaturally, or multi-step interactions fail (e.g., a hand failing to grasp an object).
• Scaling Limitations: Simply increasing dataset size or model parameters (scaling laws) has proven less effective for video than for Large Language Models (LLMs) due to the complexity of labeling video data and architectural constraints.

The core challenge is aligning generated video dynamics with real-world physics and user instructions without relying solely on massive, labor-intensive human feedback datasets.

2. Methodology

The authors propose a framework that leverages Reinforcement Learning (RL) fine-tuning guided by AI Feedback from Vision-Language Models (VLMs).

A. Theoretical Unification of RL Algorithms

The paper first establishes a theoretical equivalence between two prominent offline RL algorithms for diffusion models:

1. Reward-Weighted Regression (RWR): Derived via Expectation-Maximization (Forward KL-regularization).
2. Direct Preference Optimization (DPO): Derived via Bradley-Terry assumptions (Reverse KL-regularization).
   The authors show both stem from a unified probabilistic objective. They conclude that no single algorithm is universally dominant; the choice depends on the properties of the reward signal and data.

B. Feedback Mechanisms

The study compares several feedback sources for the RL objective:

• Metric-Based Rewards: Standard automated metrics like CLIP Score (text-image alignment), HPSv2/PickScore (human preference proxies), and Optical Flow (motion magnitude).
• AI Feedback (AIF): The core innovation. The authors use large-scale VLMs (specifically Gemini-1.5-Pro and GPT-4o) to evaluate generated videos.
  • Process: The VLM receives the text prompt and the generated video frames. It evaluates the video based on coherence, physical accuracy, task completion, and consistency.
  • Output: A binary label (Accept or Reject).
  • Rationale: VLMs can interpret video scenes similarly to humans, offering a scalable alternative to human raters.

C. Training Pipeline

1. Pre-training: A 3D-UNet based video diffusion model is pre-trained on the Something-Something-V2 dataset (160k samples) to learn basic object movements.
2. Data Collection: The pre-trained model generates videos for a specific set of challenging prompts (5 categories: Object Removal, Multiple Objects, Deformable Objects, Directional Movement, Falling Down).
3. Labeling: Generated videos are labeled by VLMs (binary Accept/Reject) or assigned metric scores.
4. RL Fine-tuning: The model is fine-tuned using either RWR or DPO to maximize the reward (either the binary AI feedback or the metric score).
5. Iterative Refinement: The process can be repeated (iterative RL) where the improved model generates new data for further fine-tuning.

3. Key Contributions

1. Unified Theoretical Perspective: Demonstrated that RWR and DPO are equivalent under a unified probabilistic objective, shifting the focus from algorithm selection to reward/data properties.
2. VLM as a Proxy for Human Feedback: Proved that VLMs (Gemini-1.5-Pro) can effectively act as a scalable proxy for human raters in evaluating video dynamics, achieving a high correlation with human preferences.
3. Superiority of AI Feedback: Showed that optimizing for binary AI feedback yields significantly better results than optimizing for standard differentiable metrics (like CLIP or Optical Flow), which often suffer from "over-optimization" (improving the metric while degrading visual quality).
4. Algorithmic Insights:
   • RWR: Performs well on training data and allows for iterative improvement but tends to overfit and degrade on unseen test data.
   • DPO: Provides better generalization and robustness against overfitting due to its negative gradient (pushing down the likelihood of bad samples), though it saturates faster in iterative settings.

4. Experimental Results

The experiments were conducted on the Something-Something-V2 dataset, focusing on five challenging categories of object movement.

• Performance Gains:

  • AI Feedback (RWR-AIF & DPO-AIF) outperformed all metric-based baselines (CLIP, HPSv2, PickScore, Optical Flow) across all evaluation metrics.
  • Human Evaluation: AI feedback models improved human acceptance rates by +11.8% (RWR) and +14.7% (DPO) compared to the pre-trained baseline.
  • VLM Evaluation: Significant gains in Gemini and GPT-4o evaluations (e.g., +3.8% and +3.5% respectively).
  • VBench Scores: AI feedback models achieved the highest scores in consistency and motion smoothness.

• Correlation Analysis:

  • AI feedback from VLMs showed a strong positive correlation (R = 0.746) with human preference, significantly outperforming metric-based rewards (which showed weak or negative correlations).
  • Metric-based rewards (e.g., HPSv2) suffered from over-optimization: the metric score increased, but the video quality and physical plausibility decreased.

• Category-Specific Improvements:

  • The most significant improvements were seen in Multiple Objects and Falling Down scenarios, which are notoriously difficult for diffusion models.
  • While pre-trained models struggled with multi-step interactions (e.g., "stuffing a bottle opener into a drawer"), RL-finetuned models showed marked improvement, though some complex failures remained.

5. Significance and Conclusion

This paper addresses a critical bottleneck in text-to-video generation: physical realism and dynamic interaction.

• Scalability: By replacing expensive human feedback loops with VLM-based AI feedback, the authors provide a scalable path to improve video generation quality.
• Beyond Visual Style: Unlike previous RL works that focused on aesthetic quality or text alignment, this work specifically targets dynamic object interactions and physics, a domain where current models fail.
• Practical Guidance: The paper offers clear recommendations for practitioners:
  • Use DPO for better generalization and robustness.
  • Use RWR if the goal is to optimize for a specific set of prompts or if iterative on-policy data is available.
  • Prioritize VLM-based binary feedback over differentiable metrics to avoid over-optimization and ensure physical plausibility.

The work suggests that while current open models lack the prior knowledge for complex dynamics, post-training with AI feedback can effectively bridge this gap, making text-to-video generation more reliable for applications like simulation, robotics, and content creation.