A Systematic Post-Train Framework for Video Generation

1. Problem Statement

While large-scale video diffusion models have achieved significant progress in generating high-resolution and semantically rich content, a critical gap remains between their pretraining performance and real-world deployment requirements. The primary challenges hindering deployment include:

• Prompt Sensitivity: Models are often unstable and fail to follow instructions accurately.
• Temporal Inconsistency: Videos suffer from flickering, jitter, and incoherent motion over long durations.
• Local Artifacts: Errors frequently occur in complex details like hands, text, and fast motion.
• Prohibitive Inference Costs: High computational requirements make real-time or scalable deployment difficult.
• Evaluation Noise: Existing evaluation signals are often noisy, and rollout generation for reinforcement learning is computationally expensive.

The paper argues that standard pretraining is insufficient and that a systematic post-training phase is required to align models with user intentions, improve quality, and optimize efficiency under strict cost constraints.

2. Methodology

The authors propose a unified, four-stage post-training framework designed to bridge the gap between likelihood-based pretraining and alignment-heavy deployment.

Phase 1: Supervised Fine-Tuning (SFT)

• Goal: To establish a stable, well-structured reference policy before applying reinforcement learning.
• Mechanism: The pretrained model is fine-tuned on curated data to eliminate "low-hanging fruit" failures such as refusal cascades, incoherent reasoning, and unsafe outputs.
• Rationale: SFT transforms an erratic policy into a coherent baseline, preventing the model from diverging into degenerate behaviors during subsequent RLHF stages and enlarging the exploration space for RL.

Phase 2: Reinforcement Learning from Human Feedback (RLHF) via GRPO

• Goal: To optimize perceptual quality, temporal coherence, and instruction following without relying on unstable value-function estimation.
• Mechanism:
  • Formulation: The flow-matching sampling process is modeled as a Markov Decision Process (MDP) under stochastic dynamics.
  • Algorithm: The authors employ Group Relative Policy Optimization (GRPO), adapted for video diffusion. They utilize a Flash-GRPO strategy with isotemporal grouping, where each prompt group performs a single Stochastic Differential Equation (SDE) transition at a specific timestep for exploration, while using deterministic Ordinary Differential Equation (ODE) updates elsewhere for stability.
  • Reward Models: A multi-dimensional reward system is used, comprising:
    1. Video Aesthetics: Lighting, composition, and cinematic appearance.
    2. Image Aesthetics: Frame-level sharpness and structure.
    3. Motion Quality: Smoothness and temporal consistency.
    4. Text-Video Alignment: Semantic consistency with the prompt.
  • Optimization: The framework balances these rewards to avoid trade-offs (e.g., sacrificing visual naturalness for text accuracy), prioritizing overall visual quality.

Phase 3: Prompt Enhancement (PE)

• Goal: To refine user inputs (prompts) to improve output robustness and visual quality.
• Mechanism: A Large Language Model (LLM) is trained as a "Prompt Enhancer" using the same GRPO framework.
• Reward Signals: The LLM is optimized for Text-Video Alignment, Video Aesthetics, and a Structure Reward (ensuring format compliance).
• Benefit: This optimizes the input side of the generation pipeline, complementing the output side optimization of RLHF, allowing the system to handle diverse user inputs more effectively.

Phase 4: Autoregressive Distillation (AD)

• Goal: To compress the high-quality post-trained model into an efficient causal architecture for faster inference.
• Mechanism:
  • Distillation: The bidirectional teacher model is distilled into a causal student model using Distribution Matching Distillation (DMD).
  • Self-Forcing: The training employs a "Self-Forcing" paradigm where frames are generated conditioned on previously self-generated outputs (using KV cache), rather than teacher forcing. This mitigates error accumulation common in long-video synthesis.
  • Architecture: The process involves ODE regression to stabilize the transition to causal masks, followed by self-forcing distillation to evaluate the quality of the entire generated sequence.

3. Key Contributions

1. Unified Post-Training Pipeline: The paper introduces a systematic framework integrating SFT, RLHF, Prompt Enhancement, and Autoregressive Distillation, addressing quality, controllability, and efficiency simultaneously.
2. GRPO for Video Diffusion: A novel adaptation of GRPO for flow-matching models, utilizing isotemporal grouping and temporal gradient rectification to handle the high cost and noise of video generation rollouts.
3. Dual-Side Optimization: The unique integration of Prompt Enhancement alongside standard RLHF, optimizing both the input prompt and the generation policy using the same reward signals.
4. Efficient Deployment Architecture: The application of Self-Forcing Distillation to transfer capabilities from a bidirectional teacher to a causal, autoregressive student, enabling real-time inference without significant quality loss.

4. Results

The framework was evaluated using a Good–Same–Bad (GSB) comparison protocol with human annotators.

• RLHF Impact: The RLHF stage alone achieved a 31% improvement in the overall GSB metric. The gains were most pronounced in Visual Quality and Motion Quality. Text alignment improvements were modest, attributed to current limitations in text-video reward models.
• Prompt Enhancement Impact: Integrating the Prompt Enhancer yielded an additional 20% improvement in the overall GSB metric.
• Overall: The combined pipeline significantly mitigates artifacts and improves temporal dynamics while preserving semantic alignment.
• Visual Evidence: Qualitative results (Figure 2) show marked improvements in complex scenes (e.g., steam trains, elephants, kitchen scenes) regarding lighting, motion smoothness, and adherence to prompt details compared to the pre-RLHF baseline.

5. Significance

• Practical Blueprint: The paper provides a scalable blueprint for building deployable video generation pipelines that balance visual excellence, temporal consistency, and system efficiency.
• Foundational Advancement: By forcing the model to internalize physical laws, object permanence, and causal interactions through strict temporal optimization, the framework contributes to the development of robust Video World Models.
• Industry Impact: The framework lowers the barrier for integrating generative video into commercial pipelines (e-commerce, marketing, entertainment) by making models more reliable, controllable, and computationally efficient.
• Future Direction: It highlights the need for more robust text-video reward models to further unlock semantic optimization potential.