AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization

1. Problem Statement

The paper addresses the challenge of aligning streaming autoregressive (AR) video generators with human preferences using Reinforcement Learning from Human Feedback (RLHF). While streaming AR models (often combined with few-step distillation like Consistency Models) offer low-latency, high-quality synthesis, they are difficult to optimize via standard RLHF methods.

Key Challenges Identified:

• Mismatch with SDE-based GRPO: Existing Group Relative Policy Optimization (GRPO) methods for video generation typically convert deterministic Ordinary Differential Equation (ODE) sampling into stochastic Stochastic Differential Equation (SDE) processes to enable exploration. However, few-step AR generators (e.g., Self-Forcing) are near-deterministic. Their output diversity is driven almost entirely by the initial noise of a chunk, while intermediate solver noise has negligible impact.
• Ineffective Exploration: SDE-based methods freeze the initial noise and inject noise at intermediate steps. In few-step AR models, this leads to a fundamental mismatch: the "action space" (intermediate noise) carries almost no entropy, resulting in near-zero policy gradients and a failure to improve rewards.
• Credit Assignment: Applying sequence-level rewards to AR generation is computationally expensive and suffers from severe credit assignment issues (determining which specific chunk caused a reward change).

2. Methodology: AR-CoPO

The authors propose AR-CoPO (AutoRegressive Contrastive Policy Optimization), a framework adapted from the "Neighbor GRPO" perspective but tailored for streaming AR generation.

A. Chunk-Level Alignment via Forking

Instead of treating the entire video sequence as a single rollout, AR-CoPO introduces a chunk-level action space:

1. Shared Context: The model generates a shared history up to a randomly selected pivot chunk $p$.
2. Forking (Exploration): At chunk $p$, the generation branches into $G$ neighbors. This is achieved by perturbing the initial noise of the pivot chunk ($\epsilon_p$) while keeping all subsequent noise (for the rest of the video) and the context identical across all branches.
3. Sequence-Level Reward: Each branch completes the full video, and a reward model assigns a sequence-level score.
4. Localized Update: Because only the pivot chunk's initial noise differs, reward differences are cleanly attributable to that specific chunk. The model performs a contrastive update restricted to the pivot chunk, bypassing the need for full-sequence backpropagation.

B. Adaptation for Consistency Models (CoPO)

Standard Neighbor GRPO measures distances between intermediate latents ($x_t$). However, for Consistency Models (CMs) used in few-step generation, the key operation is a one-step mapping to a clean prediction ($\hat{x}_0$).

• AR-CoPO redefines the distance metric in the $\hat{x}_0$ prediction space rather than the intermediate latent space. This provides a more informative signal for CM-based samplers.

C. Semi-On-Policy Training Strategy

The paper identifies that pure on-policy training (exploring via noise perturbation) is insufficient for global semantic alignment (e.g., Text Alignment), as small noise changes rarely alter high-level semantics, leading to weak gradients or "reward hacking" (degrading motion quality to satisfy text prompts).

• Semi-On-Policy (Exploitation): The authors introduce a strategy where rollouts are fixed to a reference policy ($\pi_{ref}$) stored in a replay buffer. The contrastive objective is applied to these fixed samples to upweight high-reward candidates and suppress low-reward ones.
• Trust Region: To prevent distributional collapse, ratio clipping (standard in PPO/GRPO) is retained to keep the policy within a trust region of the reference.
• LoRA Merging: Two separate LoRA adapters are trained independently:
  1. On-Policy Adapter: Focuses on exploration and improving reward scores.
  2. Semi-On-Policy Adapter: Focuses on exploiting high-quality reference data to maintain generation fidelity.
  • At inference, these adapters are merged to achieve both high reward alignment and high visual quality.

3. Key Contributions

1. AR-CoPO Framework: A novel RLHF framework that enables alignment of streaming AR video generators using a contrastive objective over neighborhood candidates, circumventing the limitations of SDE-based exploration.
2. Chunk-Level Action Space: A forking mechanism that constructs neighbors at a random pivot chunk, enabling natural, localized credit assignment from sequence-level rewards and significantly reducing training costs.
3. Semi-On-Policy Strategy: A hybrid training approach that combines on-policy exploration with off-policy exploitation (via a replay buffer), effectively improving out-of-domain generalization and preventing reward hacking.
4. Consistency Model Adaptation: A specific adaptation of the contrastive objective for Consistency Models by utilizing $\hat{x}_0$ prediction distances instead of intermediate latent distances.

4. Experimental Results

The method was evaluated on Self-Forcing and Causal-Forcing baselines using the VideoAlign (VQ, MQ, TA) and VBench benchmarks.

• vs. SDE-based GRPO: SDE-based baselines failed to improve rewards (scores remained flat or dropped) because intermediate noise injection is ineffective in few-step AR models. AR-CoPO showed consistent improvement.
• vs. Baselines:
  • Semi-On-Policy: Surpassed all streaming AR baselines on VBench Total (82.45 vs. 82.31 for LongLive), demonstrating improved generation quality and semantic coherence.
  • Merged Model: The final merged model achieved a VideoAlign Overall score of 8.22 (up from 7.76) while maintaining VBench Total at 82.17 (vs. baseline 82.15).
• Ablation Studies:
  • Pure on-policy training improved Text Alignment (TA) scores but caused a collapse in Motion Quality (MQ) and VBench scores (reward hacking).
  • Semi-on-policy training preserved quality while improving alignment.
  • Removing ratio clipping in semi-on-policy training led to distributional drift and performance degradation.
• Generalization: The method was also successful on the Causal-Forcing model, confirming broad applicability.

5. Significance

AR-CoPO provides a critical solution for the post-training alignment of the next generation of low-latency, streaming video generators. By recognizing the near-deterministic nature of few-step consistency models and shifting the exploration mechanism from intermediate SDE noise to initial chunk noise, the method unlocks effective RLHF for these architectures.

The introduction of semi-on-policy training addresses a major bottleneck in video RLHF: the trade-off between optimizing for human preference (which often leads to artifacts) and maintaining high-fidelity generation. By decoupling exploration and exploitation into separate adapters and merging them, AR-CoPO achieves genuine alignment without sacrificing video quality, setting a new standard for controllable, high-quality video synthesis.