Manifold-Aware Exploration for Reinforcement Learning in Video Generation

1. Problem Statement

The paper addresses the instability and unreliability of applying Group Relative Policy Optimization (GRPO) to video generation models. While GRPO has been successful in language models and image generation, its application to video faces unique challenges due to the complex, high-dimensional solution space of video data.

Core Issues Identified:

• Excess Noise Injection: Existing methods (e.g., FlowGRPO, DanceGRPO) convert deterministic Ordinary Differential Equations (ODEs) into Stochastic Differential Equations (SDEs) for exploration using Euler-style discretization and first-order approximations. This introduces significant truncation errors, injecting "extra energy" (noise) that pushes the generation trajectory off the valid data manifold.
• Manifold Drift: This off-manifold exploration leads to temporal jitter, artifacts, and low-quality rollouts, making reward estimates unreliable and destabilizing the alignment process.
• Gradient Imbalance: The diffusion process suffers from inherent signal-to-noise imbalances across timesteps. Gradients vanish at high noise levels ($t \to 1$) and explode at low noise levels ($t \to 0$), causing biased learning.
• Stability-Plasticity Dilemma: Traditional KL constraints (anchoring to the initial model) are too rigid, preventing the model from reaching optimal policies (underfitting). Conversely, relaxed constraints allow the policy to drift indefinitely away from the valid data manifold, leading to reward hacking or catastrophic forgetting.

2. Methodology: SAGE-GRPO

The authors propose SAGE-GRPO (Stable Alignment via Exploration), a framework that constrains exploration at both micro and macro levels to keep trajectories within the vicinity of the pre-trained video data manifold.

A. Micro-Level: Precise Manifold-Aware SDE & Gradient Equalization

1. Precise Manifold-Aware SDE:

   • Instead of linear first-order approximations, the authors derive an exact noise standard deviation ($\Sigma_t^{1/2}$) by integrating the diffusion coefficients over the time interval $[\sigma_{t+1}, \sigma_t]$.
   • Key Formula: The variance includes a logarithmic curvature correction:
     $$\Sigma_t = \eta^2 \left[ -(\sigma_t - \sigma_{t+1}) + \log\left(\frac{1-\sigma_{t+1}}{1-\sigma_t}\right) \right]$$
   • This correction accounts for the geometric contraction of the signal coefficient, ensuring the injected noise stays tangent to the flow trajectory and closer to the data manifold, reducing off-manifold drift.

2. Gradient Norm Equalizer:

   • To address the vanishing/exploding gradient problem, a Temporal Gradient Equalizer is introduced.
   • It estimates a per-timestep gradient scale proxy ($N_t$) based on the SDE parameters and applies a robust normalization (using the median of scales across timesteps).
   • This ensures that optimization pressure is balanced across all timesteps, preventing low-noise steps from dominating the update.

B. Macro-Level: Dual Trust Region Optimization

To prevent long-horizon drift while maintaining plasticity, the authors introduce a Dual Trust Region objective combining two KL divergence constraints:

1. Position Control (Periodical Moving Anchor):

   • Instead of anchoring to the initial model ($\pi_0$), the reference policy ($\pi_{ref}$) is updated periodically (every $N$ steps).
   • This creates a dynamic trust region that tracks checkpoints closer to the manifold, preventing the policy from drifting too far from valid data distributions over long training horizons.

2. Velocity Control (Step-wise Constraint):

   • A standard step-wise KL constraint ($\pi_{k-1}$) limits the magnitude of parameter updates at each step, ensuring smooth local transitions.

3. Combined Objective:

   • The total loss combines the GRPO policy loss, the temporal gradient equalizer, and the dual KL regularizer:
     $$\mathcal{L}_{SAGE-GRPO} = -\frac{1}{G}\sum A_i \cdot \sum S_t \cdot \log \pi_\theta - \lambda_{KL} \cdot (\beta_{pos} D_{KL}(\pi_\theta || \pi_{ref}) + \beta_{vel} D_{KL}(\pi_\theta || \pi_{k-1}))$$

3. Key Contributions

1. Formulation: Defined video GRPO as a manifold-constrained exploration problem, identifying that standard ODE-to-SDE conversions inject excess noise that degrades rollout quality.
2. Micro-Level Innovation: Developed a Precise Manifold-Aware SDE with logarithmic curvature correction and a Gradient Norm Equalizer to stabilize sampling and balance optimization across timesteps.
3. Macro-Level Innovation: Proposed a Dual Trust Region mechanism using a moving anchor and step-wise constraints to solve the stability-plasticity trade-off, preventing off-manifold drift while allowing necessary exploration.
4. Empirical Validation: Demonstrated consistent improvements over state-of-the-art baselines (DanceGRPO, FlowGRPO, CPS) on the HunyuanVideo1.5 model.

4. Experimental Results

The method was evaluated on HunyuanVideo1.5 using the VideoAlign reward model (without fine-tuning the reward model itself).

• Quantitative Metrics:
  • SAGE-GRPO achieved superior performance in Visual Quality (VQ), Motion Quality (MQ), Text Alignment (TA), and visual metrics (CLIPScore, PickScore).
  • In the "Alignment-Focused" setting, SAGE-GRPO with Dual Moving KL achieved the highest overall scores, outperforming baselines significantly.
• Qualitative Analysis:
  • Generated videos showed reduced temporal jitter and better preservation of visual content.
  • Improved alignment with complex prompts involving emotional descriptions, occlusion, and long-range motion.
• User Study:
  • In a study with 29 evaluators, SAGE-GRPO won against baselines in 85.9% of Visual Quality comparisons, 75.8% for Motion Quality, and 79.2% for Semantic Alignment.
• Ablation Studies:
  • Confirmed that both the micro-level (SDE + Equalizer) and macro-level (Dual Trust Region) components are necessary for stability.
  • Showed that a two-stage KL weight schedule (warm-up followed by adaptive feedback) yields the most stable rewards.

5. Significance

This paper provides a critical theoretical and practical advancement for Reinforcement Learning in video generation. By recognizing the geometric structure of the data manifold, SAGE-GRPO solves the fundamental issue of "noise injection" that has plagued previous video alignment methods.

• Reliability: It makes GRPO for video as reliable as it is for text and images, enabling stable post-training alignment.
• Quality: It produces higher-fidelity videos with better temporal consistency and semantic alignment, crucial for applications in creative tools, education, and scientific communication.
• Generalizability: The concepts of manifold-aware SDEs and dual trust regions offer a new paradigm for stabilizing reinforcement learning in high-dimensional, structured generative tasks beyond just video.