VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward

1. Problem Statement

Large-scale video diffusion models have achieved impressive visual fidelity but suffer from a critical lack of geometric and temporal consistency. Common failure modes include:

• Geometric Drift: Objects change shape or position inconsistently across frames.
• Unstable Camera Trajectories: Jittery or physically implausible camera movements.
• Inconsistent Scene Structure: The 3D layout of the scene fails to remain coherent over time.

Existing solutions fall into two categories, both with significant limitations:

1. Architectural Modifications: Adding geometry-conditioning modules or auxiliary losses. These often compromise the generalization capabilities of large-scale pre-trained models and increase computational complexity.
2. Post-Training Alignment (RL-based): Methods like Epipolar-DPO or VideoGPA that use Reinforcement Learning (RL) to align models with geometric priors. However, these methods typically:
   • Rely on static scene assumptions, failing in dynamic environments.
   • Compute rewards in RGB pixel space, requiring repeated VAE decoding. This incurs massive computational overhead and introduces noise sensitivity.
   • Often rely on offline preference data, limiting optimization efficiency.

2. Methodology: VGGRPO

The authors propose VGGRPO (Visual Geometry GRPO), a framework that performs geometry-aware post-training directly in the latent space of the video diffusion model, eliminating the need for RGB decoding during reward computation.

A. Latent Geometry Model (LGM)

To enable geometry prediction directly from latent representations, the authors introduce a Latent Geometry Model:

• Stitching Mechanism: They connect the output of a video VAE encoder (latents) to a pre-trained Geometry Foundation Model (e.g., Any4D or VGGT) via a lightweight, learnable 3D convolutional connector.
• Training: The connector is trained to map video latents to the intermediate feature space of the geometry foundation model, minimizing feature alignment error.
• Capability: The LGM can predict 4D scene geometry (camera pose, depth, point maps, and scene flow) directly from denoised latents ($z_0$) without ever decoding them into RGB images. This supports both static and dynamic 4D reconstruction.

B. Latent-Space Group Relative Policy Optimization (GRPO)

VGGRPO utilizes GRPO, an on-policy RL algorithm, to optimize the video generation policy.

• Process: For a given prompt, the model generates a group of $K$ video trajectories.
• Reward Computation: Instead of decoding to RGB, the LGM extracts geometric predictions from the latent outputs. Two complementary rewards are calculated entirely in latent space:
  1. Camera Motion Smoothness Reward ($r_{motion}$): Penalizes jittery trajectories by measuring the acceleration of the camera center and angular velocity. It encourages physically plausible, smooth camera motion.
  2. Geometry Reprojection Consistency Reward ($r_{geo}$): Enforces cross-view coherence. It constructs a 3D point cloud from the predicted geometry, reprojects it into different views, and compares the rendered depth with the predicted depth. It specifically targets the worst-performing views to minimize geometric drift.
• Optimization: The advantage is computed based on the normalized sum of these rewards. The policy is updated using a clipped surrogate objective with a KL-divergence penalty to preserve the base model's generalization.

C. Test-Time Guidance (Optional)

Since the LGM is differentiable, the framework allows for test-time reward guidance. Gradients from the geometry rewards can be back-propagated through the LGM to adjust the diffusion velocity field during inference, improving consistency without additional training.

3. Key Contributions

1. Latent-Space Geometry Rewards: Demonstrated that reliable geometry-driven rewards can be computed directly in the latent space, eliminating the computational bottleneck and noise sensitivity of repeated RGB decoding.
2. Latent Geometry Model (LGM): Proposed a model-agnostic architecture that stitches diffusion latents to geometry foundation models, enabling 4D geometry extraction from latents without pixel-space inputs.
3. VGGRPO Framework: Introduced a GRPO-based post-training method with complementary motion and geometry rewards that effectively improves world consistency for both static and dynamic scenes.
4. Efficiency & Scalability: Achieved significant reductions in training time and GPU memory usage compared to RGB-based alignment methods while maintaining or improving visual quality.

4. Experimental Results

The authors evaluated VGGRPO on Wan2.1-1B and Wan2.2-5B backbones against baselines like SFT, Epipolar-DPO, and VideoGPA.

• Quantitative Performance:
  • Geometric Consistency: VGGRPO achieved the lowest Sampson epipolar error (static scenes) and highest VideoReward win rates for Visual Quality (VQ) and Motion Quality (MQ) on both static and dynamic benchmarks.
  • Dynamic Scenes: Unlike baselines that degrade significantly on dynamic, non-rigid motion, VGGRPO remained robust, showing superior performance in maintaining coherent structure.
  • Generalization: On standard VBench captions, VGGRPO improved Subject/Background consistency and Aesthetic Quality without sacrificing the model's original generalization capabilities.
• Qualitative Improvements: Visual comparisons showed VGGRPO produces videos with smoother camera trajectories, reduced geometric drift, and fewer temporal artifacts (flickering, object morphing) compared to baselines.
• Efficiency: The latent-space reward computation reduced peak GPU memory usage by ~11% (76.8GB $\to$ 68.6GB) and reduced wall-clock time by ~24.5% compared to RGB-based methods.

5. Significance

VGGRPO represents a paradigm shift in aligning video generation models with physical and geometric constraints. By moving the reward computation from the pixel space to the latent space, it solves the efficiency and scalability issues that have hindered previous RL-based alignment methods.

Its ability to handle dynamic 4D scenes makes it particularly valuable for downstream applications requiring world consistency, such as Embodied AI, physics-aware simulation, and autonomous driving, where stable camera motion and coherent 3D geometry are non-negotiable. Furthermore, the method preserves the broad generalization of large-scale pre-trained models, offering a practical and efficient path to high-fidelity, world-consistent video generation.