💻 computer science

DEVIS-GRPO: Unleashing GRPO on Dynamic Extreme View Synthesis

The paper proposes DEVIS-GRPO, a novel online policy gradient framework that utilizes an accumulative sampling strategy (ADEVIS) and a multi-level reward function to enable efficient, high-quality trajectory-controlled video generation for extreme view synthesis without requiring expensive paired large-view training data.

Original authors: Yi Zuo, Huimin Wu, Lingling Li, Fang Liu, Licheng Jiao, Qing Li

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Zuo, Huimin Wu, Lingling Li, Fang Liu, Licheng Jiao, Qing Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a video of a busy street, but instead of just panning the camera slightly to the left or right, you want to spin around 180 degrees to see what's behind you, all while keeping the buildings and people looking exactly where they should be.

Current video AI tools are like nervous tourists: they can handle small turns, but if you ask them to spin around quickly, they get dizzy. The buildings might stretch, the people might disappear, or the background might suddenly change from a sunny park to a dark cave. This happens because the AI hasn't seen enough examples of such big, dramatic camera moves to know how to keep everything consistent.

The paper "DEVIS-GRPO" introduces a clever new way to teach AI how to handle these extreme camera spins without needing a massive library of pre-recorded "perfect" examples. Here is how they did it, broken down into simple concepts:

1. The Problem: The "Big Jump" vs. The "Baby Step"

Existing AI models try to jump from the starting camera angle to the final extreme angle in one giant leap. It's like trying to jump over a wide river in a single bound; you often miss the other side or land in the water.

The authors realized that if you break that big jump into a series of tiny, manageable steps, the AI can handle it much better. They call this ADEVIS (Accumulative Dynamic Extreme View Synthesis).

The Analogy: Imagine you are walking up a steep mountain. Instead of trying to fly to the top, you take small steps. You walk a little, look at the view, take another step, and look again. By the time you reach the top, you have built a continuous, smooth path. The AI does the same thing: it generates a tiny bit of the new view, uses that as a guide for the next tiny bit, and keeps going until the full 180-degree turn is complete.

2. The Training Trick: The "Group Game" (GRPO)

Usually, to train an AI to do something this complex, you need a huge dataset of "Before" and "After" videos that are perfectly matched. Getting these is expensive and hard to find.

The authors used a method called GRPO (Group Relative Policy Optimization). Think of this as a game show where the AI doesn't need a perfect answer key.

How it works: The AI tries to generate the video in many different ways at once (a "group"). Some attempts are okay, some are bad, and some are great.
The Judge: Instead of comparing the AI's work to a perfect human-made video, the system compares the AI's own attempts against each other. It asks, "Which of these 10 attempts looks the most consistent and smooth?"
The Reward: The AI gets a "high score" for the best attempts and learns to do more of that. This allows the AI to learn from its own "baby steps" without needing expensive, pre-recorded training data.

3. The "Bullet Time" Safety Net

When the AI takes these small steps, sometimes the math gets messy, especially when objects are hidden (occluded) or the depth is hard to guess. The authors tried four different strategies to fix this, but the best one was called BTA (Bullet Time Accumulation).

The Analogy: Imagine a movie scene where a character jumps, and the camera spins around them in "bullet time" (slow motion). The AI creates a "slow-motion" bridge between the old view and the new view. It repeats the first frame of the video a few times and makes the camera move super slowly for those few frames. This gives the AI extra time to figure out the geometry and fill in the gaps smoothly before speeding back up to the final angle.

4. The Results: No More "Glitchy" Spins

The team tested their method on three different datasets (simulated worlds, real iPhone videos, and professional movie clips). They found that their method:

Keeps things consistent: The buildings and people stay in the right place, even after a huge camera spin.
Follows the path: If you tell the camera to move 130 degrees, it actually moves 130 degrees, rather than getting lost.
Outperforms the competition: On the "Kubric-4D" dataset, their method improved the clarity of the video by over 21% compared to the second-best method. On iPhone videos, it reduced visual "fuzziness" by nearly 19%.

Summary

In short, DEVIS-GRPO is a new way to teach AI to spin its camera around wildly without getting confused. Instead of trying to learn the whole spin at once, it learns by taking tiny steps, playing a "group game" to figure out which steps are best, and using a "slow-motion bridge" to smooth out the tricky parts. This allows it to create high-quality, consistent videos from extreme angles without needing a massive library of perfect training examples.

Technical Summary: DEVIS-GRPO

Problem Statement
Trajectory-controlled video generation is essential for applications such as virtual reality, autonomous driving simulation, and cinematic production. While existing methods perform adequately under small-view camera motions (typically within 40°), they degrade significantly when handling large-view or "extreme" viewpoint changes (exceeding 90°). Current approaches generally fall into two paradigms: implicit camera embedding modeling and explicit reprojection modeling. Both struggle with extreme viewpoints: implicit methods often fail to follow the specified camera trajectory accurately, while explicit methods frequently produce inconsistencies between the foreground and background content of the source and generated videos. This limitation is largely attributed to the scarcity of synchronized video datasets featuring extreme viewpoint changes, as real-world acquisition of such data is challenging. Existing datasets typically contain only small-angle motions or static scenes, preventing models from generalizing to dynamic extreme viewpoints.

Methodology
The authors propose DEVIS-GRPO (Dynamic Extreme VIew Synthesis-GRPO), a framework that integrates Group Relative Policy Optimization (GRPO) into trajectory-controlled video generation. The core innovation lies in a novel sampling strategy called Accumulative Dynamic Extreme View Synthesis (ADEVIS).

ADEVIS Strategy: Instead of attempting to generate a large-angle video in a single step, ADEVIS decomposes a large camera motion ( $\Delta T$ ) into a sequence of smaller trajectory increments ( $\Delta T_1, \Delta T_2, \dots, \Delta T_n$ ). The model progressively accumulates these small steps to refine the generated view. This approach leverages prior knowledge from preceding steps to maintain geometric consistency during extreme transitions.
Accumulation Variants: To obtain the necessary geometry priors for each step, the paper investigates four accumulation strategies:
- Reference Video Accumulation (RFA): Uses the original source video for reprojection at every step.
- Reprojection Video Accumulation (RVA): Reprojects from the previously generated video ( $\hat{V}_{i-1}$ ).
- Occlusion Region Accumulation (ORA): Fuses the reprojection from the previous step with the current step, specifically targeting closer regions or continuous regions where depth estimation is more reliable (ORA-dt and ORA-cc).
- Bullet Time Accumulation (BTA): Creates a "bullet time" transition by repeating the first frame of the source video and decomposing the initial motion into smaller, consecutively increasing changes. This strategy was found to be the most effective in minimizing misalignments at junctions.
GRPO Optimization: The framework employs GRPO to optimize the policy model without requiring a separate value model.
- Stochastic Sampling: During training, the model samples diverse trajectory groups by randomizing the number of accumulation steps ( $n$ ) and the magnitude of motion at each step.
- Reward Function: A multi-level consistency-quality reward function evaluates generated samples based on pixel-level fidelity (PSNR, SSIM in visible regions), semantic-level fidelity (LPIPS), and video quality metrics (Temporal Flicker, Motion Smooth, Aesthetic Quality, Imaging Quality).
- Group Relative Advantage: The model is updated by maximizing the objective based on the relative advantages of samples within a group, rather than absolute scores.
Inference: Unlike the training phase which uses multi-step accumulation, the inference process utilizes the optimized model to generate large-angle videos directly in a single step, ensuring efficiency.

Key Contributions

First Application of GRPO to Extreme View Synthesis: The paper introduces DEVIS-GRPO as the first online policy gradient method specifically designed for trajectory-controlled video generation under extreme viewpoints.
ADEVIS Sampling Strategy: The proposal of Accumulative Dynamic Extreme View Synthesis, which eliminates the need for expensive, paired large-view video datasets for warm-starting the policy model. It achieves large-view motions by accumulating small-view increments.
Diverse Accumulation Strategies: The design and evaluation of four distinct accumulation strategies (RFA, RVA, ORA, BTA) to explore the paradigm of progressive view synthesis, with BTA identified as the superior approach.
Multi-Level Reward Function: The development of a reward function that balances consistency (pixel and semantic) with video quality to guide the reinforcement learning process.

Results
Experiments were conducted on the Kubric-4D, iPhone, and DL3DV datasets.

Quantitative Performance: On the Kubric-4D dataset, DEVIS-GRPO achieved relative improvements of 21.57% in PSNR and 7.31% in SSIM in non-occlusion areas compared to the second-best method. On the iPhone dataset, the LPIPS metric was reduced by 18.56%.
Consistency and Adherence: The method demonstrated superior performance in maintaining temporal and textural coherence, as well as strict adherence to the specified camera trajectory, outperforming baselines like GCD, EX-4D, and TrajectoryCrafter.
Ablation Studies: Comparisons with Vanilla-GRPO showed that DEVIS-GRPO generates more diverse samples with better consistency metrics. Comparisons with SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization) paradigms confirmed that the GRPO approach with group-relative advantages yields the best performance.

Significance
The paper claims that DEVIS-GRPO addresses the critical bottleneck of data scarcity in extreme view synthesis. By leveraging an accumulative sampling strategy, it enables effective training without the need for synchronized multi-view supervision or expensive large-angle video pairs. The framework successfully bridges the gap between small-view controllability and the generation of high-quality, consistent videos under extreme camera motions, setting a new standard for trajectory-controlled video generation.