🤖 AI

VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation

VideoGPA introduces a data-efficient, self-supervised framework that distills geometric priors from foundation models into dense preference signals to guide video diffusion models via Direct Preference Optimization, thereby significantly enhancing 3D structural consistency, temporal stability, and motion coherence without requiring human annotations.

Original authors: Hongyang Du, Junjie Ye, Xiaoyan Cong, Runhao Li, Jingcheng Ni, Aman Agarwal, Zeqi Zhou, Zekun Li, Randall Balestriero, Yue Wang

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongyang Du, Junjie Ye, Xiaoyan Cong, Runhao Li, Jingcheng Ni, Aman Agarwal, Zeqi Zhou, Zekun Li, Randall Balestriero, Yue Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Wobbly World" of AI Video

Imagine you ask a talented but slightly confused artist to draw a video of a car driving down a street. They are great at painting the colors and the clouds, but they don't quite understand how 3D space works.

As the video plays, the car might suddenly stretch like a rubber band, the wheels might detach and float away, or the street might twist into a spiral. In the paper, the authors call this "spatial drift" and "object deformation."

Current AI video models are like artists who focus only on making every single frame look beautiful on its own, without checking if the objects in frame #10 match the objects in frame #11. They lack a sense of "physics" or "geometry."

The Solution: VideoGPA (The Geometry Coach)

The authors created a new method called VideoGPA (Video Geometric Preference Alignment). Think of VideoGPA not as a new artist, but as a strict geometry coach hired to train the existing AI video model.

Here is how the coach works, step-by-step:

1. The "Magic Mirror" (The Geometry Foundation Model)

The coach uses a special tool called a Geometry Foundation Model. You can think of this tool as a "Magic Mirror" that looks at a flat 2D video and instantly figures out the hidden 3D structure behind it.

If you show it a video of a spinning cube, the Magic Mirror builds a 3D model of that cube in its head.
If the video is fake (e.g., the cube melts into a puddle), the Magic Mirror gets confused because it can't build a solid 3D model from a melting mess.

2. The "Re-Projection Test" (The Exam)

The coach takes a video generated by the AI and runs it through the Magic Mirror.

Step A: The Mirror builds a 3D model of the scene.
Step B: The Mirror tries to "project" that 3D model back onto a 2D screen to see if it matches the original video.
The Score: If the video was geometrically correct, the 3D model will perfectly recreate the video. If the video was wobbly or distorted, the recreation will look blurry or wrong. This difference is the 3D Consistency Score.

3. The "Taste Test" (Preference Learning)

Instead of forcing the AI to learn complex math rules, the coach uses a technique called Direct Preference Optimization (DPO).

The coach generates two videos from the same prompt (e.g., "a cat walking").
Video A is wobbly (low score). Video B is solid (high score).
The coach simply tells the AI: "I like Video B better because it makes sense in 3D space. Stop making Video A."
The AI learns to avoid the "wobbly" path and stick to the "solid" path.

Why This is Special

The paper highlights three key advantages of this approach:

No Human Teachers Needed: Usually, to teach an AI what "good" looks like, you need thousands of humans to watch videos and vote. VideoGPA is self-supervised. The "Magic Mirror" acts as the teacher, automatically grading the videos without a single human needing to look at them.
Tiny Data, Big Results: The coach only needs to show the AI about 2,500 pairs of "good vs. bad" videos to fix the problem. This is a tiny amount of data compared to the billions of images the AI was originally trained on.
Lightweight Training: The AI doesn't need to be retrained from scratch. The authors only tweaked about 1% of the AI's internal settings (using a method called LoRA). It's like giving the AI a pair of glasses to see 3D space better, rather than rebuilding its brain.

The Results: A More Stable World

After this training, the AI's videos become much more stable:

No More Melting: Objects keep their shape even when the camera moves around them.
No More Drifting: A car driving forward stays a car; it doesn't suddenly turn into a boat or slide sideways.
Better Textures: The details (like the pattern on a wall) stay consistent instead of flickering or changing randomly.

The Bottom Line

The paper argues that the reason AI videos look "weird" isn't because the AI is bad at drawing; it's because the AI was never explicitly taught to respect the laws of 3D geometry.

VideoGPA solves this by using a 3D "reality check" to guide the AI. It teaches the model that if a video doesn't make sense in 3D space, it's a "bad" video. The result is a video generator that creates scenes that feel physically real and stable, all without needing human feedback or massive amounts of new data.

Technical Summary: VideoGPA

Problem Statement

Recent Video Diffusion Models (VDMs) have achieved remarkable visual fidelity and generalizability but fundamentally struggle to maintain 3D structural consistency. Despite being trained on billion-scale datasets, these models often produce videos with object deformation, spatial drift, and geometric collapse over time. The authors hypothesize that this failure stems from the nature of standard denoising objectives, which incentivize pixel-level statistical matching but lack explicit geometric regularization. Consequently, models learn to hallucinate plausible textures without effectively injecting 3D consistency into the latent space. While recent Geometry Foundation Models (GFMs) can infer dense 3D structure from 2D observations, current VDMs do not inherently leverage these priors during generation.

Methodology: VideoGPA

The paper introduces VideoGPA (Video Geometric Preference Alignment), a data-efficient, self-supervised framework designed to distill geometry priors into pre-trained VDMs without requiring human annotations or retraining from scratch. The approach aligns the generative distribution toward a 3D-consistent manifold using Direct Preference Optimization (DPO).

1. Self-Supervised Preference Modeling

Instead of relying on human feedback, VideoGPA constructs preference pairs automatically using a Geometry Foundation Model (GFM) as a reward signal backbone.

Reconstruction-Based Consistency: For a generated video, the GFM infers a dense 3D structure (point clouds) and camera poses. The system then renders a "reference" video by reprojecting this 3D structure back into 2D frames.
Consistency Score: The system calculates a 3D Consistency Score by measuring the reconstruction error (MSE + LPIPS) between the original generated video and the reprojected reference. A low error indicates high geometric validity, while high error signals geometric artifacts.
Pair Construction: For a given prompt, multiple video samples are generated. The sample with the lowest reconstruction error is designated as the "winner" ( $x_w$ ), and the one with the highest error as the "loser" ( $x_l$ ).

2. Direct Preference Optimization (DPO) for Video

The framework adapts DPO to the v-prediction parameterization common in modern video diffusion transformers (DiTs).

Objective: The model is trained to minimize the difference in velocity prediction errors between the winner and loser samples.
Loss Function: The DPO loss is formulated as the negative log-likelihood of the reward margin between winning and losing samples, where the reward is derived from the difference in velocity prediction errors ( $E$ ) relative to a reference policy.
Efficiency: The method employs LoRA (Low-Rank Adaptation) to fine-tune only a small fraction (~1%) of the model parameters, making the process computationally lightweight.

3. Data Curation

The training data is curated from the DL3DV-10K dataset.

Image-to-Video (I2V): Uses initial frames as prompts with scripted camera motion primitives (e.g., "pull back," "orbit") to elicit diverse trajectories where geometric inconsistencies are likely to emerge.
Text-to-Video (T2V): Uses captions generated by CogVLM2-Video to evaluate alignment under unconstrained language inputs.
Filtering: Samples are filtered to ensure motion salience, sufficient geometric margin between pairs, and exclusion of low-quality "winners."

Key Contributions

Self-Supervised Geometric Alignment: A novel paradigm that leverages GFMs to generate dense, automated geometric preference signals, eliminating the need for human annotations in 3D-consistent video alignment.
Scene-Level Consistency Metric: Unlike prior works relying on sparse pairwise constraints (e.g., epipolar lines), VideoGPA utilizes a global reprojection constraint. This ensures that the entire scene admits a single, coherent 3D explanation, preventing spatial drift that local metrics might miss.
Data-Efficient Post-Training: Demonstrates that significant improvements in 3D consistency can be achieved with minimal data (~~2,500 preference pairs) and minimal parameter updates (~~1% of model weights via LoRA).
Motion Coherence Emergence: The paper observes that enforcing geometric consistency acts as a regularizer for the motion manifold, inadvertently improving the temporal stability and physical plausibility of dynamic object motion, even in scenes not explicitly constrained to be static during training.

Experimental Results

Extensive experiments were conducted on CogVideoX (I2V and T2V settings) and Wan2.2.

Quantitative Performance: VideoGPA consistently outperforms baselines (including SFT, Epipolar-DPO, and GeoVideo) across multiple metrics:
- 3D Reconstruction Error: Improved PSNR, SSIM, and reduced LPIPS.
- Geometric Consistency: Achieved higher Multi-View Consistency Scores (MVCS) and lower 3D Consistency Scores (3DCS) and Epipolar errors.
- Human Preference: In blind user studies, VideoGPA achieved a 76.0% win rate in I2V and 60.33% in T2V against the base model, significantly outperforming SFT and Epipolar-DPO.
Qualitative Improvements: Visual analysis shows VideoGPA effectively suppresses object splitting, preserves rigid-body integrity under camera motion, reduces texture flickering, and maintains object attribute consistency (color, material) better than baselines.
Generalization: The method generalizes to out-of-distribution datasets (WebVid, Panda-70M) and newer backbones (Wan2.2), maintaining performance even in dynamic scenes with complex object motion.

Significance and Claims

The paper claims that the geometric failures in current video generators are largely attributable to objective misalignment rather than architectural limitations. By distilling geometry priors from foundation models into the diffusion process via preference optimization, VideoGPA demonstrates that:

3D consistency is learnable: Pre-trained models can be steered toward physically plausible manifolds without explicit structural priors in the generation pipeline.
Geometry stabilizes motion: Enforcing geometric constraints serves as a foundational anchor, allowing the model to better disentangle camera movement from object motion, resulting in more coherent dynamic generation.
Efficiency: High-quality 3D-consistent video generation can be achieved through lightweight post-training, avoiding the computational cost of training massive models from scratch or the logistical burden of collecting human preference data.

The authors acknowledge limitations regarding the computational cost of feed-forward geometric reconstruction for very long video sequences but suggest this is a solvable challenge with future advancements in lightweight geometric foundation models.