💻 computer science

Epipolar Geometry Improves Video Generation Models

This paper demonstrates that integrating classical epipolar geometry constraints via preference-based optimization significantly enhances the geometric consistency and motion stability of modern video diffusion models, reducing epipolar error by 31% and improving human-rated consistency to 72% without compromising visual quality.

Original authors: Orest Kupyn, Théo Uscidda, Marta Tintore Gazulla, Fabian Manhardt, Federico Tombari, Christian Rupprecht

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Orest Kupyn, Théo Uscidda, Marta Tintore Gazulla, Fabian Manhardt, Federico Tombari, Christian Rupprecht

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented but slightly clumsy artist. This artist can paint beautiful, moving pictures (videos) based on your descriptions. They know how to paint a car, a tree, or a person. But when they try to make the camera move around these objects, things get weird. The car might suddenly stretch like taffy, the background might warp, or the camera might jitter in a way that feels physically impossible. The artist knows what things look like, but they don't quite understand the rules of how 3D space works.

This paper introduces a way to teach this artist the "rules of the road" for 3D space, using a very old, very reliable set of instructions called Epipolar Geometry.

Here is a simple breakdown of how they did it:

1. The Problem: The "Wobbly Camera"

Even though modern AI video models are trained on millions of videos, they still struggle with 3D consistency.

The Issue: If you watch a video where the camera moves, the objects in the scene should move in a predictable, mathematical way relative to each other. If they don't, the video looks fake, glitchy, or "dream-like" in a bad way.
The Analogy: Imagine watching a movie where the actors are walking on a stage, but the stage floor keeps rippling like water. It breaks your illusion that you are watching a real place.

2. The Solution: The "Old School Ruler"

Instead of trying to teach the AI a new, complex way to understand 3D space from scratch, the researchers used a classic tool from the 1980s called Epipolar Geometry.

What is it? Think of it as a strict geometric rulebook. If you take two photos of the same scene from different angles, there is a mathematical line (an "epipolar line") that any matching point must follow.
The Metaphor: Imagine you are playing a game of "Connect the Dots" with two different photos. If the dots don't line up according to the strict rules of perspective, the picture is broken. This paper uses that strict rulebook as a referee.

3. The Method: "Pick the Best, Ignore the Rest"

The researchers didn't try to force the AI to learn the math directly (which is hard because the math is "non-differentiable," meaning it's like trying to teach a computer to understand a physical ruler using only software code). Instead, they used a technique called Preference Optimization.

Here is how the training loop worked:

The Prompt: They gave the AI a prompt (e.g., "A camera flying over a city").
The Batch: The AI generated three different videos for that same prompt.
The Referee: They ran the videos through the "Old School Ruler" (the Epipolar Geometry check).
- Video A: The camera moves smoothly, and the buildings stay rigid. (Passes the test).
- Video B: The buildings warp and the camera jitters. (Fails the test).
The Lesson: The AI was told: "You did better on Video A than Video B. Remember that feeling."
The Result: Over time, the AI learned to generate more videos like A and fewer like B, simply by trying to satisfy the geometric rules.

4. The Tricky Part: Static vs. Dynamic

There was a catch. The geometric rules work perfectly when the camera moves but the objects stay still (like a cityscape). But what if the video has a running dog or a flying bird? The rules get messy because the dog is moving on its own.

To solve this, the researchers used a clever trick:

Training: They only trained the AI on videos where the camera moved but the world was still.
The Magic: Even though they only taught it about static worlds, the AI learned the general principle of how a camera should move. When they tested it on videos with moving objects (like people running), the AI still applied those smooth, stable camera rules.
The Analogy: It's like teaching a driver to drive perfectly on a straight, empty highway. When you put them in traffic, they still know how to steer smoothly because they learned the fundamental skill of steering, not just how to avoid other cars.

5. The Results: Smoother, Realer, Better

The paper claims that by using these simple geometric rules instead of complex, learned "human-like" scores, they got better results:

Less Glitching: The "wobbly" artifacts dropped significantly.
Better 3D Feel: The videos looked much more like real 3D spaces.
Human Approval: When humans watched the videos, they preferred the new version 72% of the time compared to the old version (which was only 54% consistent).
No Loss of Creativity: The AI didn't stop making cool videos; it just made them physically plausible.

Summary

The paper essentially says: "Don't reinvent the wheel."
Modern AI is great at learning patterns from data, but it sometimes forgets the basic laws of physics and geometry. By adding a simple, mathematically strict "referee" (Epipolar Geometry) to the training process, the AI learned to generate videos that are not just pretty, but also geometrically stable and realistic, without needing to be taught complex 3D reconstruction from scratch.

Technical Summary: Epipolar Geometry Improves Video Generation Models

Problem Statement

Modern video generation models, particularly those based on latent diffusion transformers trained with rectified flow techniques, have achieved significant progress in generating realistic content from text and image conditions. However, these models struggle with fundamental geometric inconsistencies, unstable motion, and visual artifacts that break the illusion of realistic 3D scenes. Despite being trained on vast datasets that are inherently 3D-consistent, the models fail to capture basic geometric principles, often producing content with imperfect geometry and perspective flaws.

Existing approaches to enhance 3D consistency face several limitations:

Noise Optimization: Methods relying on noise optimization or explicit guidance through point clouds can constrain generation capabilities.
Latent-Space Optimization: Computing direct geometric losses in latent space is difficult.
Subjective Alignment: Post-training alignment using human preference data (e.g., via Reinforcement Learning) introduces noisy signals, is expensive to collect, and may fail to capture the objective geometric principles required for 3D consistency. Human judgments are inherently subjective and do not necessarily align with mathematical geometric constraints.

Methodology

The authors propose a method to bridge modern video diffusion models with classical computer vision algorithms. Instead of incorporating explicit 3D guidance during generation or relying on subjective human annotations, the approach uses well-established, non-differentiable geometric constraints as reward signals within a preference-based optimization framework.

Core Approach: Epipolar Geometry Optimization

The method leverages epipolar geometry to assess 3D consistency between video frames. The core logic is as follows:

Sampling: Multiple videos are generated conditioned on the same prompt, resulting in diverse camera trajectories with varying degrees of geometric coherence.
Metric Computation: For pairs of frames, point correspondences are established (using SIFT or learned descriptors). A fundamental matrix is estimated via RANSAC, and the Sampson error is calculated. This error quantifies how well corresponding points satisfy the epipolar constraint ( $x'^\top F x = 0$ ). Lower Sampson error indicates better adherence to projective geometry and higher 3D consistency.
Preference Construction: Videos generated from the same prompt are ranked based on their mean Sampson error. The video with the lower error is designated as the preferred sample ( $x_w$ ), and the one with higher error as the less preferred sample ( $x_l$ ).
Optimization (Flow-DPO): The model is fine-tuned using Direct Preference Optimization (DPO) adapted for rectified flow models (Flow-DPO). This objective requires only pairwise preferences rather than absolute reward values, bypassing the need for differentiable reward functions. The loss function minimizes the velocity prediction error for preferred samples relative to less preferred ones, guided by the epipolar metrics.

Addressing Key Challenges

The paper identifies and addresses two primary challenges:

Motion-Consistency Tradeoff: A model could theoretically achieve perfect 3D consistency by generating static videos (repeating a single frame). To prevent this degenerate solution, a temporal variation penalty ( $L_{temp}$ ) is added to the objective function. This term penalizes low variance across the temporal dimension, encouraging the model to maintain motion while improving consistency.
Generalization to Dynamic Scenes: Epipolar constraints assume static scenes with camera motion; dynamic objects violate this assumption. To ensure the model generalizes to dynamic content despite being trained on static scenes:
- Low-Rank Adaptation (LoRA): Only a low-rank adapter is fine-tuned, preserving the base model's capacity for diverse dynamic content.
- Implicit KL-Divergence: The Flow-DPO objective naturally keeps the fine-tuned model close to the reference distribution, preventing overfitting to static scene characteristics.
- Adaptive Timestamp Weighting: The loss is weighted by timestep, assigning lower weights to early steps where general scene motion is encoded and higher weights to later steps where fine spatial details (and potential geometric violations) are generated.

Key Contributions

Epipolar Geometry Optimization: The introduction of a method to fine-tune video diffusion models using epipolar geometry constraints (specifically Sampson distance) as reward signals. This approach enforces 3D consistency without requiring end-to-end differentiability or explicit 3D supervision.
Analysis of Alignment Challenges: The paper analyzes the tradeoff between motion and consistency and the challenge of generalizing to dynamic scenes. It proposes simple, efficient solutions (temporal penalty, LoRA, KL regularization, and adaptive weighting) to address these issues.
Comprehensive Evaluation Framework: The authors develop an extensive evaluation protocol measuring perceptual quality, 3D consistency, motion stability, and generalization. They release a dataset of 162,000 generated videos with geometric consistency annotations to facilitate further research.
Demonstration of Classical CV Utility: The work demonstrates that a basic geometric constraint (Sampson, 1982) can recover capabilities that large-scale data-driven models fail to learn, bridging data-driven learning with classical computer vision.

Results

The proposed method was evaluated on the Wan-2.1 (1.3B parameters) model and compared against baselines and other alignment strategies.

3D Consistency: The epipolar-aligned model reduced the Sampson error by 31% (from 0.190 to 0.131). In terms of 3D scene reconstruction (via Gaussian Splatting), it improved PSNR by 3.6% and SSIM by 3.2%.
Human Evaluation: Human raters labeled 71.8% of the aligned model's videos as geometrically consistent, a significant improvement over the baseline's 54.1%. In pairwise comparisons, the aligned model was preferred over the baseline in 60.4% of cases for inconsistent content, while preserving quality for already-consistent content.
Motion Quality: The model achieved a 53.2% win rate in human preference for motion quality, indicating improved stability and reduced jitter, despite a slight reduction in dynamic degree (which the authors argue is a necessary tradeoff for stability).
Visual Quality: The method improved aesthetic quality and background consistency scores. It achieved a 72.0% win rate in VideoReward visual quality evaluations.
Generalization: Despite training exclusively on static scenes with dynamic cameras, the model generalized effectively to dynamic scenes (VBench 2.0, MiraData, VideoReward benchmarks), achieving win rates of 57.9%, 58.5%, and 58.5% respectively.
Comparison to Alternatives: The approach outperformed supervised fine-tuning on multi-view datasets and other alignment methods (Flow-RWR, DRO), which either overfit to dataset specifics or failed to learn from the gap between consistent and inconsistent trajectories.

Significance

The paper claims that classical geometric constraints provide more stable and reliable optimization signals than modern learned metrics (such as those from VLMs or human annotations), which can be noisy or subjective. By aligning models with fundamental geometric principles via preference optimization, the authors demonstrate that it is possible to significantly improve the 3D consistency and motion stability of video generation models without compromising their creative flexibility or requiring explicit 3D supervision. The work highlights that simple, mathematically grounded constraints can effectively complement deep learning, addressing limitations inherent in purely data-driven approaches.