💻 computer science

GeoT2V-Bench: Benchmarking 3D Consistency in Text-to-Video Models via 3D Reconstruction

This paper introduces GeoT2V-Bench, a novel reconstruction-based benchmark that evaluates the 3D consistency of camera-prompted text-to-video models by analyzing their ability to support explicit rigid 3D reconstruction, revealing that visible motion and static rendering metrics often capture complementary failure modes.

Original authors: Chenrui Fan, Paolo Favaro

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenrui Fan, Paolo Favaro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Magic Camera" Test

Imagine you have a magic camera that can take photos of a room, but instead of you moving the camera, you just tell it, "Fly around this statue," or "Zoom through this hallway."

Artificial Intelligence (AI) models are getting very good at doing this. They generate videos that look real. But here is the catch: Do they actually understand 3D space?

Sometimes, an AI might generate a video that looks smooth and pretty, but if you tried to build a real 3D model of that scene from the video, it would fall apart. The statue might melt, the walls might warp, or the camera path might make no sense.

This paper introduces a new test called GeoT2V-Bench. Instead of just asking, "Does this video look pretty?" it asks, "If I tried to reconstruct this video as a real, solid 3D world, would it hold together?"

The Problem: The "Flat" Illusion

The authors explain that current tests for AI videos are like judging a movie by its poster. They check if the colors are nice, if the text matches the picture, or if the movement is smooth.

But for "camera-motion" prompts (like "orbit around a tree"), the video is supposed to be a synthetic photograph of a static object.

The Failure: An AI might generate a video where the camera seems to move, but the object just stretches and squashes like a piece of taffy to fake the effect. It looks like a 3D orbit, but it's actually just a 2D trick.
The Old Way: Previous tests (like GeCo) check if the local geometry makes sense (e.g., "Does the wall look straight?"). But a video can look locally straight and still fail the global test (e.g., "The whole room is spinning in a way that defies physics").

The Solution: The "Architect's Blueprint"

The authors built a diagnostic pipeline that acts like a skeptical architect. They don't just look at the video; they try to rebuild the scene from it.

Here is how their "Architect's Test" works, step-by-step:

1. The "Guess the Camera" Step (VGGT)

First, the system looks at the generated video and tries to guess: "If this were a real video, where was the camera at every single second?"

The Analogy: Imagine watching a shaky home video and trying to trace the path the person holding the camera took.
The Test: If the AI video is too weird or static, the system can't guess a camera path. This is a red flag.

2. The "Flexible Clay" Step (DeformableGS)

Next, the system tries to build a 3D model of the scene using a super-flexible material (like DeformableGS, a type of 3D Gaussian Splatting).

The Analogy: Imagine trying to mold a statue out of warm, stretchy clay. If the video has weird glitches, the clay stretches and twists to match the video perfectly.
The Result: This step almost always succeeds because the clay is so flexible. It proves the video can be matched, but only if we allow the world to deform.

3. The "Hard Rock" Step (MedianGS)

This is the most important part. The system takes that stretchy clay model and tries to turn it into hard rock. It forces the model to be static (unchanging).

The Analogy: Now, imagine trying to fit that same video into a rigid, unchangeable stone statue.
The Test:
- If it fits: Great! The video really does look like a camera moving around a solid object.
- If it cracks: The video was relying on the "stretchy clay" (time-varying tricks) to look good. It cannot be explained as a single, solid 3D scene.

4. The "Motion Check" (CEMR)

Finally, the system checks the movement.

The Analogy: If you walk around a real statue, the background moves in a specific way (parallax). The system checks if the AI video's movement matches what a real camera would see.
The Result: Sometimes the AI gets the look right (the statue looks good) but the movement wrong (the background slides the wrong way). This test catches that.

What They Found (The "Report Card")

The authors tested 12 different AI video models. They didn't give them a simple "Pass" or "Fail" grade. Instead, they gave them a continuous profile (a detailed report card) that showed:

Who faked it: Some models (like MAGI-1) looked great on paper but failed the "active evidence" check—they didn't actually move the camera enough to prove it was a 3D scene.
Who stretched the truth: Some models (like HunyuanVideo) had high "deformation energy," meaning their 3D models had to stretch and warp wildly to match the video.
Who got the motion wrong: Some models had low errors (looked good) but terrible "flow agreement" (the movement didn't match the camera path).

The "Control Lab" (ControlBench)

To make sure their test wasn't broken, they created a "Control Lab." They took real videos and did weird things to them:

Frozen: Stopped the video (should fail the motion test).
Digital Zoom: Just zoomed in (should look like a 2D zoom, not a 3D move).
Texture Repaint: Changed the colors mid-video.

They ran these through their system to see if the test correctly identified these as "fake" or "broken" 3D scenes. It did.

The Bottom Line

The paper argues that we can no longer just ask, "Is this video pretty?" for AI video generation. We must ask, "Is this video a valid record of a 3D world?"

GeoT2V-Bench is the tool that answers that question. It doesn't just look at the surface; it tries to build the world underneath. If the world collapses when you try to build it, the video has failed the test, no matter how beautiful it looks.

Technical Summary: GeoT2V-Bench

Problem Statement

Text-to-video (T2V) models are increasingly prompted with camera-motion instructions (e.g., "orbit a statue," "dolly through a room") to synthesize virtual camera captures. While existing benchmarks evaluate visual plausibility, temporal smoothness, and text alignment, they fail to verify whether the generated frames constitute coherent multi-view evidence for a single static 3D scene. Visually plausible videos may still violate global 3D consistency by exhibiting drifting textures, 2D parallax imitations, or non-physical scene deformations. Furthermore, reconstruction-based evaluation is inherently fragile: camera pose estimation can fail due to weak parallax, and static 3D reconstruction can fail if the frames do not correspond to a stable scene. Current metrics often conflate video generation failures with reconstruction backend failures or rely on local geometric consistency (e.g., GeCo) which does not guarantee global static-scene explainability.

Methodology: GeoT2V-Bench

The authors introduce GeoT2V-Bench, a reconstruction-based diagnostic benchmark that evaluates camera-prompted T2V outputs as synthetic static-scene acquisitions. Rather than producing a scalar pass/fail label, the benchmark generates a continuous reconstruction profile.

Pipeline Overview

The evaluation pipeline processes a generated video $V$ and a prompt $p$ through the following stages:

Camera Estimation (VGGT):
The system estimates per-frame camera intrinsics ( $K_t$ ) and poses ( $T_t$ ) using VGGT [33], a feed-forward geometry estimator. This establishes a candidate camera trajectory. The pipeline first checks for "active acquisition evidence" (sufficient apparent motion and coherent trajectory) to ensure the video is not trivially static.
Flexible Fit (DeformableGS):
Using the estimated cameras, the system fits an over-parameterized DeformableGS [40] representation. This flexible 4D model can absorb inconsistencies (e.g., view-dependent repainting, non-physical drift) as temporal variations. The rendering error of this fit ( $E_{fit}$ ) serves as a baseline for how well the video can be explained by any time-varying geometry.
Static Proxy (MedianGS):
To test for static-scene explainability, the system derives a MedianGS proxy ( $G_{med}$ ) by applying a temporal-median aggregation to the DeformableGS parameters (position, rotation, and scale offsets). This creates a static 3D representation that represents the temporally stable component of the sequence.
Same-Camera Comparison:
The MedianGS proxy is rendered along the original VGGT-estimated camera path. The system compares this static render against the generated video to measure how well a static scene explains the observations.

Key Metrics

The benchmark reports a profile of continuous diagnostics:

$E_{LPIPS}^{med}$ (MedianGS Static Rendering Error): Measures the per-frame appearance difference between the generated video and the static MedianGS render.
CEMR (Camera-Explained Motion Ratio): Compares the optical flow of the generated video with the optical flow induced by the static MedianGS render. High CEMR indicates the static scene explains the temporal motion; low/negative CEMR indicates the motion is not explained by the static geometry.
$\Delta_{stat}$ (Static-vs-Flexible Gap): Defined as $E_{LPIPS}^{med} - E_{LPIPS}^{fit}$ . A large gap indicates that a flexible, time-varying representation explains the video significantly better than a static one, suggesting non-physical scene changes.
Deformation Energy ( $E_{deform}$ ): Quantifies the magnitude of temporal offsets in the DeformableGS fit, providing context for the static-vs-flexible gap.
Active Evidence Descriptors: Includes mean optical flow magnitude ( $\bar{F}_{orig}$ ) and smoothed cumulative camera turn ( $\Theta_s$ ) to validate the presence of acquisition evidence.

ControlBench

To interpret these diagnostics, the authors introduce ControlBench, a stress-test suite using real static acquisitions and real-derived transformations (frozen video, digital zoom, lateral warp, texture repaint, foreground deformation). This suite validates whether the metrics respond plausibly to known geometric and appearance changes.

Experimental Setup

Dataset: 3,840 generated clips created from 80 controlled static-scene prompts (from GeCo-Eval [9]) across 12 open-weight model configurations (including CogVideoX, HunyuanVideo, LTX-Video, MAGI-1, Open-Sora, SkyReels, Wan) and 4 generation seeds.
Format: All clips standardized to 49 frames, 256×256 resolution, 8 fps.
Comparison: Results are compared against GeCo [9], a prior work focusing on local geometric consistency.

Key Results

The evaluation reveals that single-scalar rankings are misleading and that different failure modes are often orthogonal:

Divergent Failure Modes: Models with low static rendering error (e.g., MAGI-1) often fail the active-evidence check or exhibit poor CEMR (mismatched temporal motion). Conversely, models with active motion (e.g., HunyuanVideo-1.5) may show high static rendering errors and high deformation energy.
Complementarity with GeCo: GeoT2V-Bench captures failure modes that GeCo misses. A clip can have low local geometric inconsistency (good GeCo score) but fail the global reconstruction audit (poor CEMR or high static-flex gap), indicating that local consistency does not guarantee a coherent global camera trajectory or static scene.
Metric Interdependence: Low per-frame LPIPS does not guarantee temporal coherence. CEMR is critical for distinguishing between clips that merely look static and those that genuinely support a static 3D explanation under camera motion.

Significance and Claims

The paper claims that GeoT2V-Bench shifts the evaluation paradigm from "video plausibility" to "reconstructable multi-view evidence." Its primary contributions are:

Formulation: Defining camera-prompted T2V evaluation as a test of whether a video can be explained as observations of a single static scene along an estimated camera path.
Staged Difficulty Characterization: Highlighting that both camera pose estimation and 3D reconstruction are fragile stages that can fail due to the very inconsistencies being measured.
Diagnostic Profile: Introducing a continuous profile (MedianGS error, CEMR, static-vs-flexible gap) that separates active evidence, static reconstruction behavior, and temporal-motion explanation, avoiding the conflation of different failure types.
Complementary Diagnostics: Demonstrating that reconstruction-based diagnostics reveal failure modes (global trajectory incoherence, non-physical scene changes) that local-geometry metrics like GeCo do not detect.

The authors emphasize that the benchmark is a diagnostic tool, not a visual quality ranker. Low residuals may indicate weak acquisition evidence rather than high quality, and high residuals may stem from backend limitations or complex motion rather than generation failure. The work supports the interpretation of these diagnostics through the ControlBench stress tests.