🤖 AI

VGI-Bench: Probing Visual Intelligence in Video Generation Models

This paper introduces VGI-Bench, a comprehensive benchmark comprising 27 tasks and 810 instances designed to rigorously evaluate the visual reasoning capabilities of video generation models, revealing that even state-of-the-art systems like Seedance 2.0 achieve only 51.0% accuracy and struggle with reliable self-correction during the generation process.

Original authors: Xuan He, Cong Wei, Yuhao Cheng, Linrui Ma, Yuxuan Zhang, Zuojun Li, Yuhao Wen, Jize Jiang, Zeyi Liu, Yuren Hao, Songcheng Cai, Keming Wu, Penghui Du, Kai Zou, Rui Yang, Chenkai Sun, Ke Yang, Ping Nie

Published 2026-08-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuan He, Cong Wei, Yuhao Cheng, Linrui Ma, Yuxuan Zhang, Zuojun Li, Yuhao Wen, Jize Jiang, Zeyi Liu, Yuren Hao, Songcheng Cai, Keming Wu, Penghui Du, Kai Zou, Rui Yang, Chenkai Sun, Ke Yang, Ping Nie, Kelsey R Allen, Chenglong Wang, Michel Galley, Jianfeng Gao, ChengXiang Zhai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer that doesn't just watch the world, but can imagine how it changes. For years, artificial intelligence has been trained to recognize static images: a cat sitting on a mat, a car parked on a street. More recently, these systems have learned to generate videos, creating moving pictures that look surprisingly real. Scientists now wonder if these video generators are doing something deeper than simply mimicking motion. They are asking if the machines are beginning to understand the rules of the physical world—gravity, cause and effect, and how objects interact—so that they can "think" through a sequence of events before showing the result. This question sits at the heart of a new effort to measure whether video-making AI can truly reason, or if it is merely guessing what a plausible ending looks like.

A team of researchers has built a new testing ground called VGI-Bench to answer this question. Instead of asking the AI to solve abstract puzzles drawn in simple lines, they fed it realistic photos of everyday scenes and asked it to generate videos that follow specific, logical rules. The tasks were designed to be challenging but possible, requiring the machine to simulate a process step-by-step. For example, an AI might be shown a photo of a toy car at the start of a maze and asked to generate a video of it driving to the exit without hitting the walls. Another task might show a flat, unfolded cardboard box and ask the AI to animate it folding itself into a 3D shape. The researchers did not just look at the final image; they scrutinized the entire video to see if the machine followed the rules at every moment, checking for things like objects passing through walls, cars teleporting, or items disappearing and reappearing.

When they tested the most advanced video generation models available today, the results were a mix of promise and limitation. The best-performing system, a model named Seedance 2.0, managed to solve only about half of the tasks correctly under these strict conditions. While the AI could often get the general idea right, it frequently failed to maintain a consistent story. In many videos, the physics broke down: a cup sitting on a tilted laptop would not roll off, or a car would drive straight through a solid wall. The machines also struggled with rules, sometimes skipping necessary steps or changing the identity of an object mid-way through the video. Even when the final result looked correct, the path to get there was often filled with impossible movements, suggesting the models are good at predicting a final state but poor at simulating the journey to get there.

The researchers dug deeper to understand why these failures happened. They found that the style of the input image mattered greatly; when the AI was given a realistic photo, it performed better than when given a simple line drawing, indicating that the models are trained on real-world images and struggle when the visual style changes. They also looked at how the AI "thinks" as it creates a video, step by step. They discovered that the models do not really correct their mistakes as they go. If the AI makes an error in the early stages of generating a video, it tends to stick with that error, refining the wrong idea rather than fixing it. The machine locks into a hypothesis early on and then polishes it, even if that hypothesis violates the laws of physics or the rules of the task.

This study suggests that while video generation models are becoming powerful tools for creating visual content, they have not yet developed the kind of reliable, step-by-step reasoning that humans use to navigate the world. They can simulate appearances and simple motions, but they lack a deep, consistent understanding of how things actually work over time. The new benchmark provides a clear way to measure this gap, showing that current systems are far from being true visual reasoners. By highlighting exactly where these models fail—whether in tracking objects, following rules, or correcting errors—the research offers a roadmap for building the next generation of artificial intelligence that can truly imagine the future.

Technical Summary: VGI-Bench: Probing Visual Intelligence in Video Generation Models

1. Problem Statement

Recent studies suggest that video generation models are evolving from passive visual simulators into potential "visual reasoners" capable of zero-shot reasoning through generated frame sequences. However, evaluating this capability remains challenging due to three critical gaps in existing benchmarks:

Distribution Mismatch: Many benchmarks rely on abstract or line-art inputs, which deviate from the natural-image priors of modern video models, leading to failures that reflect visual domain mismatch rather than reasoning limitations.
Lack of Process Sensitivity: Existing visual reasoning tasks often allow models to answer directly from the input or focus solely on the final state, failing to test whether the model can simulate valid intermediate trajectories and evolving processes.
Uncontrolled Difficulty: Current benchmarks include tasks that are either trivially solvable or far beyond the feasible regime of current models (e.g., requiring non-visual expertise or exceeding practical video durations), making diagnostic evaluation difficult.

2. Methodology: VGI-Bench

To address these gaps, the authors introduce VGI-Bench, a benchmark containing 27 tasks and 810 instances designed to evaluate visual intelligence at the current capability boundary of video generation models.

2.1 Taxonomy and Task Design

The benchmark employs a two-level taxonomy:

Task Domains (Mutually Exclusive):
- Visual Organization: Arranging, grouping, or selecting objects based on visual attributes.
- Physical Manipulation: Object-level actions (moving, stacking, tool use) requiring plausible physical interaction.
- Structured Puzzles: Rule-governed puzzles requiring explicit constraints to transform an initial state to a target.
- Spatiotemporal Dynamics: Reasoning about state evolution, ordering, and temporal dependencies.
Skill Tags (Non-exclusive): Tasks are annotated with tags such as Spatial, Temporal, Planning, Attribute Grounding, Physics, Topology, and Affordance.

2.2 Data Construction and Quality Control

Inputs: Tasks utilize photorealistic-style inputs (web images or generated via image models like GPT-Image-2) to align with model priors, avoiding abstract line-art.
Process-Sensitivity: Success depends on valid intermediate trajectories and rule-preserving evolution, not just the final state.
Difficulty Calibration: A pre-generation stage tests tasks on state-of-the-art (SOTA) models. Tasks are accepted only if they are solvable by at least one model but fail for at least one, ensuring they are challenging yet feasible.
Evaluation Metrics:
- Completeness (Comp.): A global metric assessing progress toward the goal (Complete, Partial, Failed).
- Rubric Score (Rub.): A local metric measuring process validity against a fine-grained checklist of rules and constraints. It uses an adaptive coarse-to-fine frame sampling strategy (4fps to 8fps) and a sliding focus window to detect transient violations.
- Final Score: The product of Completeness and Rubric Score, penalizing either goal failure or rule violation.

2.3 Evaluation Setup

The benchmark evaluates a wide range of commercial (e.g., Seedance 2.0, Sora 2, Veo 3.1) and open-source (e.g., Wan 2.7, HunyuanVideo 1.5) video models. An automatic VLM-based judge (Gemini-3-Flash) is used, with reliability validated against human annotations. A subset of tasks is also adapted for image generation models to compare static goal-state inference against procedural video generation.

3. Key Results

Overall Performance: Current models exhibit emerging reasoning abilities but remain far from reliable general-purpose visual intelligence. Even the strongest model, Seedance 2.0, achieves only 51.0% under the evaluation criteria.
Domain and Skill Analysis:
- Structured Puzzles is the most challenging domain, exposing failures in multi-step rule adherence and state tracking.
- Topology and Temporal skills are the weakest dimensions, indicating difficulties in connectivity preservation and multi-step state tracking.
- Commercial models consistently outperform open-source models, though all struggle with maintaining coherent multi-step execution.
Failure Modes: Common failures include:
- Physical Collapse: Unrealistic deformations or violation of gravity (e.g., objects not rolling down tilted surfaces).
- Rule Violation: Performing prohibited actions or skipping essential steps while reaching a visually plausible final state.
- Object/State Inconsistency: Failure to preserve object identities or positions over time (e.g., objects disappearing or duplicating).
Input Sensitivity: Performance is highly sensitive to input visual styles. Open-source models show significant drops when switching from photorealistic to line-art inputs, suggesting that abstract inputs in other benchmarks may conflate reasoning limitations with visual domain mismatch.
Synthetic Fine-Tuning Transfer: Fine-tuning on large-scale synthetic abstract data (VBVR) improves performance on structurally overlapping tasks but offers limited gains on non-overlapping tasks. Improvements are often bounded by the structural coverage of the training distribution.
Denoising Dynamics: Analysis of intermediate denoising steps reveals limited self-correction. Models tend to refine early hypotheses rather than correct reasoning errors; "self-correction" (wrong $\to$ correct) occurs in less than 1% of transitions, while models mostly lock in and refine early (often incorrect) hypotheses.

4. Significance and Contributions

The paper claims three primary contributions:

VGI-Bench: A diagnostic benchmark tailored to the current capability boundary of video models, featuring realistic inputs, process-sensitive tasks, and calibrated difficulty levels.
Broad Evaluation: A comprehensive assessment of contemporary generative models (both image and video), highlighting their emerging reasoning capabilities while exposing significant limitations in general-purpose visual intelligence.
Multi-faceted Analysis: A detailed diagnosis of video reasoning behavior, covering failure modes, input sensitivity, the transfer limits of synthetic fine-tuning, and the internal dynamics of denoising trajectories.

The authors posit that VGI-Bench will help stimulate the development of next-generation video generation models by providing a rigorous framework to distinguish between visual fidelity and genuine visual reasoning, identifying specific bottlenecks such as physical coherence, rule adherence, and self-correction mechanisms.