💻 computer science

A Survey of AI-Generated Video Evaluation

This survey introduces the emerging field of AI-Generated Video Evaluation (AIGVE) by systematically analyzing existing methodologies, identifying current gaps, and advocating for robust, multi-faceted evaluation frameworks that address the unique spatial and temporal complexities of AI-generated video content.

Original authors: Xiao Liu, Xinhao Xiang, Zizhong Li, Yongheng Wang, Zhuoheng Li, Zhuosheng Liu, Weidi Zhang, Weiqi Ye, Jiawei Zhang

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiao Liu, Xinhao Xiang, Zizhong Li, Yongheng Wang, Zhuoheng Li, Zhuosheng Liu, Weidi Zhang, Weiqi Ye, Jiawei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you just bought a brand-new, super-smart robot chef. You give it a recipe: "Make me a spicy pasta dish with fresh basil." The robot whirs, clanks, and in seconds, a steaming plate of pasta appears.

But here's the problem: How do you know if the robot actually did a good job?

Did it actually use basil, or did it just guess?
Is the pasta cooked, or is it raw?
Does the sauce look like real sauce, or like green paint?
And most importantly, does it taste good to you?

This paper is essentially a guidebook for hiring a "Quality Inspector" for AI video robots.

The Big Problem: Videos are Harder than Photos

In the past, we had robots that could make pictures (like Midjourney) or write text (like ChatGPT). We already have good ways to check if those are good.

Text: We check if the grammar is right.
Photos: We check if the colors look real.

But Video is different. A video isn't just a picture; it's a movie. It has two extra layers of complexity:

Time (The Movie Part): Things have to move smoothly. If a cat jumps, it shouldn't freeze in mid-air for three seconds.
Physics (The Real World Part): If you drop a glass, it should shatter. If an AI makes a glass float up into the sky, that's a "physics error."

The authors of this paper say: "We have all these amazing AI video generators (like Sora), but we don't have a good ruler to measure how good they are yet."

The Two Main Jobs of the Inspector

The paper suggests that a good video inspector needs to check two main things:

1. Does it look and feel real? (Alignment with Human Perception)

Imagine you are watching a movie. Even if you don't know the plot, you can tell if the movie is bad.

Blurry? Bad.
Jumpy? Bad.
Does the character's face morph into a monster halfway through? Bad.
Did the car drive through a wall? Bad.

This part of the paper looks at all the old ways we used to check video quality (like checking for static or noise) and mixes them with new AI tools to catch these "weirdness" factors.

2. Did it listen to you? (Alignment with Human Instructions)

This is the "Did you follow the recipe?" test.

Your Prompt: "A dog riding a skateboard in the rain."
Bad AI: Shows a dog on a skateboard, but it's sunny. (It missed the "rain" part).
Worse AI: Shows a cat riding a bike. (It missed the "dog" part).
Best AI: Shows a dog, on a skateboard, in the rain.

The paper reviews how we can use smart AI (like Large Language Models) to act as a strict teacher, reading your prompt and checking if the video actually did what you asked.

The "Toolbox" They Reviewed

The authors went through a massive library of existing tools and organized them into a "Toolbox" for the future:

The Rulers (Metrics): Old-school math formulas that measure blur or pixel count. Good for simple things, but they can't tell if a video is "funny" or "creepy."
The Judges (Human Evaluation): Real people watching videos and giving them stars. This is the most accurate, but it's slow and expensive.
The AI Critics (Model-Based): New AI models trained to watch videos and give scores. Some of these are like "Taste Testers" (they just give a score), while others are like "Food Critics" (they write a paragraph explaining why the pasta was salty).

The "Gotchas" (Common Mistakes)

The paper lists the six most common ways AI video fails, which they call "Error Types":

Technical Glitches: The video is too grainy or low-resolution.
The "Statue" Problem: The video is supposed to be moving, but everything is frozen.
Physics Violations: A ball rolls uphill, or water flows upward.
Identity Crisis: A person's face changes into a different person's face halfway through the video.
Detail Failures: The text on a sign is gibberish, or a car has three wheels.
Ignoring Instructions: The user asked for a beach, and the AI gave a forest.

What's Next? (The Future)

The paper concludes that we need to build better "Inspectors." Here is what they think we should do next:

Use Smarter AI Critics: Instead of just giving a number (like "8/10"), the AI should explain why it gave that score, just like a human teacher.
Check for Safety: Make sure the AI isn't accidentally making videos that are dangerous, violent, or misleading.
One Big Standard: Right now, everyone uses different tests. We need one universal "Driver's License Test" for AI videos so we can compare them fairly.

The Bottom Line

This paper is a map. It tells us that while AI video generation is getting amazing, we are currently flying blind without a good way to measure quality. The authors are calling on scientists and engineers to build better "Quality Inspectors" so that when you ask an AI to make a video, you can be sure it will look real, move smoothly, and actually do what you asked.

1. Problem Statement

The rapid advancement of generative AI models (e.g., Sora, Meta Movie Gen) has led to an explosion in AI-generated video (AIGV) content. However, evaluating this content presents unique challenges distinct from text or static image evaluation:

Complexity: Videos involve both spatial complexity (visual fidelity, object structure) and temporal dynamics (motion coherence, physics, causality).
Evaluation Gap: Traditional Video Quality Assessment (VQA) focuses on technical metrics (compression, transmission artifacts) or perceptual quality of user-generated content. It often fails to assess instruction alignment (whether the video follows the prompt) or physical plausibility (real-world laws).
Fragmentation: Existing research is scattered across different domains (VQA, multimodal alignment, safety), lacking a unified framework. Current methods often rely on coarse-grained scores or disjointed metrics, making it difficult to diagnose specific failure modes.

2. Methodology and Framework

The authors propose a unified framework for AI-Generated Video Evaluation (AIGVE), categorizing the field into two primary dimensions: Alignment with Human Perception and Alignment with Human Instructions.

A. Taxonomy of Errors

The survey first identifies six common failure modes in AIGV to guide evaluation:

Technical Errors: Low resolution, blur, compression artifacts.
Dynamic Errors: Lack of meaningful motion or static frames.
Physical Errors: Violations of physics (e.g., gravity, collisions).
Consistency Errors: Inconsistent object identity or appearance over time.
Quality Errors: Distorted geometry or missing details.
Alignment Errors: Deviation from the textual prompt/instruction.

B. Benchmark Datasets

The survey categorizes datasets based on their evaluation focus:

General Large-scale: Measure holistic quality (e.g., T2VQA-DB, AIGC-VQA).
Multi-dimensional: Decompose quality into specific aspects like visual fidelity, motion, and text alignment (e.g., VBench, EvalCrafter).
Temporal/Motion-centric: Focus on motion dynamics and long-range consistency (e.g., T2VBench, DEVIL).
Real-World Oriented: Assess physical plausibility, safety, and grounded knowledge (e.g., VIDEOPHY, T2VSafetyBench, ADGV-Bench).
Data Evolution: The field is moving from simple video-opinion pairs ( $D=\{V, S\}$ ) to video-instruction-opinion triplets ( $D=\{V, I, S\}$ ) to better capture instruction adherence.

C. Evaluation Methods

The paper classifies evaluation approaches into two main paradigms:

1. Metric Collection Evaluation:

Uses existing, established metrics to assess specific dimensions.
Examples: VBench (16 hierarchical dimensions), EvalCrafter (visual quality, text-video alignment, motion, temporal consistency).
Limitation: Computationally expensive to run multiple metrics; lacks a unified score.

2. Modeling-Based Evaluation:

Trains models to predict human judgment directly from data.
Holistic Models: Predict a single overall quality score (e.g., T2VQA, AIGC-VQA).
Multi-aspect & Comment-aware: Predict structured scores and natural language explanations for specific dimensions (e.g., VideoScore, AIGVE-MACS).
Temporal Models: Focus specifically on motion and dynamics (e.g., DEVIL, FVMD).
Real-world/Safety Models: Evaluate adherence to safety constraints and physical laws (e.g., T2VSafetyBench, GAP).

3. Foundation Models in Evaluation:

Multimodal Models (CLIP/BLIP): Used for embedding similarity between text and video frames.
Large Multimodal Models (LMMs/LLMs): Emerging as powerful evaluators (e.g., GPT-4V, Q-Align) that can reason about consistency, generate diagnostic comments, and handle complex instructions via instruction tuning.

3. Key Contributions

Definition of a New Field: The authors formally propose AIGVE as a distinct research area, distinguishing it from traditional VQA and text/image evaluation.
Comprehensive Taxonomy: They provide a structured overview of the landscape, categorizing datasets, error types, and evaluation methods (Metric-based vs. Modeling-based, Holistic vs. Multi-aspect).
Identification of Gaps: The survey highlights critical limitations in current research, such as the lack of unified frameworks, poor interpretability of scores, and insufficient handling of safety/ethical constraints.
Future Roadmap: The paper outlines specific directions for future research, including the integration of Vision-Language Models (VLMs), enhancing score interpretability, and embedding ethical/safety considerations.

4. Results and Findings

Correlation with Human Judgment: Modeling-based approaches (especially those using LMMs) show higher correlation with human subjective ratings compared to traditional metric collection, particularly in assessing semantic alignment and complex temporal dynamics.
Interpretability: Methods that provide aspect-wise scores and natural language comments (e.g., VideoScore2, AIGVE-MACS) are superior for debugging and guiding model improvement compared to single scalar scores.
Temporal Challenges: Current methods struggle significantly with long-range temporal consistency and physical plausibility. Specialized benchmarks (like VIDEOPHY) are necessary to drive progress in these areas.
Data Trends: There is a clear shift toward larger, multi-modal datasets that include detailed instructions and human feedback, moving away from simple quality ratings.

5. Significance

Foundational Resource: This survey serves as a foundational guide for researchers and practitioners, consolidating scattered research into a coherent framework.
Driving Robustness: By advocating for multi-dimensional and instruction-aligned evaluation, the paper pushes the field toward more reliable and controllable video generation models.
Ethical and Safety Implications: It emphasizes the critical need for evaluation frameworks that can detect harmful content, bias, and safety violations, which is essential for the real-world deployment of AIGV technologies.
Future Direction: The call to integrate VLMs and improve interpretability sets the agenda for the next generation of evaluation tools, moving from "black box" scoring to explainable, diagnostic assessment.

In conclusion, this survey argues that the future of AI-generated video depends not just on better generation models, but on robust, nuanced, and ethical evaluation frameworks that can handle the spatial, temporal, and semantic complexities of video content.