GenVidBench: A 6-Million Benchmark for AI-Generated Video Detection

This paper introduces GenVidBench, the largest dataset to date containing 6.78 million videos from 11 state-of-the-art generators, designed to overcome current limitations in developing and evaluating robust AI-generated video detection models through its large scale, cross-source diversity, and coverage of cutting-edge generation technologies.

The Gist

Imagine a world where anyone can snap their fingers and create a video of a dragon flying over New York City, or a politician giving a speech they never actually gave. These "AI videos" are getting so good that they look almost identical to real life. This is a problem because bad actors could use them to spread lies, ruin reputations, or cause panic.

To stop this, we need "video detectives"—AI programs trained to spot the fakes. But here's the catch: you can't train a detective if you only show them photos of the same criminal. If the detective only learns to catch a guy named "Bob," they won't recognize "Alice" when she shows up.

This is exactly the problem the paper GenVidBench solves.

The Problem: The "Bob" Detective

Before this paper, the datasets used to train video detectors were like a small, closed club. They had a few thousand videos, often made by the same few AI tools.

• The Flaw: If a detector was trained on videos made by "AI Tool A," it would get really good at spotting "AI Tool A." But if a criminal switched to "AI Tool B," the detector would be completely fooled.
• The Analogy: It's like teaching a security guard to recognize a thief only by his red hat. If the thief swaps the red hat for a blue one, the guard lets him right through.

The Solution: GenVidBench (The Ultimate Training Ground)

The researchers from Huawei Noah's Ark Lab built GenVidBench, a massive new training ground for these video detectives. Think of it as a "Super Gym" for AI models.

Here is what makes it special, using simple analogies:

1. The Size: A Library vs. A Booklet

Previous datasets were like a small booklet with 2,000 pages. GenVidBench is a 6.78-million-page library.

• Why it matters: With this much data, the AI detective sees every possible variation of a fake video. It stops guessing and starts knowing.

2. The "Cross-Source" Challenge: The Blind Test

This is the most important part. In the past, the training videos and the test videos were often made by the same tools.

• The Old Way: Training a student on math problems from "Textbook A" and then testing them on "Textbook A." They will pass easily, but they haven't really learned math.
• The GenVidBench Way: They train the AI on videos made by 11 different AI generators (like Pika, Sora, Kling, etc.) and then test it on videos made by completely different generators.
• The Analogy: It's like teaching a chef to cook a steak using a gas stove, then testing them on an electric stove, a campfire, and a microwave. If they can still cook a perfect steak, they are a master chef, not just someone who memorized one recipe.

3. The "Same Story, Different Actors" Trick

To make the test even harder, the researchers created pairs of videos.

• The Setup: They took one specific prompt (e.g., "A cat sitting on a blue chair") and asked 5 different AI tools to generate a video of it.
• The Result: You now have 5 videos that look exactly the same in terms of the story and the objects, but they were made by different "actors" (AI tools).
• The Challenge: The detector can't just look at what is in the video (a cat on a chair) to decide if it's fake. It has to look for the subtle, invisible "fingerprints" left by the specific AI tool that made it. This forces the AI to learn the truth of how fakes are made, rather than just memorizing the content.

4. The Semantic Map: Organizing the Chaos

The dataset isn't just a giant pile of random videos. The researchers organized it like a well-structured library with labels for:

• Objects: Is it a person, a car, or a plant?
• Actions: Is the person standing still or running?
• Locations: Is it a city street or a forest?
• Why it helps: This allows researchers to ask specific questions. "Can our detector spot fakes in forest scenes but fail in city scenes?" This helps them fix the weak spots.

The Results: It's Hard, But Necessary

When the researchers tested famous video-detecting AI models on this new "Super Gym," the results were humbling:

• The Good News: The models got better at spotting fakes than ever before.
• The Bad News: They still struggled. When the AI had to detect a video made by a tool it had never seen before, its accuracy dropped significantly.
• The Takeaway: This proves that current detectors are still too reliant on memorizing specific tools. They aren't "smart" enough yet to generalize.

Conclusion

GenVidBench is a massive, high-quality, and incredibly difficult training set designed to force AI video detectors to become true experts. By removing the "crutches" of similar training data and forcing the AI to learn across different tools and scenarios, this benchmark ensures that when we deploy these detectors in the real world, they won't be fooled by a simple change of hat.

It's the difference between a security guard who only knows one thief and a master detective who can spot a forgery no matter who created it.

Technical Summary

1. Problem Statement

The rapid advancement of generative video models (e.g., Sora, Kling, Pika) has blurred the line between real and AI-generated content, posing significant risks such as misinformation dissemination, reputational damage, and cybersecurity threats. While effective detectors are urgently needed, their development is currently hindered by a lack of large-scale, high-quality, and challenging datasets. Existing datasets suffer from:

• Small scale: Most contain only thousands of videos, insufficient for training robust deep learning models.
• Content bias: Training and test sets often share similar content or generators, allowing models to cheat by learning specific artifacts rather than generalizable features.
• Lack of metadata: Few datasets provide the original prompts, images, or semantic labels required for fine-grained analysis.

2. Methodology: GenVidBench Construction

The authors introduce GenVidBench, a massive dataset designed to overcome these limitations through a rigorous construction process:

• Scale: The dataset contains 6.78 million videos, making it the largest benchmark for AI-generated video detection to date. A lightweight subset, GenVidBench-143k, is also provided for rapid iteration.
• Data Sources:
  • Real Videos: Sourced from Vript and HD-VG-130M.
  • AI-Generated Videos: Sourced from 11 state-of-the-art (SOTA) generators, including Pika, VideoCraftV2, Modelscope, T2V-Zero, MuseV, SVD, Mora, CogVideo, Sora, and Kling.
• Cross-Source and Cross-Generator Design:
  • The dataset is structured into two distinct video pairs derived from the same prompts or images to ensure content parity.
  • Training Set: Contains videos from Pair 1 (Pika, VideoCraftV2, Modelscope, T2V-Zero).
  • Test Set: Contains videos from Pair 2 (MuseV, SVD, Mora, CogVideo, HD-VG-130M) and newer models like Sora/Kling.
  • Challenge: This setup forces models to detect generation artifacts without relying on specific generator signatures or content biases, simulating real-world scenarios where the generator is unknown.
• Semantic Annotation:
  • Unlike previous benchmarks, GenVidBench provides rich semantic labels for a subset of videos (143k) across three dimensions:
    1. Object Categories: (e.g., People, Animals, Buildings, Plants).
    2. Actions: (e.g., Static Postures, Active Engagement, Observing).
    3. Locations: (e.g., Natural Landscapes, Cities, Indoor).
  • These labels were extracted using Large Language Models (LLMs) to facilitate scenario-specific analysis.

3. Key Contributions

1. Unprecedented Scale: The first dataset to reach the 6-million video scale for AI-generated video detection, ensuring statistical robustness.
2. Rigorous Benchmarking Protocol: Introduces a Cross-Source and Cross-Generator evaluation protocol. By training on one set of generators and testing on completely different ones (with identical content prompts), it eliminates content-based shortcuts and tests true generalization capabilities.
3. Rich Metadata: Provides original text prompts, image inputs, and multi-dimensional semantic labels, enabling researchers to analyze detection performance based on specific content attributes (e.g., "How well does the detector work on 'Animals' in 'Natural Landscapes'?").
4. Comprehensive Baselines: Evaluates a wide range of SOTA video classification models (CNNs, Transformers, Mamba-based) on this benchmark to establish a solid foundation for future research.

4. Experimental Results

The authors evaluated multiple models (including VideoSwin, UniFormerV2, DeMamba, I3D, SlowFast) on the cross-source/cross-generator task:

• High Difficulty: The task proved extremely challenging. When training and testing on the same generator, accuracy exceeded 97%. However, under the cross-generator setting, performance dropped significantly.
  • Example: VideoSwin-Tiny dropped from ~99% (same source) to 54.66% when trained on Pika and tested on SVD.
• Model Performance:
  • DeMamba (Mamba architecture) achieved the highest Top-1 accuracy of 85.47%.
  • MViTv2 and VideoSwin followed with ~80%.
  • Transformer-based models generally outperformed CNN-based models.
• Generator Difficulty:
  • Videos from Sora were the hardest to detect (high realism), while CogVideo was easier due to lower temporal continuity.
  • SVD generated the most "hard cases" (videos where the detector assigned low probability to being fake).
• Comparison with Existing Datasets: Models trained on GenVidBench showed significantly lower accuracy compared to their performance on smaller, less diverse datasets (e.g., GVF, GenVideo), confirming GenVidBench's superior difficulty and realism.
• Scenario-Specific Analysis: Performance varied drastically by semantic category. For instance, in the "Plants" category, TimeSformer achieved 75.09% accuracy, while VideoSwin dropped to 52.86%, highlighting the need for scenario-specific detection strategies.

5. Significance

• Advancing Detection Robustness: GenVidBench shifts the focus from detecting specific generator artifacts to identifying fundamental inconsistencies in AI-generated video, which is crucial for real-world deployment where the generator is unknown.
• Standardization: It sets a new standard for dataset construction in the field, emphasizing the need for large-scale, cross-generator, and semantically annotated data.
• Research Catalyst: By providing the 143k subset with detailed semantic labels, the paper enables researchers to dissect why models fail (e.g., specific object types or motion patterns), guiding the development of more generalized and effective detection algorithms.
• Open Resource: The dataset and code are publicly available, fostering collaborative progress in AI safety and media forensics.

In conclusion, GenVidBench represents a critical step forward in the fight against AI-generated misinformation, offering a challenging, large-scale, and analytically rich environment to train and evaluate the next generation of video detection models.