Technical Summary: Artifact-Bench

Problem Statement

Despite significant advancements in video generative models, AI-generated videos (AIGC) still exhibit perceptible artifacts, including temporal inconsistencies, structural distortions, and semantic incoherence. While Multimodal Large Language Models (MLLMs) have demonstrated strong visual understanding capabilities, their ability to perceive and reason about these specific generative artifacts remains unclear. Existing benchmarks often lack systematic evaluation of artifact-aware perception and fine-grained diagnostic reasoning, particularly across diverse domains beyond photorealistic content (e.g., animated or CG-style videos). Furthermore, current evaluations frequently rely on isolated tasks or limited scenarios, making it difficult to determine if MLLMs rely on genuine artifact detection or superficial semantic priors and dataset biases.

Methodology

1. Hierarchical Taxonomy of Realism Artifacts

The authors establish a three-level hierarchical taxonomy to organize AIGC video artifacts, moving from coarse visual abnormalities to fine-grained structural and temporal inconsistencies:

• Top-Level Domains:
  • Surface Artifacts: Low-level visual defects identifiable via local appearance cues (e.g., color, exposure, texture).
  • Structural Defects: Failures requiring understanding of object/scene organization (e.g., identity, morphology, spatial depth).
  • Temporal-Semantic Violations: High-level failures requiring cross-frame integration and commonsense/causal reasoning (e.g., motion, causality, scene continuity).
• Mid-Level Failure Families: Decomposes top-level domains into specific failure sources (e.g., "Camera & Lens," "Functional Structure," "Violation of Causality").
• Fine-Grained Labels: A set of 30 concrete, visually observable failure modes (e.g., "Texture Inconsistency," "Irreversibility Violation," "Action-Context Mismatch").

2. Benchmark Design (Artifact-Bench)

Artifact-Bench introduces three complementary tasks to evaluate MLLMs progressively from coarse recognition to diagnostic reasoning:

• Task 1: Real vs. AI-Generated Video Classification (RVAC): A binary classification task where models determine if a video is real or synthetic. Real videos are paired with semantically aligned AI-generated counterparts to isolate realism cues from semantic differences.
• Task 2: Pairwise Video Realism Comparison (PVRC): Models compare two AI-generated videos with similar semantics but varying realism levels, selecting the more realistic one. This evaluates relative realism judgment.
• Task 3: Artifact Identification (AID): A fine-grained diagnostic task where models must identify specific artifacts causing unrealism from a set of 6 candidates (derived from the 30 fine-grained types). This requires multi-label selection and explicit reasoning about the cause of unreality.

3. Data Construction and Stratification

• Hybrid Data Pipeline: The dataset combines real-world videos, controlled generation using multiple state-of-the-art models (e.g., Kling, Veo, HunyuanVideo), and targeted artifact synthesis via intentional failure prompts.
• Difficulty Stratification: Samples are categorized into three difficulty levels (L1–L3) based on visual realism and artifact subtlety. L1 contains obvious artifacts, while L3 features highly realistic videos with subtle inconsistencies.
• Annotation Protocol: A rigorous human annotation process involving multiple rounds of review by experienced annotators and expert verification ensures high-quality labels and consistency.

Key Contributions

1. Systematic Taxonomy: The paper provides the first comprehensive, three-level hierarchical taxonomy of AIGC video artifacts, covering 30 fine-grained types across photorealistic, animated, and CG-style domains.
2. Comprehensive Benchmark: Artifact-Bench is introduced as a multi-granularity evaluation framework featuring three progressive tasks (RVAC, PVRC, AID) and a difficulty-stratified dataset of 1,350 videos (1,100 annotated samples).
3. Empirical Findings: Extensive experiments on 19 leading MLLMs (including proprietary, open-source general-purpose, and specialized models) reveal fundamental limitations in current models' ability to perceive and reason about generative artifacts.

Experimental Results

• Performance Limitations: Even the best-performing proprietary model (Gemini 3.1 Pro) achieved an overall score of 47.5% on Artifact-Bench. Many models performed at or near random guessing levels (50% for binary tasks), particularly in the AID task where average accuracy dropped below 10% for most models.
• Task Difficulty: Performance degraded significantly as task complexity increased. While models showed some capability in coarse classification (RVAC), they struggled severely with pairwise comparison (PVRC) and fine-grained diagnosis (AID).
• Model Scaling and Reasoning: Increasing model size (e.g., from 8B to 38B parameters) or enabling "thinking" (reasoning-enhanced) variants did not consistently improve performance. In some cases, larger or reasoning-enabled models underperformed their instruction-tuned counterparts.
• Human Misalignment: A significant gap exists between MLLM judgments and human perceptual preferences. Unlike humans, whose performance consistently declines with increasing difficulty (L1 to L3), MLLMs often showed irregular fluctuations or failed to degrade, suggesting reliance on superficial statistical cues rather than genuine artifact perception.

Significance and Claims

The paper claims that Artifact-Bench exposes a critical gap in current MLLM capabilities: while they excel at general semantic understanding, they lack the fine-grained perceptual sensitivity and temporal-spatial reasoning required to detect and diagnose generative artifacts.

The authors emphasize that current MLLMs are not yet reliable as general evaluators or reward providers for optimizing video generative models, as their judgments often misalign with human preferences and fail to follow the difficulty hierarchy. The findings highlight an urgent need for future MLLMs to develop stronger human-aligned realism understanding, fine-grained perceptual reasoning, and robust temporal consistency modeling to effectively bridge the gap between evaluation and generation.