VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation

1. Problem Statement

The rapid advancement of AI-generated content (AIGC) video models has outpaced the development of comprehensive evaluation frameworks. Existing benchmarks (e.g., V-Bench, V-Bench2.0) suffer from three primary limitations:

• Over-simplification of Aesthetics: They often reduce "video aesthetics" to a single holistic score, lacking fine-grained analysis of specific elements like composition, lighting, and color harmony.
• Reliance on External Models: Many metrics depend on pre-trained image/video models (e.g., MUSIQ, DINO) for indirect inference, which introduces bias and fails to align with genuine human perceptual judgment.
• Limited Scope: Existing benchmarks focus heavily on technical fidelity (temporal consistency, prompt alignment) while neglecting the artistic and perceptual qualities that define high-quality video generation.

There is a critical need for a unified benchmark that jointly evaluates generation quality (technical/semantic correctness) and aesthetic quality (artistic appeal) with fine-grained, interpretable dimensions.

2. Methodology

The authors propose VGA-Bench, a unified benchmark and multi-model framework built on a principled three-tier taxonomy.

A. Evaluation Dimension Suite (The Taxonomy)

The framework decomposes video evaluation into three core dimensions, further broken down into 52 sub-dimensions:

1. Aesthetic Quality (10 Dimensions): Adapted from the VADB dataset, these measure perceptual appeal.
   • Dimensions: Overall Score, Composition, Shot Size, Lighting, Visual Tone, Color, Depth of Field, Expression, Costume, and Makeup.
   • Focus: Evaluates how well the model captures artistic principles (e.g., rule of thirds, light source positioning) rather than just visual clarity.
2. Aesthetic Tagging (11 Dimensions): Structured annotations of quantifiable visual features.
   • Dimensions: Composition Types, Number/Position/Quality/Color of Light Sources, Shot Type, Depth of Field, Saturation, Brightness, Color Temperature, and Contrast.
   • Goal: To move from subjective "is it beautiful?" to objective "what specific aesthetic elements are present?"
3. Generation Quality (31 Dimensions): Refines the V-Bench framework into three categories:
   • Video-Text Consistency (11): Alignment between prompt and content (characters, actions, scenes, objects, style, etc.).
   • Realism & Plausibility (14): Adherence to physical laws (rigid body collisions, fluid motion, weather, lighting).
   • Basic Quality (6): Technical stability (clarity, noise, static content distortion, abnormal lighting).

B. Data Construction

• Prompt Suite: Designed 1,016 diverse prompts explicitly targeting specific dimensions (e.g., prompts explicitly mentioning "rule of thirds" for composition assessment).
• Dataset: Generated 60,000+ videos using 12 state-of-the-art video generation models (including Sora2, HunyuanVideo, Mochi, Latte, etc.) to ensure broad coverage of styles and artifacts.
• Human Annotation: Employed an "expert-led + crowd-assisted" paradigm. Domain experts defined guidelines and performed exemplary annotations, followed by crowdsourced labeling and batch auditing to ensure high-quality ground truth.

C. Multi-Model Evaluation Framework

To enable scalable, automated evaluation, the authors trained three dedicated neural assessors:

1. VAQA-Net (Video Aesthetic Quality Assessment Network): Predicts aesthetic scores (0–10 scale) for the 10 aesthetic dimensions. Initialized with encoders pre-trained on real-world video data (VADB) to inherit professional cinematographic understanding.
2. VTag-Net (Video Tagging Network): Performs multi-label classification for the 11 aesthetic tags.
3. VGQA-Net (Video Generation Quality Assessment Network): Evaluates the 31 generation quality dimensions. Unlike the other two, it is trained specifically on generated video data to better understand AIGC artifacts.

3. Key Contributions

1. Systematic Three-Dimensional Framework: Introduced a granular taxonomy covering 52 sub-dimensions across aesthetics, tagging, and generation quality, moving beyond single-score metrics.
2. Large-Scale Benchmark Dataset: Created the largest integrated testing platform to date, featuring 1,016 prompts, 60,000+ generated videos, and high-quality human annotations.
3. Specialized Automated Evaluators: Developed three end-to-end neural networks (VAQA-Net, VTag-Net, VGQA-Net) that eliminate reliance on external scoring models, achieving high alignment with human judgments.
4. Comprehensive Empirical Analysis: Evaluated 12 cutting-edge models, revealing distinct strengths and weaknesses across different dimensions (e.g., some models excel in realism but fail in color harmony).
5. Open Source Commitment: The authors commit to releasing the full benchmark suite, prompt templates, annotation data, and public API interfaces for all evaluation models.

4. Results

• Model Performance:
  • Sora2 achieved the highest overall aesthetic score (0.50) and generation level, demonstrating superior control over lighting and color.
  • Wan2.1 and HunyuanVideo also showed strong performance in aesthetic quality and generation consistency.
  • Older models like SVD and LTXVideo struggled significantly with aesthetic dimensions (e.g., lighting and composition scores < 0.25).
• Evaluator Accuracy:
  • VAQA-Net achieved ~70–77% accuracy in 5-class aesthetic scoring across dimensions.
  • VTag-Net showed high accuracy in predicting specific tags (e.g., 90% for Contrast, 89% for Saturation), though composition types were harder to predict (~45%).
  • VGQA-Net demonstrated robust cross-model generalization, with accuracy ranging from 70% to 92% across the 31 generation quality sub-dimensions.
• Human Alignment: A user study involving 40 participants showed that the model-based rankings correlated well with human preferences, achieving Recall@5 of 0.80 for aesthetic quality and 0.83 for generation quality.

5. Significance

VGA-Bench represents a paradigm shift in AIGC video evaluation, moving the field's focus from "how real" (technical fidelity) to "how beautiful" (aesthetic intelligence).

• Infrastructure for Advancement: It provides the necessary infrastructure to train models not just to be technically correct, but to possess "aesthetic intelligence" and artistic controllability.
• Practical Applications: The framework is applicable for content moderation, model debugging, and optimizing generative systems for professional use cases in film, advertising, and digital art.
• Standardization: By offering a unified, fine-grained, and open benchmark, it sets a new standard for the research community to compare and improve future video generation models.