💻 computer science

VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality

This paper introduces VGA-BenchV2, an expanded human-aligned benchmark and multi-model framework that significantly scales up annotations and evaluation dimensions to jointly assess video generation quality and aesthetics, while also establishing an evaluation-to-optimization pipeline to fine-tune generators for improved aesthetic alignment.

Original authors: Longteng Jiang, DanDan Zheng, Qianqian Qiao, Heng Huang, Huaye Wang, Yihang Bo, Bao Peng, Jingdong Chen, Jun Zhou, Xin Jin

Published 2026-08-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Longteng Jiang, DanDan Zheng, Qianqian Qiao, Heng Huang, Huaye Wang, Yihang Bo, Bao Peng, Jingdong Chen, Jun Zhou, Xin Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the last few years, computers have learned to create moving pictures from simple written descriptions. This technology, known as text-to-video generation, has moved from experimental sketches to tools capable of producing coherent, stable, and visually striking scenes. Artists, filmmakers, and advertisers are beginning to use these systems to imagine new worlds, but a fundamental question remains: how do we know if the result is actually good? For a long time, scientists measured these videos by checking technical details, such as whether the frames flowed smoothly or if the computer followed the instructions literally. However, a video can be technically perfect and still feel flat, lifeless, or artistically dull. The missing piece has been a way to measure the human experience of watching the video—its beauty, its mood, and its emotional resonance.

A team of researchers from Ant Group, the Beijing Film Academy, and the Beijing Institute for General Artificial Intelligence has tackled this challenge with a new framework called VGA-BenchV2. Their work moves beyond simple technical checks to create a system that understands and evaluates video aesthetics the way a human does. They built a massive library of over 60,000 videos generated by 12 different computer models, using 1,016 carefully written prompts to test specific visual qualities. To teach their computers how to judge these videos, the team enlisted human experts to provide over 36,000 new ratings and labels. This human feedback was used to train a hybrid team of artificial intelligence evaluators: one specialized in giving a continuous score for beauty, and two others that act like visual critics, identifying specific artistic tags and checking if the video makes sense. The result is a system that not only ranks how good a video looks but also uses that judgment to help the computer models learn and improve themselves.

The core of this new framework is a shift from passive measurement to active guidance. In the past, benchmarks were like report cards that simply told a student they passed or failed. VGA-BenchV2 acts more like a tutor that explains exactly what needs improvement and then helps the student practice. The researchers organized their evaluation into two main categories: aesthetics and generation quality. The aesthetic side looks at the artistic elements, such as lighting, color harmony, composition, and the overall mood of the scene. The generation side checks if the video is physically plausible, if the characters move naturally, and if the story matches the written prompt. By breaking these broad concepts down into 52 specific sub-categories, the team ensured that the evaluation was detailed and precise, covering everything from the type of camera shot to the realism of fluid motion.

To build this system, the team started with a large set of prompts, which are the written instructions given to the video generators. These 1,016 prompts were designed to trigger specific visual effects, ensuring that the resulting videos could be judged fairly against clear criteria. They then collected the output from 12 leading video generation models, creating a pool of more than 60,000 videos. The crucial step was the human annotation phase. The researchers gathered a massive amount of human feedback, adding 36,000 new task-level annotations to their existing data. This included over 16,000 ratings for aesthetic quality, where humans scored videos on a scale of zero to ten, and over 13,000 labels for aesthetic tagging, where humans identified specific visual attributes like the color of the light or the type of shot. They also added thousands of notes on generation quality to ensure the videos were realistic and consistent.

With this vast amount of human-labeled data, the team trained three specialized artificial intelligence evaluators. The first, called VAQA-Net, learned to predict a continuous score for how beautiful a video is, effectively mimicking the human eye for composition and tone. The other two evaluators, VTag-Net and VGQA-Net, are based on large vision-language models, which are advanced systems capable of understanding both images and text. These models were trained to act as critics: one identifies specific artistic tags, such as whether a scene uses soft lighting or high contrast, while the other answers detailed questions about the video's realism and consistency. This combination allows the system to provide both a numerical score and a detailed, interpretable explanation of why a video is good or bad.

The true power of VGA-BenchV2 lies in its ability to close the loop between evaluation and improvement. Once the evaluators were trained to match human preferences, the researchers used them as a reward system for reinforcement learning. In this process, the computer models that generate the videos are shown their own creations and given a score based on the aesthetic quality predicted by the evaluator. If the score is high, the model is encouraged to make similar videos; if the score is low, it is nudged to change its approach. In a specific test, the researchers applied this method to fine-tune a video generator called Wan2.1. The results showed that the model's average aesthetic score increased, and the resulting videos displayed more appealing visual compositions and higher overall quality. This demonstrates that the framework does not just measure performance; it actively drives the technology toward better artistic outcomes.

The study also provided a comprehensive ranking of the current state of the art in video generation. By applying their new evaluators to the 12 different models, the team was able to compare them across all 52 dimensions. The results revealed significant differences in how well each model handled artistic control versus technical consistency. Some models excelled at following complex prompts, while others produced more visually pleasing results. The data showed that while many models have improved in technical fidelity, there is still a wide gap in their ability to consistently produce videos that align with human aesthetic preferences. This highlights the importance of the new framework, which provides a standardized way to measure these subtle but critical qualities.

Ultimately, VGA-BenchV2 represents a significant step forward in making artificial intelligence more attuned to human sensibilities. By combining a massive dataset of human judgments with a sophisticated hybrid evaluation system, the researchers have created a tool that can assess video generation with a depth and nuance that was previously impossible. The framework bridges the gap between technical capability and artistic expression, offering a path for future models to learn not just how to generate pixels, but how to create art that resonates with people. As video generation continues to evolve, tools like this will be essential for ensuring that the machines we build can create content that is not only real but also beautiful.

Technical Summary: VGA-BenchV2

Problem Statement

The rapid advancement of AI-generated content (AIGC) video models has outpaced the development of evaluation frameworks capable of assessing human-centered aesthetic qualities. Existing benchmarks primarily focus on technical fidelity, utilizing metrics like FVD (Fréchet Video Distance) and CLIP Score to measure distributional similarity, temporal consistency, and low-level visual artifacts. These metrics fail to capture fine-grained perceptual factors essential for real-world applications in digital art and film, such as composition, lighting, color harmony, and artistic style. Furthermore, conventional evaluation tools are often passive; they can rank models but lack the capacity to provide human-aligned supervision signals necessary to actively improve generative models through optimization. While previous efforts like V-Bench and the original VGA-Bench introduced multi-dimensional taxonomies, they were limited by coarse granularity, reliance on off-the-shelf scoring models, and insufficient scale of human-labeled supervision to train robust evaluators.

Methodology

VGA-BenchV2 addresses these limitations by extending the original VGA-Bench into a unified framework that integrates large-scale annotation, a hybrid automated evaluator architecture, and a reinforcement learning (RL) based optimization pipeline.

1. Inherited and Expanded Taxonomy

The framework retains the fine-grained taxonomy of VGA-Bench, organized into two primary dimensions with 52 sub-dimensions:

Aesthetic Dimension:
- Aesthetic Quality: Continuous scoring (0–10) across 10 dimensions (e.g., Overall Score, Composition, Lighting, Color, Depth of Field).
- Aesthetic Tagging: Discrete classification labels across 11 dimensions (e.g., Composition Types, Shot Type, Light Source Position, Color Temperature).
Generation Dimension:
- Video-Text Consistency: 10 sub-dimensions ensuring semantic alignment between prompts and video content (e.g., Character, Action, Object, Camera Movement).
- Reality & Plausibility: 16 sub-dimensions evaluating physical realism (e.g., Fluid Motion, Rigid Body Collision, Scene Realism).
- Basic Quality: 5 sub-dimensions covering noise, clarity, and stability.

2. Data Construction

Prompt Suite: The framework utilizes 1,016 diverse prompts designed to explicitly trigger specific dimensions, ensuring that evaluations are semantically grounded.
Video Pool: Over 60,000 videos were generated by 12 mainstream text-to-video models (including SVD, AnimateDiff, HunyuanVideo, and Sora2) using the prompt suite.
Human Annotation Expansion: A critical component of VGA-BenchV2 is the substantial expansion of human-labeled data. The authors collected 36,000 new task-level annotations, resulting in a total of 50,600 annotations. This represents a 13.46× increase in aesthetic quality annotations, an 11.15× increase in aesthetic tagging, and a 1.55× increase in generation quality annotations compared to the original VGA-Bench.

3. Hybrid Evaluator Architecture

Leveraging the expanded dataset, the authors developed a hybrid evaluation system:

VAQA-Net: A specialized network for continuous aesthetic scoring. It employs a two-stage transfer learning strategy, initializing with a VADB pre-trained video encoder and fine-tuning on the large-scale generated video dataset.
VTag-Net & VGQA-Net: Two evaluators based on the Qwen3-VL-32B Large Vision-Language Model (LVLM).
- VTag-Net: Trained via instruction tuning for aesthetic tagging (classification).
- VGQA-Net: Trained via instruction tuning for generation quality assessment (question-answering/reasoning).

4. Evaluation-to-Optimization Pipeline

The framework introduces a mechanism to convert evaluators into reward models for Reinforcement Learning (RL). The aesthetic score predicted by VAQA-Net serves as a scalar reward signal ( $r_\phi(v)$ ). This signal is integrated into a Flow-GRPO (Group Relative Policy Optimization) pipeline to fine-tune video generators. The process includes KL regularization to prevent policy drift, allowing models to optimize toward higher aesthetic quality while maintaining stability.

Key Contributions

Large-Scale Human Annotation Infrastructure: VGA-BenchV2 transforms the benchmark from a passive evaluation suite into a training infrastructure by adding 36,000 new task-level annotations, significantly increasing the scale of human supervision for aesthetic and generation tasks.
Human-Supervised Hybrid Evaluator: The paper introduces a system combining specialized regression networks (VAQA-Net) with LVLM-based semantic reasoning (VTag-Net, VGQA-Net). This architecture achieves high alignment with human judgments while offering interpretability and scalability.
Evaluation-to-Optimization Framework: Unlike previous benchmarks that stop at ranking, VGA-BenchV2 closes the loop by using the learned evaluators as reward models. This enables the direct fine-tuning of video generators via RL to align outputs with human aesthetic preferences.
Unified Benchmark and Infrastructure: The work unifies 52 evaluation dimensions, 1,016 prompts, 60,000+ videos, and optimization pipelines into a single framework, facilitating systematic analysis and fair cross-model comparison.

Experimental Results

Evaluator Validation:
- VAQA-Net: Achieved a Spearman Rank Order Correlation Coefficient (SROCC) of 87.6% on the overall aesthetic score, demonstrating strong ranking consistency with human experts.
- VTag-Net: Showed robust classification accuracy, particularly in objective dimensions like Light Color (93.6%) and Number of Light Sources (83.7%).
- VGQA-Net: Achieved an average accuracy of approximately 71.3% across 31 generation quality sub-dimensions, with high scores in challenging categories like Scene Realism (89.2%).
Model Evaluation: The framework evaluated 12 state-of-the-art models. Sora2 achieved the highest scores across Aesthetic Score (0.50), Tag Classification (0.64), and Generation Level (0.80), followed by HunyuanVideo and Wan2.2.
Optimization Efficacy: In an RL fine-tuning experiment on Wan2.1, using VAQA-Net as a reward model increased the average aesthetic score from 0.49 to 0.52 on a held-out test set. Qualitative analysis confirmed improvements in visual composition and overall aesthetic appeal.

Significance and Claims

The authors position VGA-BenchV2 not merely as a rigorous evaluation platform but as a key infrastructure for advancing the next generation of video generation systems. The paper claims that by unifying large-scale annotation, hybrid automated evaluation, and RL-based optimization, the framework enables video generators to improve not only in technical realism but also in aesthetic quality and alignment with human preferences. The work addresses the gap between technical metrics and human-centric artistic values, providing a closed-loop system that moves from benchmark construction to actionable model improvement. The authors emphasize that this approach supports the development of AIGC systems that harmonize fidelity, artistry, and human preference.