🤖 AI

RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement

This paper introduces RAVEN-Eval, a scalable, rubric-guided automated evaluation framework that leverages Large Multimodal Model (LMM) preference judgments to reliably distinguish fine-grained quality differences among state-of-the-art AI video generation models while minimizing human annotation costs.

Original authors: Ziheng Jia, Jiaying Qian, Zicheng Zhang, Xiaorong Zhu, Lancheng Gao, Xiongkuo Min

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziheng Jia, Jiaying Qian, Zicheng Zhang, Xiaorong Zhu, Lancheng Gao, Xiongkuo Min

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the director of a massive, high-tech movie studio where the actors are actually super-smart computers. These computers, known as AI video generators, can now create entire scenes from scratch just by reading a sentence you type. But here's the catch: they are getting so good that the difference between a "pretty good" video and a "mind-blowing" one is as subtle as the difference between two shades of blue. In the past, you could just ask a human to watch and say, "That one looks better." But now, with thousands of videos being made every day, hiring enough humans to watch them all would cost a fortune and take forever. Plus, even humans get tired and might miss tiny details. This is the problem scientists are facing: how do you grade these super-advanced computer artists when they are all performing at a near-perfect level, and you can't afford to have a human watch every single clip?

This is where the paper "RAVEN-Eval" comes in. It's like a new, ultra-strict referee system designed to judge these AI movies without needing a human to sit through every single one. The researchers realized that instead of asking a computer, "Rate this video from 1 to 10," which is often confusing and inconsistent, it's better to ask, "Which of these two videos is better?" This is called a "pairwise comparison," and it's much easier for both humans and computers to decide. But to make sure the computer referee doesn't just pick the flashiest video, the team gave it a special rulebook, or "rubric," for every single task. It's like giving a judge a checklist that says, "If the prompt asked for a red ball, the ball must be red," or "If the prompt asked for a hydraulic press crushing a duck, the duck must squish, not explode." By using these specific rules, the system can spot the tiny, fine-grained differences that separate the best AI models from the rest.

The team behind this study, led by researchers from Shanghai Jiao Tong University and the Shanghai Artificial Intelligence Laboratory, built a massive testing ground called the RAVEN-Eval Benchmark. They didn't just throw random prompts at the AI; they carefully curated 250 different tasks, ranging from simple text-to-video requests to complex image-to-video challenges where the AI has to fill in the missing middle of a scene. They generated over 4,500 videos to test 20 of the world's top AI video models. Instead of letting the AI judge itself blindly, they used a "judge" system powered by Large Multimodal Models (LMMs)—which are basically AI brains that can see and understand images and text. These AI judges were given the specific rulebooks for each task and asked to compare videos side-by-side.

The results were fascinating. The researchers found that when they gave the AI judges these detailed, task-specific rulebooks, the system became incredibly good at telling the difference between top-tier models. In fact, their new system agreed with human experts much more closely than previous methods that just asked for a general score or used fixed checklists. They even created two leaderboards: one to rank the 20 AI video models (showing who is currently the best artist) and another to rank the 13 different AI judges themselves (showing which AI brain is the most accurate referee). One of the coolest tricks they invented is called "anchor-based insertion." Imagine you have a leaderboard of the top 20 runners, and a new runner shows up. Instead of making the new runner race against all 20 people (which would take forever), you just have them race against a few carefully chosen "anchor" runners from the top, middle, and bottom of the pack. This lets you figure out where the new runner fits in the ranking with much less effort.

Ultimately, the paper suggests that this new, rule-guided way of using AI to judge AI is a scalable and reliable path forward. It proves that we don't need to rely solely on expensive human annotators to keep up with the rapid pace of AI video generation. By using smart, rule-based comparisons, we can build a trustworthy system that evolves as fast as the models it is trying to measure, ensuring that as these digital artists get better, our ability to appreciate and rank their work gets better, too.

Technical Summary: RAVEN-Eval

Problem Statement

The rapid advancement and commercialization of AI Video Generation Models (AIVGMs) have rendered conventional evaluation paradigms insufficient. Existing methods face two primary challenges:

Scalability and Cost of Human Evaluation: As the quality gap between state-of-the-art (SOTA) models narrows, discerning fine-grained differences requires annotators with higher expertise and sustained attention, leading to prohibitive costs and potential inconsistencies in annotation.
Limitations of Automated Metrics: Current automatic frameworks often rely on predefined dimensions (e.g., visual fidelity, motion quality) or absolute scoring. These approaches struggle to distinguish between advanced models where differences are subtle and context-dependent. Furthermore, absolute scoring lacks the granularity to capture relative performance nuances effectively.

There is a critical need for an automated evaluation framework that can reliably distinguish fine-grained capability differences among SOTA AIVGMs with minimal human intervention while maintaining trustworthiness.

Methodology

The authors propose RAVEN-Eval, a rubric-guided, task-centric automatic evaluation framework built on the "LMM-as-a-judge" paradigm. The framework consists of four core components:

1. The RAVEN-Eval Benchmark

The benchmark is constructed through an automatic pipeline involving task curation, quality filtering, and video generation.

Task Types: It comprises 250 diverse tasks split into Text-to-Video (T2V), Keyframe Expansion (KFT), and First-Last Frame Completion (FLT).
Capability Dimensions: Tasks are constructed based on three complementary dimensions to ensure discriminative power:
- Scene Complexity (S): Ranging from simple layouts to complex spatial structures with multiple objects.
- Dynamic Interaction Complexity (D): Ranging from static scenes to coordinated, multi-stage interactions.
- Knowledge Requirement (K): Ranging from everyday scenarios to those requiring expert-level domain knowledge (e.g., physics, chemistry, professional aesthetics).
- Reasoning Tasks: Approximately 20% of tasks are reasoning-based, where prompts provide only clues, requiring the model to infer the expected outcome.
Data Scale: The benchmark includes over 4,600 high-quality generated videos from 20 different AIVGMs.
Filtering: A two-stage filtering process removes low-difficulty tasks and ensures that retained tasks exhibit clear performance gaps between solvers of varying capabilities.

2. Rubric-Guided Pairwise Preference Judgement

Instead of absolute scoring, RAVEN-Eval employs pairwise comparisons guided by task-specific rubrics.

Rubric Construction: For each task, a two-level rubric is generated:
- Primary Criteria: Task-specific requirements derived from the prompt and enriched details (e.g., specific object counts, interaction processes, physical consistency).
- Basic Criteria: General quality metrics (e.g., visual fidelity, naturalness) applied uniformly across all tasks.
Judgement Protocol: Large Multimodal Models (LMMs) act as judges, comparing video pairs based on the rubric. To ensure stability, the system employs an order-robust protocol where pairs are evaluated twice with reversed input orders.
Tie-Aware Optimization: The framework utilizes a modified Davidson tie-aware Maximum Likelihood Estimation (MLE) to estimate model capabilities. This explicitly models the probability of a "tie" and downweights uncertain judgments, addressing the issue of indistinguishable pairs.

3. Anchor-Based Model Insertion

To address the quadratic growth of pairwise comparisons as new models are added, RAVEN-Eval introduces an anchor-based insertion strategy.

Anchor Selection: A subset of existing models is selected as "anchors" based on their reliability scores (agreement with joint evaluations and stability across judges).
Efficiency: New models are compared only against these anchors rather than the entire pool, significantly reducing evaluation costs while maintaining ranking accuracy.

4. Leaderboards

The framework establishes two complementary leaderboards:

AIVGM Leaderboards: Rankings of 20 recent open-source and proprietary generation models across T2V, KFT, and FLT tasks.
Judge Leaderboards: Benchmarks evaluating the preference-judging capabilities of 13 general-purpose LMMs against human preferences.

Key Results

Model Performance: The RAVEN-Eval AIVGM Leaderboards successfully rank 20 SOTA models. Seedance 2 emerged as the top performer across T2V, KFT, and FLT categories, followed by Grok-Imagine Video and Kling 3.0 Pro.
Evaluation Reliability: The framework demonstrates strong alignment with human preferences. The Spearman Rank Correlation Coefficient (SRCC) between RAVEN-Eval rankings and human-annotated rankings reached 0.872 for T2V and 0.903 for KFT, outperforming existing automatic methods (e.g., VideoScore, LOVE) and ablation variants without rubrics.
Rubric Effectiveness: Ablation studies confirm that rubric-guided pairwise evaluation significantly outperforms both rubric-guided absolute scoring and non-rubric pairwise evaluation, highlighting the necessity of task-specific guidance for fine-grained discrimination.
Anchor Efficiency: The anchor-based insertion strategy reduces pairwise annotation costs by 34.2% (for $K=1$ ) and 21.1% (for $K=2$ ) while maintaining ranking consistency (SRCC > 0.8) with fully connected graph evaluations.
LMM Judge Capabilities: The Judge Leaderboards reveal that larger, more capable LMMs generally achieve higher agreement with human preferences and produce more confident judgments.

Significance and Claims

The paper claims that RAVEN-Eval establishes a scalable and trustworthy paradigm for the automatic evaluation of rapidly evolving AIVGMs. Its primary contributions are:

Fine-Grained Discrimination: By moving away from fixed dimensions to task-centric, rubric-guided scenarios, the framework can capture subtle capability differences that traditional metrics miss.
Cost-Effective Automation: The LMM-as-a-judge paradigm, combined with the anchor-based insertion strategy, offers a viable path to continuously update benchmarks without the prohibitive costs of full-scale human annotation.
Comprehensive Benchmarking: The establishment of dual leaderboards (for models and judges) provides a standardized reference for both model development and the evaluation of LMM judging capabilities.
Training-Free Flexibility: The framework demonstrates strong performance across diverse LMM judges without requiring specific training for the evaluation task, suggesting a robust "model-harnessing" capability.

The authors conclude that RAVEN-Eval paves the way for efficient, automated, and reliable assessment of AI video generation, addressing the critical bottleneck of evaluation in the era of SOTA model proliferation.