💻 computer science

Comparison Drives Preference: Reference-Aware Modeling for AI-Generated Video Quality Assessment

This paper proposes RefVQA, a novel reference-aware framework that improves AI-generated video quality assessment by modeling inter-video relationships through a query-centered graph to aggregate differences from semantically related samples, thereby outperforming existing independent evaluation methods.

Original authors: Minghao Zou, Gen Liu, Guanghui Yue, Baoquan Zhao, Zhihua Wang, Paul L. Rosin, Hantao Liu, Wei Zhou

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minghao Zou, Gen Liu, Guanghui Yue, Baoquan Zhao, Zhihua Wang, Paul L. Rosin, Hantao Liu, Wei Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Judging a Video in a Vacuum

Imagine you are a food critic asked to rate a new burger. If you've never tasted a burger before, or if you are eating it in a dark room with no other food to compare it to, it's incredibly hard to say, "Is this a 7 out of 10 or a 9 out of 10?" You might just guess.

This is exactly the problem with current AI tools that rate AI-generated videos.

The Old Way: Current AI models look at a video all by itself. They try to guess, "Is this blurry? Is the motion weird?" without any context. It's like trying to judge a burger's taste in a vacuum.
The Human Way: Humans don't judge in a vacuum. When we see a new video, our brains instantly compare it to other videos we've seen. "This one looks clearer than that one I saw yesterday," or "The motion here is smoother than that other clip." We use comparison to form an opinion.

The Solution: The "Reference-Aware" Approach

The authors of this paper realized that to make AI better at judging video quality, we need to teach it to compare, not just observe. They call their new method RefVQA (Reference-aware Video Quality Assessment).

Think of RefVQA not as a lone critic, but as a judge in a talent show who has a panel of previous contestants to compare against.

How It Works (The 3-Step Process)

1. The "Prompt" is the Menu Order
AI videos are created using text prompts (e.g., "A dog running in a park"). The researchers realized that the text prompt is the best way to find similar videos.

Analogy: If you order a "Spicy Tacos" dish, you don't compare it to a "Chocolate Cake." You compare it to other "Spicy Tacos."
The Tech: When a new video needs a rating, the system looks at its text prompt and goes to a library of thousands of other videos to find ones with similar "orders" (prompts).

2. Building a "Comparison Graph"
Once it finds similar videos, it doesn't just look at them randomly. It builds a map (a graph) connecting the new video to its "cousins."

Analogy: Imagine the new video is a student taking a test. The system gathers a group of students who took the same test questions (similar prompts). It creates a study group where they can compare their answers.
The Tech: This map connects the "Target Video" to "Reference Videos" based on how similar their text descriptions are.

3. The "Difference Detector"
This is the secret sauce. Instead of just looking at the new video, the system looks at the difference between the new video and its references.

Analogy: If you are judging a runner, you don't just time them. You look at how much faster they are than the person who came in second.
The Tech: The AI calculates the "gap" between the new video and the similar ones.
- Is the new video blurrier than the similar ones? -> Lower score.
- Is the new video moving more smoothly than the similar ones? -> Higher score.
- Does the new video actually match the text prompt better than the others? -> Higher score.

Why This is a Big Deal

1. It Mimics Human Brains
Humans are naturally comparative. We rarely say, "This is perfect" without thinking, "Perfect compared to what?" This method forces the AI to think the same way, making its ratings much closer to what a human would say.

2. It Catches Subtle Mistakes
AI videos often have weird glitches (like a hand with six fingers or a car floating in the air).

Old AI: Might miss these because it's looking at the video in isolation.
RefVQA: Sees a video of a "dog running" and compares it to 50 other "dog running" videos. If the new dog's legs are glitching while the others look normal, RefVQA immediately spots the error because it has a baseline to compare against.

3. It Works on New Data
The paper tested this on three different datasets (like three different schools). RefVQA didn't just memorize the answers; it learned the skill of comparing. This means it works well even on videos it has never seen before, as long as it can find similar examples to compare them to.

The Result

In their experiments, RefVQA beat all the previous "state-of-the-art" methods. It was better at:

Visual Quality: Spotting blur or weird shapes.
Motion Quality: Spotting jerky or unnatural movement.
Alignment: Checking if the video actually matches the text description (e.g., if the prompt says "sunset," is it actually a sunset?).

Summary

Imagine you are trying to guess the price of a used car.

Old Method: You look at the car and guess based on how shiny it is.
RefVQA Method: You look at the car, then you pull up a list of 10 other cars of the same make, model, and year. You compare the mileage, the dents, and the engine sound. Then you give a price based on how that car stacks up against the others.

This paper teaches AI to do the second thing: Stop judging in a vacuum, and start judging by comparison. And because humans do the same thing, the AI becomes much smarter at understanding what we think is "good" or "bad."

1. Problem Statement

The rapid proliferation of AI-generated videos (AIGVs) has created an urgent need for automated quality assessment (AIGC-VQA). Existing methods typically evaluate each video independently, analyzing intrinsic features like motion smoothness or semantic consistency in isolation. However, the authors argue that human perception of video quality is inherently comparative; observers rarely judge a video in a vacuum but rather compare it against other videos with similar objects, actions, or styles.

Current independent evaluation models fail to capture these relative perceptual differences, leading to suboptimal alignment with human subjective judgments, particularly regarding complex artifacts like structural inconsistencies, motion instability, and semantic misalignment with prompts.

2. Methodology: RefVQA

The authors propose RefVQA (Reference-aware Video Quality Assessment), a framework that reformulates AIGC-VQA as a reference-aware evaluation problem. Instead of predicting quality in isolation, the model assesses a target video by comparing it to semantically related reference videos.

The framework consists of four main modules:

A. Reference Graph Construction

Retrieval: Given a query video and its text prompt, the system retrieves semantically similar videos from a training pool. Similarity is measured using Sentence-BERT on the text prompts (not visual features), as prompts provide a stable, cross-model semantic anchor.
Graph Structure: A query-centered graph is constructed where the query video is the central node, and retrieved videos are neighbor nodes. Edges are weighted by the prompt similarity score. This topology defines the relational structure for feature modeling.

B. Dual-Branch Feature Modeling

The model processes information through two parallel branches, both utilizing the reference graph:

Visual Branch: Uses a Swin Transformer to extract spatiotemporal features (appearance and motion).
Alignment Branch: Uses BLIP to encode the semantic consistency between the generated video and its conditioning prompt.

C. Graph-Guided Difference Aggregation

This is the core innovation. Rather than aggregating raw reference features, the model explicitly models the difference between the query and its references:

Difference Features: For each reference node $n$ , the model computes $\Delta v_n = v_q - v_n$ (visual difference) and $\Delta s_n = s_q - s_n$ (alignment difference).
Aggregation: These difference features are aggregated from reference nodes to the query node using the graph topology. The aggregation is weighted by the prompt similarity scores.
Gating Mechanism: A learnable gating mechanism adaptively regulates how much the aggregated reference information contributes to the final representation, ensuring the model focuses on relevant discrepancies.

D. Quality Prediction

The enhanced visual and alignment representations are fused and passed through a regression head to predict the final Mean Opinion Score (MOS). The model is trained using a hybrid loss function combining PLCC loss (linear consistency) and Rank loss (pairwise ordering).

3. Key Contributions

Paradigm Shift: The paper reformulates AIGC-VQA from an independent regression task to a reference-aware comparative reasoning task, aligning the model's logic with human perception.
RefVQA Framework: A novel architecture that learns reference-conditioned discrepancy representations in both visual and alignment spaces.
Graph-Guided Difference Aggregation: A lightweight mechanism that organizes retrieved references into a graph and encodes query-reference differences under a structured topology, rather than simple averaging.
Prompt-Based Retrieval: Demonstrates that using text prompt similarity (rather than visual feature similarity) is more robust for establishing semantic comparability across different generative models.

4. Experimental Results

The method was evaluated on three benchmark datasets: T2VQA-DB, LGVQ, and FETV.

In-Domain Performance: RefVQA achieved State-of-the-Art (SOTA) results across all metrics (SRCC, PLCC, KRCC, RMSE) on T2VQA-DB and LGVQ. Notably, it showed significant improvements in the "Alignment" dimension, proving that reference comparison helps assess video-prompt consistency better than independent models.
Cross-Dataset Generalization: When trained on LGVQ and tested on FETV, RefVQA outperformed all competing methods, demonstrating strong generalization capabilities. This suggests the model learns relative quality cues rather than dataset-specific feature distributions.
Efficiency: Despite the added complexity of reference retrieval, RefVQA achieved faster inference times (99.265 ms) compared to baselines like T2VQA (122.187 ms) due to a staged training strategy where backbone features are pre-extracted and cached.
Ablation Studies:
- Removing reference modeling caused a significant performance drop.
- Prompt-based retrieval outperformed random, batch, or visual-feature-based retrieval.
- Modeling difference features (Query - Reference) was more effective than using raw reference features.
- Combining both Visual and Alignment branches yielded the best results, confirming their complementary nature.

5. Significance and Impact

Human-Centric Assessment: By mimicking the human tendency to compare videos, RefVQA bridges the gap between algorithmic prediction and subjective human judgment.
Robustness: The reliance on prompt similarity for reference selection makes the system robust to domain shifts and variations in generation quality, as prompts are more stable than visual features.
Future Direction: This work establishes a new direction for AIGC-VQA, suggesting that future quality assessment models should move beyond isolated sample analysis to incorporate inter-video relationships and comparative reasoning. It opens avenues for improved reference selection strategies and more expressive relational modeling (e.g., full graph propagation).

In conclusion, the paper demonstrates that "comparison drives preference" in video quality assessment, and leveraging this insight through a structured, reference-aware framework leads to superior, more generalizable, and human-aligned quality predictions.