Q-Save: Towards Scoring and Attribution for Generated Video Evaluation

This paper introduces Q-Save, a holistic benchmark dataset and unified evaluation model that jointly assesses visual quality, dynamic quality, and text-video alignment for AI-generated videos through a three-stage training strategy, thereby overcoming the limitations of isolated and undefined evaluation approaches.

The Gist

Imagine you just bought a brand-new, high-tech video camera that claims to create movies from thin air just by listening to your voice commands. You ask it to "show a cat riding a skateboard through a neon city," and it spits out a video. But is it good? Is the cat real? Did the skateboard actually move, or did it just slide like a ghost? Is the cat even wearing a helmet as you asked?

Right now, checking these videos is like trying to grade a thousand essays with a broken ruler. Existing tools are either too simple (just giving a single number like "7/10" without saying why) or they look at the video too briefly (like glancing at a movie and judging the whole film based on two random frames).

Enter Q-Save. Think of Q-Save as the ultimate, super-strict film critic who doesn't just give a grade but writes a detailed report card explaining exactly what went right and what went wrong.

Here is how Q-Save works, broken down into simple parts:

1. The Three-Part Report Card

Instead of just saying "Good" or "Bad," Q-Save judges every video on three specific dimensions, like a teacher grading a student on different subjects:

• Visual Quality (The "Look"): Is the picture sharp? Are the colors bright? Does it look like a glitchy mess or a professional movie?
• Dynamic Quality (The "Move"): This is the tricky part. Does the movement make sense? If a person runs, do their legs move naturally, or do they look like they're sliding on ice? Does the water flow like water, or does it freeze in mid-air?
• Text-Video Alignment (The "Listening"): Did the AI actually do what you asked? If you asked for a "red dog," did it give you a "blue cat"?

2. The "Slow and Fast" Camera Trick

Most AI critics are lazy; they only look at a few frames of a video to save time. It's like judging a marathon runner by looking at a photo taken at the start line and another at the finish line, missing everything that happened in between.

Q-Save uses a clever trick called SlowFast. Imagine watching a movie with two pairs of eyes:

• The "Fast" Eyes: These scan the whole video quickly to catch the general flow and timing.
• The "Slow" Eyes: These zoom in on the important, changing moments (like a ball hitting the ground or a character turning their head) to see the tiny details.
  By combining these, Q-Save catches errors that other critics miss, like a wobbly leg or a sudden glitch, without needing to read the entire video frame-by-frame.

3. The "Why" Factor (Attribution)

This is Q-Save's superpower. Old critics just say, "This video is bad." Q-Save says, "This video is bad because the cyclist's legs are bending backward, and the background is blurry."

The researchers taught Q-Save to write these explanations (called Chain-of-Thought). It's like training a student not just to get the right answer on a math test, but to show their work. This helps the AI learn why something is wrong so it can get better at spotting those specific mistakes next time.

4. The Training Boot Camp

To build this critic, the researchers didn't just feed it a few videos. They created a massive boot camp:

• The Dataset: They made 10,000 videos using the best AI generators available.
• The Teachers: They hired humans to watch these videos and give detailed scores and reasons.
• The Training Strategy: They trained the AI in three stages:
  1. SFT (The Classroom): Teaching the AI the basic rules and how to write reports.
  2. RL (The Practice Match): Letting the AI play against itself and learn from its mistakes to become more accurate.
  3. Cool Down (The Final Polish): A final round of training to make sure the AI is calm, consistent, and doesn't get confused.

Why Does This Matter?

Think of AI video generators as a factory churning out millions of videos a day. Without a good inspector, bad videos slip through, wasting money and confusing users.

Q-Save is that super-inspector. It helps:

• Developers fix their AI models faster by telling them exactly what's broken.
• Users know if the video they generated is actually usable.
• Researchers build better AI by having a clear, fair way to measure progress.

In short, Q-Save turns the chaotic world of AI video generation into something we can actually measure, understand, and improve, ensuring that the movies of the future look as good as they sound.

Technical Summary

1. Problem Statement

The rapid advancement of Text-to-Video (T2V) models has led to a surge in AI-generated content (AIGC), but current automated evaluation frameworks face significant limitations:

• Lack of Systematic Definitions: Existing benchmarks often lack clear, standardized definitions for evaluation dimensions.
• Isolated Evaluation: Most models treat visual quality, dynamic quality, and text-video alignment as separate tasks rather than a unified problem.
• Data Quality Issues: Many datasets suffer from low-quality prompts (sampled from public sources), insufficient annotation calibration, and a lack of fine-grained attribution (explanations for why a video is good or bad).
• Preprocessing Mismatch: Many Vision-Language Models (VLMs) use sparse frame sampling (e.g., 2 fps), which fails to capture the rich temporal evidence required for human-like dynamic quality assessment.
• Training Limitations: Current approaches often rely on single-stage supervised training, failing to fully leverage modern VLM capabilities for reasoning and alignment.

2. Methodology

The authors propose Q-Save, a holistic framework comprising a high-quality dataset and a unified evaluation model.

A. The Q-Save Dataset

• Scale & Composition: Contains nearly 10,000 video samples generated by six state-of-the-art T2V models (including Kling, Hunyuan, Veo2, etc.).
• Three Core Dimensions: Every video is annotated across:
  1. Visual Quality (VQ): Frame-level perceptual quality (fidelity, sharpness, artifacts).
  2. Dynamic Quality (DQ): Temporal dynamics (smoothness, coherence, physical plausibility).
  3. Text-Video Alignment (TA): Semantic consistency between the prompt and the generated video.
• Annotation Protocol:
  • Prompts: Manually crafted with balanced distribution across categories and explicit dynamic actions/camera motions.
  • Human Ratings: Uses Mean Opinion Scores (MOS) on a 1–5 scale.
  • Attribution: For videos with defects, raters provide fine-grained natural language explanations identifying the cause of degradation (e.g., "unnatural leg pedaling," "text distortion").
  • Quality Control: Rigorous rater training, calibration, and a "Sample and Scrutinize" strategy to ensure high inter-rater reliability.

B. The Q-Save Model Architecture

• Backbone: Built upon Qwen3-VL-8B-Instruct, a strong video-capable VLM.
• Video Preprocessing (SlowFast Strategy):
  • To balance token budget and temporal detail, the model employs a SlowFast encoding strategy.
  • Slow Pathway: Processes key frames (high resolution) where significant changes occur.
  • Fast Pathway: Processes static frames (lower resolution) to maintain temporal context.
  • This ensures the model captures both spatial details and motion cues efficiently.
• Scoring Mechanism:
  • The model predicts discrete rating tokens (Bad, Poor, Fair, Good, Excellent).
  • A softmax-based expectation is used to convert these probabilities into a continuous score, reducing discretization noise.
  • The output includes both the score and a Chain-of-Thought (CoT) attribution analysis.

C. Three-Stage Training Pipeline

To maximize performance and stability, the authors employ a hybrid training strategy:

1. Stage 1: Supervised Fine-Tuning (SFT) - Cold Start:
   • Trains the model on the Q-Save dataset with CoT-formatted data to learn basic instruction following and the rating format.
2. Stage 2: Reinforcement Learning (RL) - Warm Up:
   • Uses Group Relative Policy Optimization (GRPO).
   • Rewards are based on Accuracy (matching ground truth labels) and Format (including the CoT reasoning).
   • This stage aligns the model with human preferences and improves robustness against shortcut learning.
3. Stage 3: SFT - Cool Off:
   • A final round of SFT using the high-quality rollouts from the RL stage.
   • This "distills" the RL improvements, stabilizing the scoring behavior and reducing variance introduced by RL.

3. Key Contributions

1. Q-Save Dataset: A large-scale, attribution-aware benchmark with ~10k videos, strict quality control, and fine-grained explanations for three core evaluation dimensions.
2. SlowFast Preprocessing for VLMs: A novel adaptation of the SlowFast architecture for VLM-based video evaluation, optimizing token usage while preserving critical temporal evidence.
3. Three-Stage Training Pipeline: A novel SFT $\to$ RL $\to$ SFT strategy that effectively unlocks VLM capabilities, achieving superior alignment and stability compared to single-stage training.
4. Unified Evaluation: A single model capable of jointly performing quality scoring and generating interpretable attribution explanations.

4. Experimental Results

• In-Domain Performance: Q-Save achieves state-of-the-art results on the Q-Save test set across all three dimensions (Visual, Dynamic, Alignment).
  • It significantly outperforms baselines like FastVQA, SimpleVQA, Q-Align, and VideoScore-v2 in both instance-level and model-level correlations (SRCC/PLCC).
  • Notably, it achieves near-perfect correlation (PLCC ~0.97–1.00) in model-level evaluation, indicating excellent capability in ranking different video generators.
• Cross-Dataset Generalization: The model demonstrates strong transferability on out-of-domain benchmarks (e.g., VideoGen-RewardBench, VideoPhy2-test), outperforming specialized baselines in both preference accuracy and MOS correlation.
• Ablation Studies:
  • SlowFast: Improves instance-level correlation by better allocating capacity to informative frames.
  • RL: Significantly boosts model-level ranking consistency.
  • Final SFT: Further stabilizes predictions and reduces variance.
• Reward Model Application: When used as a reward model to fine-tune T2V generators (Wan2.2, Hunyuan), Q-Save yields higher human evaluation scores compared to using open-source baselines like HPSv3.

5. Significance

• Interpretability: By providing attribution explanations alongside scores, Q-Save moves beyond "black box" evaluation, offering actionable feedback for model developers.
• Efficiency & Accuracy: The SlowFast preprocessing allows for high-fidelity dynamic quality assessment within the token constraints of current VLMs.
• Standardization: The three-dimensional framework (Visual, Dynamic, Alignment) offers a comprehensive standard for future AIGC evaluation.
• Feedback Loop: The ability to serve as a reward model suggests Q-Save can directly drive the optimization of next-generation T2V models, creating a virtuous cycle of improvement.

In conclusion, Q-Save represents a significant step forward in automated video evaluation by combining high-quality data, efficient temporal processing, and advanced training strategies to deliver both accurate scoring and human-interpretable reasoning.