Analytic Score Optimization for Multi Dimension Video Quality Assessment

This paper introduces UltraVQA, a large-scale multi-dimensional video quality assessment dataset with human-annotated dimensions and AI-generated rationales, alongside Analytic Score Optimization (ASO), a theoretically grounded post-training method that improves prediction accuracy and alignment with human preferences by reframing quality assessment as a regularized decision-making process.

The Gist

Imagine you are a food critic reviewing a new restaurant. In the old days, you might just give the meal a single number: 7 out of 10. That's helpful, but it's also vague. Did you give it a 7 because the steak was tough? Because the lighting was bad? Or because the dessert was amazing but the soup was cold? A single number hides the why.

This paper is about upgrading that food critic system for videos. Instead of just giving a video one "score," the authors built a system that breaks the review down into five specific categories: Motion, Movement Size, Beauty, Story, and Clarity.

Here is a simple breakdown of their three big contributions:

1. The "UltraVQA" Dataset: A Massive, Detailed Scorecard

Imagine you want to teach a robot how to judge videos. If you just show it 1,000 videos and say "this one is good, that one is bad," the robot gets confused. It doesn't know what makes a video good.

The authors created UltraVQA, a giant library of 40,000 videos. But here's the twist:

• The Human Panel: Instead of one person judging, they used a team of 40 trained experts. Every video was watched by at least three different people.
• The 5-Point Menu: Instead of one score, the experts rated every video on five specific things:
  1. Motion Quality: Is the movement smooth, or is it jittery like a shaky phone camera?
  2. Motion Amplitude: Is there a lot of action, or is it a still image?
  3. Aesthetic Quality: Is it pretty? Good lighting? Nice colors?
  4. Content Quality: Does the story make sense? Is the subject clear?
  5. Clarity Quality: Is it sharp, or is it blurry and pixelated?
• The "Why" (Rationale): Crucially, the experts didn't just write numbers. They wrote short explanations (e.g., "The video is blurry because the camera shook"). The authors then used an AI (GPT) to turn these human notes into clear, structured paragraphs. This teaches the robot not just what the score is, but why it got that score.

2. The "Analytic Score Optimization" (ASO): The Smart Math Trick

This is the technical heart of the paper, but let's use an analogy.

Imagine you are training a dog to fetch a ball.

• The Old Way (Regression): You tell the dog, "Bring the ball to the 7.5-meter mark." If the dog brings it to 7.4 meters, you say "Close, but no." This is frustrating because human opinions are rarely exact. One person might think a video is a "3.5," and another might think it's a "4.0."
• The New Way (ASO): The authors realized that video scores are like a ladder, not a ruler. You can be on rung 3, rung 3.5, or rung 4. You can't be "between" rungs.

They invented a math method called Analytic Score Optimization (ASO).

• Think of it as a GPS for the AI. Instead of guessing randomly and hoping to get the right answer (which is slow and unstable), ASO calculates the perfect probability distribution for the answer.
• It says: "Based on the human data, there is a 60% chance the score is 3.5, a 30% chance it's 4.0, and a 10% chance it's 3.0."
• It forces the AI to learn this probability map rather than just memorizing a single number. This makes the AI much more stable and accurate, especially for tricky things like "Motion," where the difference between a "good" and "bad" video can be very subtle.

3. The Results: A Smarter, More Human-Like Critic

When they tested their new system (UltraVQA + ASO) against other top AI models and even expensive closed-source APIs (like GPT-4):

• It was more accurate: It predicted scores closer to what humans actually thought.
• It was better at explaining: Because it was trained on the "rationale" (the "why"), it could give better reasons for its scores.
• It generalized well: Even when shown videos it had never seen before (like sports clips or news), it still performed better than specialized video models.

The Big Picture

Before this paper, AI video judges were like a student who memorized the answer key but didn't understand the math. They could guess a score, but they couldn't explain it, and they struggled with the nuances of human taste.

This paper gives the AI a detailed textbook (the dataset) and a better study method (the ASO math). The result is a video judge that doesn't just say "This is a 7/10," but can say, "This is a 7/10 because the story is great, but the camera shake makes the motion quality a bit rough."

It's a step toward AI that doesn't just see pixels, but truly understands the experience of watching a video.

Technical Summary

1. Problem Statement

Video Quality Assessment (VQA) is transitioning from single-number Mean Opinion Scores (MOS) to multi-dimensional evaluations. However, existing approaches face three critical limitations:

• Lack of Interpretability: Traditional methods output a single scalar, obscuring why a video is rated poorly (e.g., failing to distinguish between motion artifacts and aesthetic issues).
• Inadequate Supervision: Current Vision-Language Models (VLMs) often treat scoring as continuous regression or free-form generation. This ignores the discrete, ordinal nature of human ratings (e.g., 1.0 to 5.0 in 0.5 steps) and fails to capture the uncertainty and subjectivity inherent in human judgment.
• Training Instability: Standard Reinforcement Learning (RL) methods like PPO or GRPO rely on stochastic policy gradients, which suffer from high variance and sample inefficiency, particularly when aligning models to subtle ordinal distinctions in dynamic video dimensions (e.g., motion quality).

2. Methodology

The paper proposes a two-pronged solution: a new large-scale dataset and a novel optimization algorithm.

A. UltraVQA Dataset

The authors introduce UltraVQA, a large-scale dataset designed for comprehensive and interpretable VQA.

• Scale & Diversity: Contains ~40,000 video clips from diverse User-Generated Content (UGC) and professional sources, covering 16 thematic categories (e.g., gaming, vlogs, sports).
• Multi-Dimensional Taxonomy: Videos are annotated across five key dimensions:
  1. Motion Quality: Temporal smoothness and stability.
  2. Motion Amplitude: Degree and extent of motion.
  3. Aesthetic Quality: Composition, lighting, and visual appeal.
  4. Content Quality: Semantic relevance and coherence.
  5. Clarity Quality: Sharpness, resolution, and compression artifacts.
• Annotation Protocol: Each video is rated by at least 3 trained annotators on a 1.0–5.0 scale (0.5 intervals) with fine-grained sub-attribute tags (e.g., "motion blur," "poor lighting").
• Rationale Synthesis: To provide interpretable supervision, the authors use GPT-4.1 to synthesize concise, evidence-grounded textual rationales based on human scores and tags, ensuring the model learns to justify its predictions.

B. Analytic Score Optimization (ASO)

To leverage the discrete and ordinal nature of the dataset, the authors propose ASO, a theoretically grounded post-training objective.

• Formulation: ASO reframes discrete scoring as a KL-regularized one-step bandit problem. Instead of stochastic sampling, it seeks a closed-form optimal policy distribution over discrete score levels.
• Mathematical Derivation:
  • The objective maximizes expected reward $R(s, s^*)$ subject to a KL-divergence constraint against a reference policy $\pi_{ref}$ (the SFT model).
  • By solving the Lagrangian of this convex optimization problem, the authors derive a closed-form optimal policy:
    $$\pi^*(s|x) = \frac{1}{Z(x)} \pi_{ref}(s|x) \exp\left(\frac{1}{\lambda} R(s, s^*)\right)$$
  • Here, $R(s, s^*)$ is a reward function (e.g., negative absolute error), and $\lambda$ controls the regularization strength.
• Training Strategy:
  1. SFT: Fine-tune a VLM (e.g., Qwen2.5-VL) to output structured scores and rationales.
  2. ASO Alignment: Treat the derived closed-form distribution $\pi^*$ as a "soft target" (teacher). The model is trained to minimize the KL divergence between its current policy $\pi_\theta$ and $\pi^*$.
  • Advantage: Unlike GRPO, which relies on high-variance sampling, ASO directly optimizes the entire probability mass, providing a stable, sample-efficient, and theoretically optimal alignment for discrete ordinal tasks.

3. Key Contributions

1. UltraVQA Dataset: A high-quality, multi-dimensional benchmark with 5 quality dimensions, fine-grained sub-attributes, and human-verified rationale explanations, addressing the scarcity of interpretable VQA data.
2. Analytic Score Optimization (ASO): A novel, RL-inspired objective that derives a closed-form solution for discrete score alignment. It effectively models human rating uncertainty and ordinal relationships without the instability of online RL.
3. Rationale Supervision: Demonstrates that training with synthesized, evidence-grounded rationales improves both score prediction accuracy and model interpretability.

4. Experimental Results

The method was evaluated on UltraVQA and multiple public benchmarks (LSVQ, KoNViD-1k, VideoPhy2, MJ-Video).

• Performance on UltraVQA:
  • The ASO-trained model significantly outperformed strong baselines, including closed-source APIs (GPT-4.1, Gemini-2.5Pro) and specialized VQA models (FineVQ, Q-Align).
  • Metrics: Achieved state-of-the-art results in Accuracy (Acc@0.5), Spearman Rank Correlation (SRCC), and Mean Absolute Error (MAE) across all five dimensions. For example, on Motion Quality, ASO achieved 81.5% Acc and 0.430 MAE, surpassing the best baseline (VideoScoreV2) which scored 69.8% Acc.
• Generalization:
  • ASO demonstrated robust cross-benchmark generalization, outperforming generalist VLMs on physical reasoning (VideoPhy2) and preference tasks (MJ-Video).
• Ablation Studies:
  • ASO vs. GRPO: ASO consistently outperformed GRPO, particularly on dynamic dimensions like Motion Quality. This confirms that the analytic, closed-form approach captures subtle ordinal distinctions better than stochastic policy gradients.
  • Necessity of Alignment: Moving from SFT to ASO yielded significant performance leaps, proving that standard teacher-forcing is insufficient for ordinal quality calibration.

5. Significance

This work represents a paradigm shift in VQA by moving away from "black-box" regression toward interpretable, multi-dimensional assessment.

• Theoretical Contribution: It provides a mathematically rigorous solution for aligning LLMs with discrete, ordinal human preferences, solving the instability issues common in RLHF for scoring tasks.
• Practical Impact: The UltraVQA dataset and ASO method enable the development of VQA systems that not only predict how good a video is but explain why, which is crucial for content moderation, recommendation systems, and generative video evaluation.
• Future Direction: The paper highlights the importance of combining high-quality human annotations with theoretically grounded optimization objectives to advance multimodal understanding.