Score2Instruct: Scaling Up Video Quality-Centric Instructions via Automated Dimension Scoring

1. Problem Statement

Video Quality Assessment (VQA) traditionally relies on generating a single numerical score (e.g., Mean Opinion Score) to represent overall visual fidelity. While effective for ranking, this approach has significant limitations:

• Lack of Interpretability: A single score fails to describe complex, interrelated quality dimensions (e.g., noise, motion blur, focus, color vibrancy).
• Scalability Bottleneck: Existing methods for training Large Multimodal Models (LMMs) to provide quality justifications rely heavily on human expert annotations or proprietary APIs (like GPT-4). These are time-consuming, expensive, prone to bias, and difficult to scale to large datasets.
• Temporal Complexity: Previous work has largely focused on Image Quality Assessment (IQA), lacking deep understanding of temporal dynamics specific to video (e.g., flicker, motion blur, frame fluency).
• Simultaneous Capability Gap: Current models struggle to perform both precise quality scoring and comprehensive quality justification simultaneously.

2. Methodology

The authors propose Score2Instruct (S2I), a framework centered on a novel Score-based Instruction Generation (SIG) pipeline. This pipeline automates the creation of a large-scale, high-quality instruction dataset without human annotation.

A. The SIG Pipeline

The SIG pipeline consists of three key stages:

1. Video Source Collection:

   • To overcome the scarcity of labeled VQA datasets, SIG samples from both labeled VQA databases (e.g., Maxwell, which has multi-dimensional labels) and unlabeled general video databases (e.g., Kinetics, YFCC).
   • Criteria: Videos are selected based on "annotation richness" (number of quality dimensions) and "balanced quality distribution." A lightweight video quality assessor is used to estimate scores for unlabeled videos, followed by uniform sampling to ensure a balanced distribution of quality levels.
   • Scale: This process yields over 104K videos.

2. Automated Quality Dimension Scoring:

   • Instead of asking an LLM to guess quality, SIG uses expert-trained models (from a video processing platform) to score 14 distinct quality dimensions automatically.
   • Dimensions: These cover shooting, editing, compression, and transmission phases (e.g., Focus, Sharpness, Noise, Motion Blur, Flicker, Colorfulness, Composition).
   • Mapping: Continuous scores (0-1) are mapped to discrete, text-defined levels based on ITU standards (Bad, Poor, Fair, Good, Excellent) to align with LLM token outputs. Definitions are added to dimension names to eliminate ambiguity.

3. Hierarchical Chain-of-Thought (CoT) Aggregation:

   • To mimic the Human Visual System (HVS), the pipeline aggregates the 14 dimension ratings into a coherent justification using a hierarchical CoT.
   • Process:
     1. Group dimensions into "Distortion" and "Aesthetic" categories.
     2. Derive intermediate ratings for each group.
     3. Synthesize a final overall quality rating.
   • Refinement: An open-source LLM (Vicuna) rephrases the CoT output to improve naturalness and integrates high-level video captions (from ShareCaptioner) to ensure the justification is grounded in the video content.
   • Expansion: The LLM further expands the dataset by generating 216K Question-Answer (QA) pairs (Yes/No and What/How formats) based on the justifications.

B. The Score2Instruct (S2I) Dataset

The resulting dataset contains 320K instruction-response pairs:

• 104K detailed quality justifications with fine-grained dimension ratings.
• 216K diverse QA pairs.
• Key Feature: It is fully automated, scalable, and covers comprehensive quality dimensions without relying on proprietary APIs or massive human labeling efforts.

C. Progressive Tuning Strategy

To fully leverage S2I, the authors propose a two-stage tuning strategy for Video LMMs:

1. Stage I (Pretraining): The model is trained on the 100K dimension descriptions to learn quality scoring. The task is simple (predicting text-defined levels for specific dimensions), allowing the model to focus on visual features without overfitting to complex language.
2. Stage II (Instruction Tuning): The model is fine-tuned on the 220K justifications and QA pairs to enhance quality justification and reasoning capabilities. Low-Rank Adaptation (LoRA) is used for efficiency.

D. S2I-Bench

A new benchmark comprising 400 open-ended questions derived from S2I (with manual verification) is introduced to evaluate video LMMs. It utilizes VCG scores (Correctness, Detail Orientation, Contextual Understanding, Temporal Understanding) adapted from prior work to provide a more holistic evaluation than simple accuracy metrics.

3. Key Contributions

1. SIG Pipeline: A scalable, automated pipeline for generating video quality-centric instructions that eliminates reliance on human experts and proprietary APIs.
2. S2I Dataset: A massive dataset (320K pairs) covering 14 quality dimensions, enabling instruction tuning for both scoring and justification.
3. S2I-Bench: A rigorous benchmark with open-ended questions and VCG-based metrics to evaluate the depth of quality understanding in video LMMs.
4. Progressive Tuning Strategy: A novel two-stage training approach that effectively decouples and then integrates quality scoring and justification learning.

4. Experimental Results

The method was evaluated on six open-source video LMMs (e.g., LLaVA-OneVision, InternVL-Chat, VideoLLaMA3) and compared against baselines and closed-source models (GPT-4o).

• Quality Justification (S2I-Bench):

  • Models tuned on S2I showed significant improvements in VCG scores.
  • LLaVA-OneVision improved its total VCG score from 7.96 to 9.88 (+1.92).
  • Notable gains were observed in Temporal Understanding (TU) and Correctness of Information (CI), indicating better handling of video-specific issues and factual accuracy.
  • S2I-tuned models outperformed closed-source models like GPT-4o and Gemini-1.5 Pro in justification tasks.

• Quality Scoring (SRCC/PLCC):

  • On five standard in-the-wild VQA datasets (Maxwell, LSVQ, KoNViD-1k, LIVE-VQC), S2I-tuned models achieved state-of-the-art performance.
  • LLaVA-OneVision saw SRCC improvements from ~0.47 to 0.795 on the Maxwell dataset.
  • Significant gains were observed in cross-dataset scenarios, demonstrating strong generalization capabilities despite distribution shifts.

• Ablation Studies:

  • Data Source: Combining labeled and unlabeled data yielded the best results; unlabeled data boosted scoring, while labeled data boosted justification.
  • CoT Design: The hierarchical CoT and high-level caption refinement were critical for improving justification quality (Contextual Understanding and Detail Orientation).
  • Progressive Tuning: Removing Stage I (pretraining) caused a significant drop in scoring accuracy, validating the necessity of the two-stage approach.

5. Significance

• Paradigm Shift: The paper moves VQA from a purely numerical regression task to a generative, explainable task powered by LMMs.
• Scalability: By automating the data generation process via dimension scoring, the authors break the "annotation bottleneck," making it feasible to train models on millions of videos without human intervention.
• Comprehensive Assessment: The framework addresses the gap between "scoring" (how good is it?) and "justification" (why is it good/bad?), providing a more useful tool for video enhancement, transcoding, and content moderation.
• Future Direction: The work suggests that automated, dimension-aware scoring can serve as a foundation for training versatile, human-aligned quality assessors, potentially extending to other modalities like audio or 3D content.

In summary, Score2Instruct demonstrates that automated, dimension-based scoring can effectively scale up instruction data, enabling video LMMs to achieve superior performance in both predicting video quality scores and providing human-readable, detailed justifications.