Technical Summary: Breaking Annotation Barriers via Ranking-based Self-Supervision

1. Problem Statement

Video Quality Assessment (VQA) is critical for optimizing the Quality of Experience (QoE) in modern video processing workflows. While supervised deep learning models have achieved significant progress, they suffer from a fundamental limitation: generalization. Current state-of-the-art (SOTA) models rely heavily on manually annotated datasets, which are labor-intensive, costly, and difficult to scale. Consequently, these models often exhibit poor performance on out-of-distribution (OOD) data, failing to generalize to unseen video content, distortions, or media formats (e.g., gaming videos, high-frame-rate content, or AI-generated content).

Existing self-supervised approaches attempt to mitigate annotation costs but typically lag behind supervised methods. They often rely on contrastive learning with synthetic proxy tasks (e.g., distortion classification) that fail to capture complex perceptual characteristics or real-world authentic distortion patterns.

2. Methodology

The authors propose LMM-PVQA, a self-supervised learning framework that reformulates VQA as a pairwise ranking problem rather than absolute regression. The core components of the methodology are:

A. Large-Scale Pairwise-Labeled Dataset (PLVD)

The authors constructed a massive dataset comprising 700,000 video pairs sampled from 3 million source videos across four major social media platforms (TikTok, YouTube, Youku, Bilibili). The dataset covers over 20 content categories and diverse distortion scenarios.

• Sampling Strategy: A mixed-integer programming approach was used to select 438k videos, ensuring the distribution of nine low-level metrics (blockiness, blur, contrast, noise, flickering, colorfulness, luminance, spatial information, and temporal information) matches mainstream UGC benchmarks.
• Annotation Strategies: Two methods were employed to generate relative quality labels without human intervention:
  1. Pseudo-labeling via VQA Models: Five SOTA VQA models (Minimalistic-VQA, FAST-VQA, DOVER, Q-Align) act as "judges" to score video pairs. An ensemble approach determines the relative quality. Labels are categorized into Hard Ranking (better/worse) and Soft Ranking (superior/better/similar/worse/inferior) based on statistical significance of score differences.
  2. Synthetic Distortion Simulations: A subset of data (PLVD-DS) is generated by applying controlled distortions (spatial: blur, noise, resizing; temporal: jitter, stuttering; streaming: H.264/H.265 compression) at varying severity levels. The severity levels serve as ground truth for relative ranking.

B. Model Architecture (LMM-PVQA)

The model is built upon a Large Multimodal Model (LMM), specifically Qwen-2, integrated with a dual-branch visual feature extractor:

• Spatial Branch: Uses a pre-trained image encoder (SigLIP) to process key frames.
• Temporal Branch: Uses a pre-trained motion encoder (SlowFast) to analyze frame sequences for motion-related distortions.
• Fusion: Visual features are projected into the language space and concatenated with text prompts. The model is trained via Supervised Fine-Tuning (SFT) to output a textual judgment (e.g., "The second video is better than the first").

C. Iterative Self-Improvement Strategy

A novel training pipeline is introduced where the model iteratively refines its own training data:

1. Stage 1: Train a base model on the initial 500k pairs (PLVD-Part1).
2. Stage 2: Use the trained Stage 1 model as an enhanced "judge" to re-annotate the next 100k pairs (PLVD-Part2), combining them with the original data to train Stage 2.
3. Stage 3: Repeat the process for the final 100k pairs (PLVD-Part3).
   This strategy allows the model to progressively distill knowledge and improve its own annotation quality, enhancing generalization.

D. Inference Mechanism

Since the model predicts relative rankings, an adaptive soft comparison method is used to derive absolute scores. The test video is compared against a set of five anchor videos (representing different quality intervals). The resulting probability distribution is converted to a calibrated quality score using Maximum A Posteriori (MAP) estimation under Thurstone's Case V model.

3. Key Contributions

1. Large-Scale Dataset: Construction of PLVD, a dataset with 700k video pairs featuring both authentic and synthetic distortions, annotated automatically via ensemble VQA models and distortion simulations.
2. Self-Supervised Framework: A novel framework that learns quality assessment through pairwise comparisons, eliminating the need for human-labeled regression targets during pre-training.
3. Iterative Self-Improvement: A training strategy where the model acts as an improved annotator for subsequent iterations, continuously refining its performance.
4. State-of-the-Art Performance: The model achieves strong zero-shot performance across diverse benchmarks and sets new records when fine-tuned on human-labeled data.

4. Experimental Results

The model was evaluated on ten benchmarks, categorized into in-domain (UGC videos) and out-of-distribution (OOD) datasets (gaming, high-frame-rate, 4K compression).

• Zero-Shot Performance: LMM-PVQA achieves zero-shot performance comparable to or surpassing existing supervised SOTA models (e.g., Minimalistic-VQA, FAST-VQA, DOVER) on in-domain benchmarks.
• Out-of-Distribution Generalization: The model demonstrates superior generalization on OOD datasets, achieving a 24.7% improvement in SRCC compared to competing approaches. Notable gains were observed on challenging datasets like LIVE-YT-HFR (high frame rate), Waterloo-IVC-4K (4K compression), and KVQ (enhanced UGC).
• Supervised Fine-Tuning: When fine-tuned on human-labeled datasets, the model outperforms all competing supervised baselines by 3.5% in SRCC.
• Ablation Studies:
  • Soft Ranking: Outperforms hard ranking by incorporating "quality distance" information.
  • Iterative Strategy: Confirmed that performance gains stem from the self-improvement loop, not just increased data volume.
  • Motion Encoder: Essential for temporal distortions, yielding a 33.5% SRCC improvement on the high-frame-rate benchmark (LIVE-YT-HFR).

5. Significance and Claims

The paper claims that by breaking the reliance on expensive human annotations and leveraging large-scale, automatically labeled video pairs, it is possible to train VQA models with superior generalization capabilities.

• Scalability: The approach addresses the "easy dataset" problem and the prohibitive cost of subjective experiments, enabling the use of millions of unlabeled web videos for training.
• Robustness: The iterative self-improvement strategy and the inclusion of synthetic distortions allow the model to adapt to unseen distortion types (e.g., specific compression artifacts or frame rate inconsistencies) that traditional supervised models fail to handle.
• Foundation for Future Research: The authors posit that this framework sets a new standard for generalized VQA, offering a path toward models that can assess quality across diverse, real-world scenarios without domain-specific retraining.

The authors acknowledge a limitation: the inference process requires comparing a video against multiple anchors, which increases inference time compared to direct regression models. Future work aims to explore more automated annotation strategies and extend the framework to multi-modal quality assessment (incorporating audio and text).