DPC-VQA: Decoupling Quality Perception and Residual Calibration for Video Quality Assessment

Technical Summary: DPC-VQA

Problem Statement
Multimodal Large Language Models (MLLMs) have demonstrated promising capabilities in Video Quality Assessment (VQA), yet their practical deployment faces significant hurdles. Adapting these large-scale models to new video domains or scoring protocols typically requires expensive end-to-end retraining and large-scale Mean Opinion Score (MOS) annotations, which are costly to collect due to the need for repeated human subjective evaluation. Existing MLLM-based VQA methods often struggle with cross-dataset adaptation, as they either require scenario-specific fine-tuning or fail to align their outputs with the specific MOS scales of target benchmarks. Furthermore, current few-shot VQA approaches often entangle representation learning with dataset-specific calibration, limiting their generalization efficiency.

Methodology: DPC-VQA
The authors propose DPC-VQA (Decoupling Perception and Calibration for Video Quality Assessment), a framework that separates the task of quality perception from score calibration. The core hypothesis is that a pretrained MLLM already encodes a useful generic perceptual prior; the challenge lies not in relearning quality perception, but in efficiently calibrating this prior to the target MOS space.

The framework consists of two main components:

1. Frozen Quality Perception (Base Prior):

   • A frozen MLLM (specifically Qwen3-VL-8B in the experiments) processes input video frames alongside a fixed quality prompt.
   • The model outputs a base quality score ($q_b$) derived from a "quality verbalizer" distribution (e.g., mapping tokens like "bad," "poor," "fair," "good," "excellent" to scalar anchors).
   • The model also outputs a confidence score ($u_b$) based on the entropy of the verbalizer distribution and the last-layer visual hidden states ($H_{vis}$).
   • Crucially, the MLLM parameters remain frozen during training, preserving the pre-trained perceptual knowledge.

2. Lightweight Residual Calibration:

   • A trainable branch predicts a residual correction ($\Delta$) to adjust the base score to the target MOS space.
   • Input Features: The calibration branch utilizes the frozen visual tokens ($H_{vis}$), auxiliary video features ($H_{aux}$) extracted from a separate pretrained encoder (SlowFast), the base score ($q_b$), and the confidence score ($u_b$).
   • Mechanism:
     • Features are projected into a shared latent space and concatenated into an augmented token bank.
     • Learnable calibration queries perform cross-attention on this token bank to extract calibration-aware tokens.
     • These tokens are modulated by the base condition ($q_b, u_b$) via feature-wise rescaling and shifting.
     • Each modulated token predicts a local residual proposal (magnitude and direction), which are then aggregated via importance weighting to produce the final residual $\Delta$.
   • Final Prediction: The predicted score is $\hat{y} = q_b + \Delta$.

Training Objective
Training is restricted to the lightweight calibration branch. The objective function combines:

• Smooth L1 Loss: Supervising the final prediction against ground-truth MOS.
• Residual Regularization: A term penalizing the magnitude of the residual ($\sum |\Delta|$) to ensure the correction remains a bounded adjustment rather than a complete re-estimation, stabilizing training and preserving the prior.

Key Contributions

1. Decoupled Framework: The proposal of DPC-VQA, which decouples quality perception (frozen MLLM) from score calibration (lightweight branch), avoiding costly end-to-end retraining.
2. Residual Calibration Mechanism: A novel token-wise bounded residual estimation and adaptive aggregation strategy that conditions corrections on the base score and confidence, enabling fine-grained adaptation.
3. Efficiency: The method uses less than 2% of the trainable parameters compared to full fine-tuning of MLLM-based VQA methods and remains effective with as little as 20% of MOS labels.

Experimental Results
The authors evaluated DPC-VQA on five benchmarks: three User-Generated Content (UGC) datasets (KoNViD-1k, YouTube-UGC, LIVE-VQC) and two AI-Generated Content (AIGC) datasets (T2VQA, Human-AGVQA).

• Few-Shot Performance: Under a 5-fold few-shot split (using only 20% of data for training), DPC-VQA achieved the best performance among all few-shot baselines across all five datasets.
  • On UGC datasets, it outperformed the best few-shot baseline by SRCC gains of 0.032, 0.036, and 0.026.
  • On AIGC datasets, it showed significant improvements over few-shot baselines (SRCC gains of 0.106 and 0.114), remaining competitive with dedicated supervised AIGC methods despite using minimal labeled data.
• Ablation Studies:
  • Residual vs. Direct Regression: The residual formulation outperformed direct regression and score-conditioned regression, confirming that learning a correction is more effective than relearning the score.
  • Information Sources: Combining frozen MLLM tokens, auxiliary features, base score, and confidence yielded the best results, with the frozen MLLM serving as the primary perceptual basis.
  • Query Aggregation: Learnable calibration queries outperformed mean pooling and self-attention aggregation.
• Data Efficiency: The method maintained superior performance even with training ratios as low as 10–20%, demonstrating strong robustness in low-data regimes.
• Parameter Efficiency: The trainable parameter count was approximately 155M (1.9% of the full 8.2B model), significantly reducing optimization costs.

Significance and Claims
The paper claims that the primary bottleneck in adapting MLLMs to VQA is not the lack of perceptual capability in the pretrained model, but the difficulty of aligning its output with specific target scoring protocols. By treating the MLLM as a fixed perceptual prior and formulating adaptation as a lightweight residual calibration task, DPC-VQA offers a more practical and scalable path for VQA deployment. The authors argue that this approach shifts the focus from scenario-specific relearning to efficient calibration, making high-quality VQA feasible in scenarios where MOS annotations are scarce or expensive. The results suggest that future VQA systems can benefit from leveraging frozen perceptual priors rather than relying solely on end-to-end training for every new domain.