💻 computer science

QPT V2: Masked Image Modeling Advances Visual Scoring

This paper introduces QPT V2, a novel masked image modeling pretraining framework that unifies quality and aesthetics assessment by curating specific data, introducing degradations, and modifying the model structure to achieve state-of-the-art performance across 11 downstream benchmarks.

Original authors: Qizhi Xie, Kun Yuan, Yunpeng Qu, Mingda Wu, Ming Sun, Chao Zhou, Jihong Zhu

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qizhi Xie, Kun Yuan, Yunpeng Qu, Mingda Wu, Ming Sun, Chao Zhou, Jihong Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge on a talent show. Your job is to look at a performance and give it a score based on two things: Technical Quality (is the singer off-key? is the video blurry?) and Aesthetic Beauty (is the lighting dramatic? is the outfit stylish?).

For a long time, teaching computers to do this has been like trying to teach a child to judge art by showing them only a handful of pictures. The computer gets confused because it hasn't seen enough examples of "bad lighting" or "beautiful composition." It needs a massive library of examples to learn the difference.

This paper introduces QPT V2, a new way to teach computers to be expert judges. Instead of just showing them pictures and asking "How good is this?", the authors use a clever game called "Masked Image Modeling" (MIM).

Here is how QPT V2 works, explained through simple analogies:

1. The Game: "Cover and Reconstruct"

Imagine you take a beautiful photo and cover 75% of it with a giant sheet of black paper. You then ask the computer: "Based on the tiny 25% of the photo you can see, please draw the rest of the picture."

To do this well, the computer can't just guess random colors. It has to understand:

The Big Picture (Semantics): If it sees a dog's ear, it knows the rest of the head must be there.
The Tiny Details (Texture): If it sees a blurry patch, it knows the original might have been sharp, or maybe the blur is part of the style.

This "fill-in-the-blanks" game forces the computer to learn deeply about both the meaning of the image and the quality of its pixels.

2. The Three Secret Ingredients

The authors realized that the standard version of this game wasn't perfect for judging quality. So, they upgraded it with three specific improvements:

A. The Training Data: "High-Definition & High-Action"

The Problem: Standard training data (like ImageNet) is often low-resolution or has too much empty background (like a photo of a person where 90% is just sky).
The Fix: The authors curated a special dataset of High-Resolution (HR) and High Foreground Coverage (HFC) images.
The Analogy: Imagine training a chef. Instead of giving them blurry photos of food with huge empty plates, you give them 4K close-ups of the food where the plate is full. The computer learns to appreciate fine details (like the texture of a steak) because it's forced to look at them.

B. The Distortions: "The Stress Test"

The Problem: Real-world photos aren't perfect. They get blurry, noisy, or have weird colors due to bad cameras or internet compression. Standard training often ignores these "flaws."
The Fix: Before the computer tries to reconstruct the image, the authors intentionally "ruin" it in specific ways: adding noise, blurring it, changing the colors, or shifting the perspective.
The Analogy: Think of a stress test for a bridge. You don't just check if it holds weight; you shake it, spray water on it, and hit it with wind. By training the computer to fix images that have been "shaken" and "sprayed" with digital noise, it learns to recognize what a "good" image looks like even when it's been damaged. They found that messing with Color Spaces (changing how colors are mathematically stored) was the most effective way to teach the computer about aesthetics.

C. The Brain Structure: "Multi-Scale Vision"

The Problem: Humans judge quality in layers. We look at the whole scene (is the composition balanced?) and then zoom in (is the focus sharp?). Standard computer models often struggle to do both at once.
The Fix: They built a "Multi-Scale" model. It's like giving the computer a pair of binoculars and a magnifying glass simultaneously.
The Analogy: A regular model might look at a painting from 10 feet away. QPT V2 looks at it from 10 feet away and from 1 inch away, then combines those two views to make a final decision. This helps it catch both the big artistic vibe and the tiny pixel errors.

3. The Results: The New Champion

After playing this "Cover and Reconstruct" game with their special high-quality data and stress tests, the computer became an expert judge.

When they tested QPT V2 on 11 different benchmarks (ranging from checking video quality to rating photo aesthetics), it beat almost every other existing method.

It became better at spotting real-world flaws (like a shaky video or a grainy photo).
It became better at rating beauty (like a well-composed sunset).
It did all this without needing millions of human-labeled scores, which are expensive and hard to get.

The Bottom Line

QPT V2 is like a new training camp for AI judges. Instead of just memorizing answers, the AI plays a challenging reconstruction game using high-definition, "damaged" images. This teaches it to understand both the technical perfection and the artistic beauty of visual content, making it the new state-of-the-art for evaluating images and videos.

1. Problem Statement

Visual Scoring (VS) encompasses Image Quality Assessment (IQA), Video Quality Assessment (VQA), and Image Aesthetics Assessment (IAA). The core objective is to mimic the Human Visual System (HVS) to generate scores aligned with human perception.

The Challenge: Current learning-based methods suffer significantly from the scarcity of labeled data. Collecting Mean Opinion Scores (MOS) via subjective studies is expensive and time-consuming, resulting in VS datasets that are often 1/10th to 1/100th the size of standard classification datasets.
Limitations of Existing Solutions:
- Data Augmentation/Mixed Training: Often fails because local patch scores differ from global content scores, and subjective differences exist across datasets.
- Contrastive Learning (CL): Current self-supervised pretraining (e.g., MoCo, SimCLR) relies on "sample-level" discrimination (grouping similar samples). This is insufficient for capturing local distortions and fine-grained visual attributes crucial for quality and aesthetics.
- Masked Image Modeling (MIM): While MIM (e.g., MAE, BEiT) excels at pixel-level reconstruction and learning both semantics and textures, it has not been effectively adapted for VS tasks. Standard MIM often uses low-resolution data (ImageNet) and simple augmentations, which lack the specific quality/aesthetic factors needed for VS.

2. Methodology: QPT V2

The authors propose QPT V2, the first pretraining framework based on MIM designed specifically to unify IQA, VQA, and IAA. The framework improves upon vanilla MIM through three targeted dimensions: Data, Degradation, and Model.

A. Data Curation: High Resolution & High Foreground Coverage (HR & HFC)

Rationale: Visual quality and aesthetics are heavily influenced by spatial complexity and foreground details. Standard datasets like ImageNet have low resolution and low foreground coverage (FC).
Implementation: The authors curated a pretraining dataset from SA-1B (Segment Anything 1 Billion).
- Selection Criteria: Images must have High Resolution (~1600×2100) and High Foreground Coverage (containing 50+ objects).
- Result: A dataset of 1.28 million images that forces the model to reconstruct rich textures and intricate structures, enhancing sensitivity to fine-grained details.

B. Degradation Strategy: Quality- and Aesthetics-Aware

Rationale: To learn robust representations, the model must perceive distortions found in real-world scenarios (compression, transmission, shooting errors).
Implementation: Instead of simple random cropping, the authors apply a comprehensive set of degradations to the reconstruction target (the masked area):
- Geometry: Resizing.
- Frequency: Blurring, Sharpening, Gaussian Noise.
- Color: Color Jittering and Color Space Transformation (CST).
Key Finding: The study revealed that CST (transforming between RGB, LAB, HSV, Grayscale) is the most effective degradation for VS, as it exposes the model to complementary information across different color spaces. Interestingly, complex sequential or "advanced" compositions of degradations did not yield better results than single, strong degradations like CST.

C. Model Architecture: Multi-Scale Feature Fusion

Rationale: HVS assesses quality at multiple scales. Standard MIM often uses plain ViT backbones which may miss hierarchical features.
Implementation:
- Encoder: Uses HiViT (a hierarchical Vision Transformer) instead of a plain ViT to leverage inductive biases for multi-scale learning.
- Feature Fusion: A pretrain-only fusion module is introduced. It adaptively fuses features from different stages (shallow to deep) of the encoder before feeding them to the decoder.
- Mechanism: Features $X = \{x_i\}$ from $N$ stages are projected to a shared space and fused via a weighted pooling mechanism. This allows the model to capture both coarse semantics and fine-grained distortions simultaneously.

3. Key Contributions

First MIM-based Unified Framework: QPT V2 is the first work to validate that Masked Image Modeling can effectively unify downstream Visual Scoring tasks (IQA, VQA, IAA).
Systematic Decomposition: The authors decompose MIM into Data, Degradation, and Model, identifying specific gaps in vanilla MIM for VS and proposing targeted solutions (HR/HFC data, CST degradation, Multi-scale fusion).
State-of-the-Art Performance: The framework achieves SOTA results on 11 benchmarks without relying on text modalities or large multi-modal models (LMMs).
Data Efficiency: Demonstrates that QPT V2 can achieve parity with SOTA methods using only 60% of the finetuning data, proving the efficacy of the pretraining priors.

4. Experimental Results

The model was evaluated on 11 benchmarks across three tasks:

Image Quality Assessment (IQA):
- Tested on 3 synthetic (LIVE, TID2013, KADID) and 3 real-world (KonIQ10K, CLIVE, FLIVE) datasets.
- Result: QPT V2 outperformed previous SOTA methods (including contrastive learning-based QPT and Re-IQA). For example, it achieved +1.1% SRCC on TID2013 and +0.4% SRCC on FLIVE compared to the previous best.
Video Quality Assessment (VQA):
- Tested on 4 datasets (LSVQ, LIVE-VQC, KoNViD-1k).
- Result: Surpassed traditional methods and most deep learning approaches. It showed strong generalization in cross-dataset settings (e.g., +0.9% PLCC gain on LIVE-VQC).
Image Aesthetics Assessment (IAA):
- Tested on the AVA dataset.
- Result: Achieved 0.865 SRCC and 0.875 PLCC, significantly outperforming the previous SOTA (Q-Align, which uses Large Multi-modal Models) by +4.3% SRCC. Notably, QPT V2 achieved this without using text descriptions, relying solely on visual pretraining.

5. Significance and Impact

Paradigm Shift: The paper challenges the dominance of Contrastive Learning for visual scoring, demonstrating that Pixel-level reconstruction (MIM) is superior for capturing the local distortions and fine-grained details required for quality and aesthetics assessment.
Practical Utility: By providing a unified framework, QPT V2 reduces the need for task-specific architectures and massive labeled datasets, making high-performance visual scoring more accessible.
Insightful Findings: The discovery that Color Space Transformation (CST) is a critical degradation factor and that simple, high-quality degradations outperform complex compositions offers new directions for designing self-supervised objectives in computer vision.
Resource Efficiency: The model achieves superior performance with smaller model sizes and lower computational costs compared to LMM-based approaches, making it suitable for deployment in real-world applications like video transcoding and content moderation.