Technical Summary: GameScope

Problem Statement

The rapid expansion of video game streaming has created a critical need for robust Video Quality Assessment (VQA) models capable of operating consistently across diverse codecs and content types. While general-purpose VQA metrics exist, prior research indicates that gaming content possesses statistical characteristics distinct from standard video, rendering existing non-gaming metrics ill-suited. Furthermore, current gaming-specific datasets suffer from significant limitations: they often lack codec diversity (typically restricted to H.264), offer limited game genre variety, focus exclusively on either User-Generated Content (UGC) or mobile gaming, or contain insufficient sample sizes for training deep learning models. There is a lack of a comprehensive, large-scale dataset that bridges the gap between UGC and Professionally-Generated Content (PGC) while covering multiple modern codecs and providing granular perceptual attributes.

Methodology

To address these gaps, the authors constructed GameScope, a multi-attribute, multi-codec benchmark dataset, and conducted a large-scale subjective evaluation.

1. Dataset Construction:

• Source Material: The dataset comprises 424 pristine source clips derived from 74 distinct games, covering a wide range of genres (Action, RPG, Sports, etc.). The sources are split between UGC (sourced from YouTube under Creative Commons, featuring overlays and edits) and PGC (captured directly via PlayStation 5 hardware, representing raw rendering).
• Encoding: The source clips were encoded using three widely used codecs: H.264, H.265, and AV1.
• Bitrate Ladder: A custom bitrate-resolution ladder was applied to ensure perceptually distinct quality levels. Unlike standard recommendations, specific bitrates were manually determined to guarantee separation between quality tiers across resolutions ranging from 360p to 2160p.
• Scale: The final dataset contains 4,048 video samples.

2. Subjective Study:

• Platform: An online subjective study was conducted via Amazon Mechanical Turk (AMT).
• Protocol: Stimuli were organized into batches containing a single source sequence and its encoded variants to minimize content bias. Participants were restricted to PC/laptop form factors to ensure standardized viewing conditions.
• Data Collection: The study collected 150,874 ratings, averaging 37 Mean Opinion Scores (MOS) per video.
• Attributes: Beyond overall quality, participants evaluated coarse-grained perceptual attributes: Clarity (blur, motion artifacts), Pixelation/Blockiness, and Immersive Game Experience.
• Quality Control: Rigorous screening was applied, including intra-rater consistency checks (repeated videos), "golden video" validation against ground truth, and detection of insincere behaviors (e.g., straight-lining). Data from subjects with excessive buffering or inconsistent ratings were discarded.

3. Benchmarking:
The authors evaluated a suite of state-of-the-art VQA methods, including:

• Traditional NR-VQA models (TLVQM, RAPIQUE, VIDEVAL).
• Gaming-specific models (GAME-VQP, GAMIVAL).
• Deep learning-based models (VSFA, FAST-VQA, DOVER).
• Vision-Language Models (LLMs) and Large Multimodal Models (LMMs), specifically LLaVA-OneVision-1.5-4B and Qwen3-VL-4B, as well as reinforcement learning-based assessors (VisualQuality-R1, VQAThinker).

Key Contributions

1. GameScope Dataset: The authors present the largest gaming video quality dataset to date, featuring 4,048 samples with extensive diversity in game genres, resolutions, and content types (both UGC and PGC).
2. Multi-Codec Support: It is the first dataset to comprehensively cover H.264, H.265, and AV1 codecs within a single gaming context.
3. Granular Attributes: Unlike previous datasets focusing solely on scalar MOS, GameScope includes coarse-grained quality attributes (clarity, artifacts, immersion), enabling multidimensional perceptual analysis.
4. Comprehensive Benchmarking: The paper provides a rigorous benchmark of existing VQA algorithms and introduces the evaluation of Vision-Language Models for gaming VQA.

Results

• Subjective Data: The collected MOS distribution is well-spread across the quality domain, facilitating robust algorithm evaluation. Split-half analysis using the SUREAL model demonstrated high consistency (median PLCC and SROCC of 0.92).
• Model Performance:
  • Among traditional and deep learning-based VQA methods, GAMIVAL and VSFA achieved strong performance (PLCC ~0.85).
  • Vision-Language Models showed superior results. Specifically, Qwen3-VL-4B outperformed all other benchmarks with a PLCC of 0.910 and SROCC of 0.906 for MOS prediction.
  • Multi-task Capability: Qwen3-VL-4B and LLaVA-OneVision-4B demonstrated the unique capability to predict both overall MOS and text-based quality attributes in a single inference pass. Qwen3-VL-4B achieved 98% accuracy in predicting "Immersion," while LLaVA-OneVision-4B showed competitive performance in predicting "Artifacts" (54% accuracy).

Significance

The paper claims that GameScope is the first dataset to comprehensively address gaming video quality assessment across multiple codecs, content types (UGC/PGC), and resolutions, while incorporating quality attributes. By providing a large-scale, diverse, and publicly available resource, the authors aim to enable the development of more robust VQA algorithms that can generalize across the complex and varied landscape of modern game streaming. The results suggest that recent Vision-Language Models represent a significant advancement in this domain, potentially outperforming specialized gaming VQA metrics. The dataset is intended to serve as a foundational resource for future research in gaming VQA.