LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models

Technical Summary: LongVQUBench

Problem Statement
The evaluation of long-term video quality understanding (LVQU) remains an open challenge for Large Vision-Language Models (LVLMs). While recent LVLMs have demonstrated strong capabilities in semantic video understanding and short-term reasoning, they lack robust mechanisms for assessing perceptual fidelity, temporal coherence, and cumulative degradation over extended durations. Existing video quality benchmarks (e.g., LIVE-VQA, KoNViD-1k) predominantly focus on short clips with isolated distortions, failing to capture the temporal continuity and contextual dynamics inherent in long-form content. Conversely, long-video benchmarks (e.g., LongVideoBench, Video-MME) prioritize semantic event comprehension over perceptual quality. Consequently, there is a lack of a unified framework to evaluate how LVLMs reason about perceptual quality across minute-to-hour video spans.

Methodology: LongVQUBench
To address these gaps, the authors introduce LongVQUBench, a comprehensive benchmark designed to evaluate the LVQU capabilities of LVLMs.

• Dataset Composition: The benchmark comprises over 1,200 diverse videos sourced from movies, documentaries, surveillance footage, egocentric recordings, and animated content. Video durations range from a few minutes to nearly two hours. The dataset includes 1,500 multiple-choice and open-ended questions.
• Distortion Strategy: To ensure controlled evaluation, the benchmark utilizes a "needle distortion" approach. High-quality source videos are segmented, and specific spatial (e.g., blur, noise) and temporal (e.g., flicker, frame drops) distortions are sparsely inserted at varying intensities. This allows for precise ground-truth alignment while simulating realistic degradation patterns.
• Hierarchical Evaluation Framework: The benchmark defines three progressively complex evaluation levels to assess reasoning across different temporal scopes:
  1. Local Event Quality Understanding (LQU): Focuses on detecting, classifying, and localizing single, temporally bounded distortion events (e.g., a 5–20 second blur). Questions cover detection, localization, classification, severity assessment, and open reasoning.
  2. Cross-Event Quality Reasoning (CQR): Evaluates the model's ability to compare, associate, and integrate multiple distortion events distributed across the video. This level tests reasoning about relative severity, cumulative effects, and temporal relationships between distinct degraded segments.
  3. Global Quality Understanding (GQU): Assesses holistic perceptual judgment over the entire video duration. It requires reasoning about temporal trends, stability, dominant degradation factors, and the evolution of quality over time.
• Needle Distortion Question-Answering (NDQA): Embedded across all three levels, this paradigm introduces sparse artifacts to probe fine-grained perceptual sensitivity. It utilizes both Multiple-Choice Questions (MCQ) for objective recognition and open-ended questions for explanatory reasoning.

Experimental Setup
The authors evaluated 14 state-of-the-art LVLMs, including proprietary models (GPT-5, Gemini-3, Qwen-VL-Max), open-source models (LLaVA-NeXT-Video, Qwen3-VL, VQA2, etc.), and agentic LVLMs (DeepVideoDiscovery, VideoAgent). Experiments were conducted under zero-shot settings using a frame-sampling inference pipeline. A stratified 40% validation / 60% test split was used, with validation determining the optimal frame sampling budget.

Key Results
Extensive experiments reveal significant limitations in current LVLMs regarding long-term video quality understanding:

• Performance Degradation with Complexity: There is a consistent decline in performance as the reasoning depth increases from LQU to CQR and finally to GQU. For instance, GPT-5 achieved 81.2% on CQR but dropped to 65.8% on GQU.
• Limited Benefit from Increased Sampling: Increasing the number of sampled frames yields diminishing returns. While some models (e.g., GPT-5) benefit moderately from higher frame counts, others (e.g., VQA2) perform best with very few frames (8 frames), suggesting that simply increasing temporal coverage does not solve the reasoning bottleneck.
• Proprietary vs. Open-Source: Proprietary models generally dominate the leaderboard, with GPT-5 achieving the highest overall accuracy (69.5%). However, even leading models struggle with global quality reasoning.
• Agentic Models: Among agentic approaches, DeepVideoDiscovery showed competitive performance (66.0% overall), approaching proprietary models, while simpler agents lagged significantly. This suggests that effective frame exploration and reasoning strategies are critical for agent-based quality understanding.
• Open-Ended Reasoning: While relevance scores for open-ended answers were high (e.g., GPT-5: 88.9%), completeness scores remained low (e.g., GPT-5: 45.8%), indicating that models struggle to produce comprehensive explanations for quality assessments.

Significance and Claims
The authors position LongVQUBench as a foundational step toward the systematic, hierarchical, and explainable evaluation of LVLMs' long-term video quality understanding. The paper claims that the benchmark establishes the first unified testbed integrating perceptual fidelity, temporal coherence, and multimodal reasoning for long-form videos. The results highlight that current LVLMs have limited capacity for long-range temporal integration and perceptual attribution, identifying long-term perceptual reasoning as a critical open challenge for future research. The work aims to advance perceptual modeling and temporal reasoning toward human-level long-form video comprehension.