Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videos

This paper addresses the limitations of existing Standard Dynamic Range (SDR) models in assessing High Dynamic Range (HDR) user-generated content by introducing "Beyond8Bits," a large-scale subjective dataset, and "HDR-Q," a state-of-the-art Multimodal Large Language Model equipped with an HDR-aware vision encoder and a novel reinforcement learning framework to achieve superior quality assessment.

The Gist

Imagine you just bought a brand-new, ultra-high-definition TV that can show colors so vivid and lights so bright they feel almost real. This is HDR (High Dynamic Range) video. It's like upgrading from a black-and-white radio to a surround-sound concert hall.

But here's the problem: Most of the "quality inspectors" (software that judges if a video looks good) were trained on old, standard TVs (SDR). They are like a music critic who only knows how to judge a piano; when you play a full orchestra, they get confused. They miss the deep shadows, the blinding highlights, and the subtle color shifts that make HDR special.

This paper, "Seeing Beyond 8bits," is like a team of engineers building a brand-new inspector specifically for this new, fancy world. Here is how they did it, broken down into simple parts:

1. The Problem: The "Blind" Inspector

Old video quality tools are "blind" to the magic of HDR.

• The Analogy: Imagine trying to judge the quality of a spicy curry using a thermometer that only measures temperature, not heat. You might think the curry is fine because it's warm, but you miss the fact that it's actually burning your tongue!
• The Reality: HDR videos have more colors and brighter lights. Old tools miss specific errors like "crushed blacks" (shadows turning into solid black blobs) or "clipped highlights" (bright lights turning into white blobs). They also struggle with videos made by regular people (UGC) on phones, which often have weird compression glitches.

2. The Solution Part A: The "Super-Database" (Beyond8Bits)

To teach a computer how to judge HDR, you need to show it thousands of examples of what humans actually think looks good.

• What they did: The researchers created Beyond8Bits, a massive library of about 44,000 videos taken by regular people on phones and cameras.
• The Scale: They got over 1.5 million human ratings (like asking 1.5 million people to rate a video from 0 to 100).
• The Metaphor: It's like opening a massive "Taste Test" competition. Instead of just asking a few food critics, they asked a whole city to taste every dish and rate it. This gives them a perfect map of what humans actually prefer.

3. The Solution Part B: The "Smart Detective" (HDR-Q)

They built a new AI model called HDR-Q. Think of this AI as a detective who doesn't just look at the clues; it understands the context.

• The Vision Encoder (The Eyes): They gave the AI special "HDR glasses." Instead of squashing the bright and dark parts of the image (like old cameras do), these glasses preserve the full range of light and color. It allows the AI to see the difference between a dark shadow and a black screen.
• The Brain (HAPO): This is the secret sauce. The AI uses a technique called HAPO (HDR-Aware Policy Optimization).
  • The Analogy: Imagine a student taking a test.
    • Old AI: Reads the question and guesses the answer based on what it memorized from a textbook (ignoring the actual picture).
    • HDR-Q with HAPO: The teacher (the training system) says, "If you ignore the picture and just guess, you get a penalty. You must look at the picture to get the points."
  • How it works: The system forces the AI to compare the HDR video with a "dimmed down" version (SDR). If the AI gives the same answer for both, it gets punished. This forces the AI to actually look at the HDR details (like the glowing sun or the deep shadows) to make its decision.

4. The Result: A New Standard

When they tested this new "Smart Detective" against all the old "Blind Inspectors":

• It won. It predicted human ratings much more accurately than anything else.
• It explains itself: Unlike old tools that just give a number (e.g., "Score: 85"), HDR-Q can tell you why. It might say, "The video is great, but the bright sun is slightly too white, and the shadows are a bit too dark."

Why This Matters

As more people upload 4K and HDR videos to YouTube, TikTok, and Instagram, we need a way to make sure they look good on our new fancy screens. This paper provides the data (the taste test) and the AI (the smart detective) to ensure that the videos we watch look as stunning as the creators intended, without the weird glitches that ruin the experience.

In short: They built a massive library of human opinions and taught a super-smart AI to use those opinions to judge video quality, specifically for the bright, colorful, high-definition world we are living in today.

Technical Summary

1. Problem Statement

The proliferation of High Dynamic Range (HDR) user-generated content (UGC) on platforms like YouTube and TikTok has outpaced the capabilities of existing Video Quality Assessment (VQA) systems.

• Limitations of Current Models: Most VQA models are trained on Standard Dynamic Range (SDR) data or professionally generated HDR content. They fail to generalize to the heterogeneous capture conditions, device variations, and uncontrolled distortions typical of real-world HDR-UGC.
• Specific HDR Artifacts: HDR's extended luminance range, wide color gamut, and higher bit depth expose unique distortions that are less visible in SDR, such as near-black crushing, highlight clipping, color banding, and exposure flicker. These are often compounded by compression artifacts common in UGC.
• Data Scarcity: Existing subjective HDR datasets are small, limited in scope, or focus on synthetic distortions, lacking the scale and diversity required to train models that align with human perceptual judgments for real-world HDR-UGC.
• MLLM Challenges: While Multimodal Large Language Models (MLLMs) show promise for explainable VQA, they suffer from modality neglect (relying on text priors rather than visual HDR cues) and struggle with fine-grained regression (predicting continuous Mean Opinion Scores, or MOS) within a next-token prediction paradigm.

2. Key Contributions

The authors introduce two primary contributions to address these gaps:

1. Beyond8Bits Dataset: The first large-scale, crowdsourced subjective dataset for HDR-UGC.
   • Scale: Contains ~44,000 video clips from 6,861 diverse sources.
   • Ratings: Over 1.5 million human ratings collected via Amazon Mechanical Turk (AMT) using verified HDR-capable devices.
   • Diversity: Covers human-centric, nature, and low-light scenes with varied capture devices (iPhone, Pixel, Galaxy) and compression settings (simulating real-world streaming bitrates).
2. HDR-Q Model: The first MLLM specifically designed for HDR-UGC quality assessment.
   • It integrates a novel HDR-aware vision encoder and a reinforcement learning framework called HDR-Aware Policy Optimization (HAPO).

3. Methodology

A. Beyond8Bits Dataset Construction

• Data Collection: Combined 2,253 user-contributed HDR clips (captured on consumer devices) and 4,608 public Creative Commons HDR videos.
• Processing: Videos were verified for HDR metadata (PQ transfer, 10-bit HEVC, BT.2020), trimmed to 10 seconds, and transcoded into a bitrate ladder (0.2–5 Mbps) to simulate streaming conditions while retaining full HDR signaling.
• Annotation: A rigorous subjective study was conducted where workers with verified HDR displays rated clips on a 0–100 scale. The SUREAL method was used to aggregate ratings into reliable MOS values, accounting for subject bias and inconsistency.

B. HDR-Q Architecture

HDR-Q is built upon the Ovis2.5 MLLM backbone and consists of two main innovations:

1. HDR-Aware Vision Encoder

• Input: Processes 10-bit HDR video frames in full precision (PQ/BT.2020) without tone compression to preserve near-black structure and highlight dynamics.
• Dual-Domain Supervision: To prevent the encoder from collapsing into SDR-like representations, the model is trained with a contrastive loss. It learns to distinguish between an HDR frame ($x_t$) and its tone-mapped SDR counterpart ($x^{SDR}_t$) while maintaining semantic alignment with captions generated by a large language model.
• Goal: Produce embeddings that are semantically faithful but intrinsically sensitive to HDR-specific luminance and color cues.

2. HDR-Aware Policy Optimization (HAPO)
HAPO is a Reinforcement Learning (RL) fine-tuning framework based on Group Relative Policy Optimization (GRPO), enhanced with three specific mechanisms to enforce HDR grounding:

• HDR–SDR Contrastive KL: Maximizes the Kullback-Leibler (KL) divergence between the policy's output when given HDR inputs versus SDR-only inputs. This forces the model to rely on HDR visual information rather than textual priors, preventing modality neglect.
• Dual-Entropy Regularization: Prevents the policy from trivially satisfying the contrastive objective by generating overly uncertain (high-entropy) outputs. It regularizes entropy on both HDR and SDR pathways to ensure sharp, grounded distributions.
• High-Entropy Weighting (HEW): Instead of treating all tokens equally, HEW amplifies the learning signal for tokens with high entropy (typically those identifying specific HDR distortions like banding or clipping). This directs the policy to focus on the most informative reasoning steps.
• Reward Structure: The model is optimized using a composite reward including format accuracy, Gaussian-weighted regression (for precise MOS calibration), and self-consistency rewards.

4. Experimental Results

Performance on Beyond8Bits:

• HDR-Q achieved State-of-the-Art (SOTA) performance, significantly outperforming traditional DL models (e.g., CONTRIQUE, FastVQA), existing HDR-specific models (e.g., HDR-ChipQA, HIDRO-VQA), and general MLLM baselines.
• Metrics: Achieved a Spearman Rank Correlation (SRCC) of 0.9206 and Pearson Linear Correlation (PLCC) of 0.9118, with a low RMSE of 5.1594.
• Comparison: It notably outperformed the base MLLM (Ovis2.5) and other VQA models, demonstrating the necessity of the HDR-aware encoder and HAPO.

Generalization (Zero-Shot Transfer):

• Tested on public benchmarks LIVE-HDR and SFV+HDR without retraining.
• HDR-Q maintained high correlation and low error, proving its ability to generalize across different UGC and professionally generated HDR (PGC) domains.

Ablation Studies:

• Removing the HDR encoder caused a marked drop in performance, confirming the importance of 10-bit cues.
• Removing the HDR–SDR Contrastive KL led to modality neglect (the model ignored visual HDR data).
• Disabling entropy regularization resulted in unstable, verbose reasoning.
• HEW improved token-level credit assignment, leading to more precise MOS predictions.

5. Significance and Impact

• Bridging the Gap: This work addresses the critical lag between HDR content adoption and quality assessment tools, providing the first large-scale dataset and model tailored for real-world HDR-UGC.
• Methodological Advance: The introduction of HAPO demonstrates a novel approach to grounding MLLMs in specific perceptual modalities (HDR) using contrastive RL and entropy-weighted reasoning, which can be applied to other specialized vision tasks.
• Practical Utility: By accurately predicting MOS and generating interpretable, HDR-grounded rationales (e.g., identifying "highlight clipping" or "color blooming"), HDR-Q offers a tool for content platforms to automate quality control and optimize streaming pipelines for high-fidelity HDR delivery.
• Open Science: The release of the Beyond8Bits dataset and the HDR-Q model is expected to catalyze further research in HDR perception, evaluation, and generative model alignment.