Scaling Audio-Visual Quality Assessment Dataset via Crowdsourcing

This paper addresses the limitations of existing small-scale audio-visual quality assessment datasets by proposing a practical crowdsourcing framework and systematic data preparation strategy to construct YT-NTU-AVQ, the largest and most diverse AVQA dataset to date, which features 1,620 user-generated sequences with rich annotations for multimodal perception research.

The Gist

Imagine you are a chef trying to create the perfect recipe for a "Quality Taste Test." You want to know exactly how humans judge the flavor of a dish when they are eating it and listening to the sizzle of the pan at the same time. This is what Audio-Visual Quality Assessment (AVQA) is all about: figuring out how people rate videos when both the picture and the sound matter.

For a long time, this research was stuck in a tiny, expensive kitchen (a university lab). Only a few people could come in, sit in perfect silence, and taste the food. The result? The "recipes" (datasets) were too small, too boring, and didn't taste like real life.

This paper is like a bold new chef who decides to open the kitchen to the whole world using a massive online crowd, while still making sure the food tastes good. Here is how they did it, explained simply:

1. The Problem: The "Tiny Lab" Trap

Before this, researchers could only test a few hundred videos in a controlled lab. It was like trying to learn how the whole world eats by only asking your immediate family.

• The Issue: The videos were fake or too perfect. Real life is messy (bad microphones, shaky cameras, loud neighbors).
• The Result: The computer models trained on this data were like students who only studied for a test in a quiet library; they failed when they went out into the noisy cafeteria of the real internet.

2. The Solution: The "Global Taste Test" (Crowdsourcing)

The authors decided to ask thousands of regular people on the internet (via Amazon Mechanical Turk) to rate videos. But they knew that asking random people on the internet is risky. Some might be distracted, some might have bad headphones, and some might just click "5 stars" for everything.

To fix this, they built a Smart Guard System with three layers of protection:

• Layer 1: The "Check-In" (Environment Control): Before you can taste, you have to prove you're in a quiet room, using a real computer (not a phone), and wearing headphones. It's like a bouncer checking your ID and making sure you aren't wearing noise-canceling earplugs that block the music.
• Layer 2: The "Training Wheels" (Qualification): You can't just jump into the main event. You have to take a practice test. If your answers are too random or weird compared to the group, you get kicked out. It's like a cooking contest where you have to chop an onion perfectly before you're allowed to cook the main dish.
• Layer 3: The "Double-Check" (Data Filtering): Even after people rate the videos, the system looks at the results. If someone gave every video a "3" or rated them in a weird pattern, their answers are thrown out. They only keep the answers that make sense.

3. The Result: The "Giant Library" (YT-NTU-AVQ)

Thanks to this system, they created the YT-NTU-AVQ dataset.

• Size: It has 1,620 videos. That's huge compared to the old datasets which had maybe 50 or 100.
• Variety: These aren't fake movies. They are real YouTube videos (Creative Commons) featuring everything from dancing cats to cooking shows and music performances.
• Depth: Instead of just asking "How good is this?", they asked four questions:
  1. How good is the whole experience?
  2. How good is just the picture?
  3. How good is just the sound?
  4. The Secret Sauce: "Which one did you care about more?" (Did you focus on the singer's voice or the stage lights?)

4. The Big Discovery: "The Eyes Have It"

When they analyzed the data, they found something surprising.

• The Finding: Even though people said they were listening to the sound, their final score was almost entirely driven by how good the video looked.
• The Analogy: Imagine you are eating a delicious meal, but the plate is chipped and ugly. You might say, "The food tastes great," but you'll still give the whole experience a lower score because the plate ruined the vibe. In these videos, if the picture was blurry, people hated the video, even if the music was perfect. If the picture was great, they forgave a slightly bad sound.
• The Exception: If the sound was terrible (like a loud screech), people finally noticed. But for normal videos, the eyes are the boss.

Why This Matters

This paper is like building a giant, realistic simulator for AI.

• Before: AI models were trained on tiny, fake datasets. They were like drivers who only learned to drive in an empty parking lot.
• Now: They have a dataset that looks and sounds like the real, messy internet. This helps build AI that can actually understand how humans feel when watching a video on their phone in a noisy coffee shop.

In short: They figured out how to ask thousands of people to judge video quality without losing their minds, built a massive library of real-world videos, and discovered that when it comes to video, what you see is usually more important than what you hear.

Technical Summary

1. Problem Statement

Audio-Visual Quality Assessment (AVQA) research is currently hindered by significant limitations in existing datasets:

• Scale and Diversity: Existing datasets are small-scale, lack diversity in content (semantic scenarios) and quality distributions, and often rely on synthetic distortions rather than real-world User-Generated Content (UGC).
• Annotation Limitations: Most datasets provide only a single overall quality score, failing to capture the complex interplay between audio and visual modalities or the specific contributions of each modality to the final perception.
• Experimental Constraints: Traditional AVQA experiments require controlled laboratory settings with high-end equipment and trained subjects, making large-scale data collection expensive and unscalable.
• Crowdsourcing Challenges: While crowdsourcing offers scalability, it introduces noise due to uncontrolled environments (varying devices, network conditions) and inconsistent subject behavior, which is particularly problematic for AVQA where audio and video must be judged jointly.

2. Methodology

The authors propose a comprehensive framework to construct a large-scale, diverse AVQA dataset (YT-NTU-AVQ) using a rigorous crowdsourcing approach. The methodology consists of three main pillars:

A. Crowdsourced Subjective Experiment Framework

To overcome the lack of lab control, the authors designed a custom platform (based on jsPsych) with specific measures to ensure data reliability:

• Environment Control: Subjects must pass a pre-experiment check confirming a quiet environment, desktop usage, headphone usage, and stable network. Automatic checks verify screen resolution (>720p) and audio output availability.
• Multi-dimensional Rating: Instead of a single score, subjects provide four ratings per video:
  1. AVQA Score: Overall audio-visual quality.
  2. AV_VQA Score: Video-only quality.
  3. AV_AQA Score: Audio-only quality.
  4. Attention Weight: A 0–100% split indicating the relative attention paid to audio vs. video.
• Multi-Stage Subject Screening: A three-stage process ensures only reliable subjects participate in the main data collection:
  1. Pretest: Establishes reference scores and filters initial subjects.
  2. Qualification Test: New subjects rate a subset of pretest videos; only those passing dynamic filtering criteria proceed.
  3. Formal Test: Qualified subjects rate the remaining 1,500 videos.

B. Dynamic Data Filtering

To address unreliable submissions (random or careless ratings), the authors propose a Ranking-based Filtering method:

• Consistency Check: Calculates the Spearman Rank-Order Correlation Coefficient (SROCC) between a subject's ratings and the group consensus (Mean Opinion Score).
• Dispersion Check: Calculates the Standard Deviation (STD) of the subject's ratings to ensure they are not overly uniform or concentrated.
• Thresholds: Submissions are retained only if both SROCC > 0.5 and STD > 0.5.

C. Stratified Data Preparation Strategy

To ensure diversity in content and quality:

• Source Pool: Leveraged the VALOR dataset (1M+ YouTube sequences) and manually added recent (post-2017) Creative Commons licensed videos.
• Stratified Sampling: Used pseudo-labels from AudioBox (for audio quality) and FasterVQA (for video quality), combined with AudioSet semantic labels (Speech, Music, Sound).
• Balancing: Applied a soft-balancing algorithm to sample 10,000 candidates, ensuring coverage across quality levels and semantic combinations, followed by manual selection to reach the final 1,620 clips.
• Rich Annotations: Used Gemini 2.5 to generate video summaries and verify audio/video categories for all sequences.

3. Key Contributions

1. YT-NTU-AVQ Dataset: The largest and most diverse AVQA dataset to date, containing 1,620 user-generated audio-video sequences. It features multi-dimensional labels (overall, audio-only, video-only, and attention weights) rather than just a single score.
2. Scalable Crowdsourcing Framework: The first practical framework demonstrating that reliable AVQA can be achieved outside the lab through rigorous environment checks, multi-stage subject screening, and dynamic ranking-based filtering.
3. Multimodal Perception Insights: The dataset enables the study of how humans perceive audio-visual quality, specifically the relationship between modality-specific quality and overall QoE, and how attention shifts based on content.

4. Results and Analysis

• Data Quality: The multi-stage screening significantly improved data reliability. The acceptance rate for the Formal Test was 67.4%, with an average of 63.1 ratings per video. The SROCC between subject ratings and the consensus increased progressively from the Pretest to the Formal stage.
• Quality Distribution: The dataset exhibits a near-Gaussian distribution of quality scores, covering a wide range of both audio and video distortions.
• Modality Dominance:
  • High correlation was found between AVQA scores and Video-only (AV_VQA) scores (SROCC = 0.9938), suggesting that visual quality dominates AVQA judgments for UGC content.
  • However, when audio and video qualities differ, human attention shifts toward the degraded modality, yet the overall score remains anchored closer to the higher-quality modality.
• Baseline Performance:
  • Unimodal VQA models (e.g., Q-Align, FastVQA) achieved surprisingly high performance on this dataset (SROCC > 0.93), reinforcing the finding that video quality is the primary driver of AVQA scores in UGC.
  • Traditional fusion methods (e.g., BRISQUE+MFCC) performed poorly compared to deep learning-based AVQA models.

5. Significance

• Advancing Multimodal Learning: By providing rich, multi-dimensional annotations, the dataset allows researchers to move beyond simple quality prediction to understanding the mechanisms of human multimodal perception.
• Scalability Proof: The paper validates that with proper experimental design and filtering, crowdsourcing can replace expensive lab studies for AVQA, enabling the creation of massive datasets necessary for training large-scale AI models.
• Real-World Applicability: The focus on "in-the-wild" UGC content makes the dataset highly relevant for optimizing real-world streaming services, video compression, and content recommendation algorithms used by platforms like YouTube.

The dataset and code are publicly available at https://github.com/renyu12/YT-NTU-AVQ.