⚡ electrical engineering

VGGSounder: Audio-Visual Evaluations for Foundation Models

This paper introduces VGGSounder, a comprehensively re-annotated multi-label test set designed to overcome the labeling and alignment limitations of the original VGGSound dataset, thereby enabling more reliable and precise evaluation of audio-visual foundation models through detailed modality analysis and a new confusion metric.

Original authors: Daniil Zverev, Thaddäus Wiedemer, Ameya Prabhu, Matthias Bethge, Wieland Brendel, A. Sophia Koepke

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniil Zverev, Thaddäus Wiedemer, Ameya Prabhu, Matthias Bethge, Wieland Brendel, A. Sophia Koepke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world by showing it videos. You want the robot to know that if it sees a drum being hit and hears a thump, it should say, "That's a drum!"

For a long time, researchers used a giant library of videos called VGGSound to test these robots. But the authors of this paper, Daniil Zverev and his team, discovered that this library was like a messy, poorly organized attic. It had three big problems that made it impossible to tell if the robots were actually smart or just lucky:

The "One Label" Trap: The library forced every video to have only one tag. But in real life, a video might show a dog barking while a car honks. The old library would just pick one, ignoring the other. It's like describing a pizza with pepperoni and mushrooms as just "a pizza with cheese."
The "Confusing Names" Problem: Some labels were twins. One tag might say "dog barking" and another "dog bow-wow." The robots got confused because they were essentially the same thing, but the test treated them as different.
The "Ghost" Problem: Sometimes the library claimed a sound was in the video when it wasn't actually visible, or vice versa. For example, a video might be labeled "orchestra," but you can only hear the music, not see the instruments. The old test didn't care about this mismatch.

The Solution: VGGSounder

The team built a new, upgraded library called VGGSounder. Think of this as renovating that messy attic into a high-tech, perfectly organized museum.

Here is what they did differently:

Multi-Labeling: Instead of forcing one tag, they let every video wear as many tags as it needs. A video can be tagged "drums," "guitar," and "singing" all at once.
Modality Tags: They added a special "checklist" for every single tag. They asked: Is this thing visible? Is this thing audible? This lets them test if a robot is good at seeing, good at hearing, or good at doing both.
Meta-Labels: They added "warning signs" for tricky situations. If a video has background music, a voice-over, or is just a static photo with sound, they flagged it. This helps researchers see if a robot gets distracted by noise.

The Big Discovery: The "Distraction" Effect

The most surprising thing the team found was how the robots behaved when they were given both eyes and ears.

They invented a new score called "Modality Confusion." Imagine you are good at solving a puzzle using only your eyes. Then, someone hands you a second puzzle piece (the sound) and suddenly you can't solve the first one anymore. That is Modality Confusion.

The Finding: Many of the newest, most advanced "Foundation Models" (the super-smart AI robots) actually got worse when they were allowed to listen and watch at the same time.
The Bias: These robots seemed to be "blind" to sound. If you showed them a video of a dog barking but only let them hear it, they failed. But if they could see the dog, they succeeded. When they tried to use both, the sound often confused them, and they ignored it entirely, relying only on what they saw.

Why This Matters

The paper argues that we can't just throw a bunch of data at a robot and hope it learns. We need a better test (VGGSounder) that tells us exactly how the robot is thinking.

Old Test (VGGSound): Like a driving test where the road is foggy and the signs are missing. You might pass by luck, but you don't know if you're a safe driver.
New Test (VGGSounder): Like a driving test with clear signs, a co-pilot who points out hazards, and a report card that tells you exactly which senses you used to make your decisions.

The authors conclude that while these new AI models are impressive, they often struggle to combine sight and sound effectively. They tend to ignore the audio if they can see the video, which is a major flaw for machines that are supposed to understand the world holistically. VGGSounder is the tool designed to expose these flaws so engineers can fix them.

Technical Summary: VGGSounder: Audio-Visual Evaluations for Foundation Models

Problem Statement

The emergence of audio-visual foundation models necessitates reliable benchmarks to assess their multi-modal understanding. While the VGGSound dataset is the standard benchmark for audio-visual classification, the authors identify critical limitations that distort the evaluation of auditory and visual capabilities:

Incomplete Labelling: VGGSound employs a single-label classification scheme, whereas video content is inherently multi-label (e.g., a video may contain both "playing drum kit" and "playing acoustic guitar").
Overlapping and Ambiguous Classes: The 309 classes in VGGSound include synonymous pairs (e.g., "timpani" vs. "tympani"), strict subclasses (e.g., "cow lowing" vs. "bull bellowing"), and semantically overlapping concepts that are not mutually exclusive.
Modality Misalignment: Despite verification steps, a significant portion of samples (48.43% in the test set) contain labels that are not visible, not audible, or both. Common confounders include background music, voice-overs, and static images, which create cues that do not align with the ground-truth label in both modalities.

These issues lead to systematic under-evaluation of multi-modal models, particularly foundation models that were not specifically fine-tuned on VGGSound, and obscure the specific reliance of models on audio versus visual cues.

Methodology

To address these limitations, the authors introduce VGGSounder, a comprehensively re-annotated, multi-label test set extending VGGSound. The methodology involves a rigorous annotation pipeline:

Multi-Label Classification Shift: The benchmark transitions from single-label to multi-label classification to accommodate co-occurring events.
Human Annotation Pipeline:
- Gold Standard Creation: A subset of 417 samples was manually labeled by four computer vision experts to establish a reference set.
- Proposal Generation: A hybrid strategy combining predictions from state-of-the-art models and manual heuristics was used to generate label proposals with >93% recall.
- Crowdsourced Re-annotation: The entire VGGSound test set (15.1k clips) was re-annotated via Amazon Mechanical Turk. Annotators were tasked with:
  - Identifying the presence of background music, voice-over, or static images (meta-labels).
  - Determining the modality of each label proposal (Audible, Visible, or Both).
  - Adding missing classes not present in the proposals.
Label Merging: Final labels were merged via majority voting. Synonymous classes and superclass/subclass relationships were automatically enforced to ensure consistency (e.g., if "cow lowing" is present, "cattle mooing" is added).
New Evaluation Metrics:
- Standard Metrics: Subset Accuracy, F1-score, and Hit score were computed for unimodal (audio-only, visual-only) and multi-modal inputs.
- Modality Confusion ( $\mu$ ): A novel metric proposed to quantify the negative impact of adding a second modality. It measures the fraction of samples a model classifies correctly with a single modality but incorrectly when both modalities are present. This exposes "distracted" models where modalities fail to merge successfully.

Key Contributions

VGGSounder Dataset: A multi-label audio-visual classification benchmark with detailed per-label modality annotations (Audible/Visible) and meta-labels for common confounders (background music, voice-over, static images).
Limitation Analysis: A qualitative and quantitative demonstration of VGGSound's flaws, including high rates of modality misalignment and class co-occurrence.
Comprehensive Benchmarking: Evaluation of 11 state-of-the-art models, including four audio-visual embedding models (CAV-MAE, DeepAVFusion, AV-Siam, Equi-AV) and seven foundation models (Gemini family, VideoLLaMA-2, Unified-IO-2, PandaGPT, Ola).
Modality Confusion Metric: The introduction of $\mu$ to detect performance degradation when models are exposed to additional input modalities.

Results

The authors benchmarked models on VGGSounder, revealing several key findings:

Foundation vs. Embedding Models: While foundation models perform comparably to specialized embedding models on multi-modal inputs, their unimodal performance differs significantly. Embedding models (fine-tuned on VGGSound) rely heavily on audio cues, whereas foundation models exhibit a strong bias toward visual cues, often performing poorly on audio-only inputs.
Modality Confusion: All models suffer from modality confusion. A notable fraction of samples (4–11%) are correctly classified with a single modality but misclassified when both are provided. For most foundation models, the confusion is higher when audio is added to visual input ( $\mu_A > \mu_V$ ), suggesting they forget audible labels when visual information is present.
Impact of Meta-Classes:
- Background Music: All models struggle to decouple background music from the primary audio source, leading to performance drops.
- Voice-Over: Embedding models suffer significant performance drops with voice-overs, while foundation models are less distracted or even improve.
- Static Images: Visual performance drops for all models on static images, indicating a reliance on temporal cues.
Label Quality: The inclusion of human-curated labels (as opposed to just automatically generated synonyms) significantly reduced false positives and improved the accuracy of performance profiling compared to the original VGGSound.

Significance

The paper argues that VGGSounder is a critical tool for the accurate assessment of the next generation of audio-visual foundation models. By providing modality-aware annotations, it enables:

Precise Profiling: Researchers can distinguish whether a model relies on audio, visual, or both cues, and identify specific failure modes (e.g., distraction by background music).
Robust Evaluation: The multi-label setting and removal of misaligned samples prevent the distorted evaluation of model capabilities found in the original VGGSound.
Diagnostic Insight: The modality confusion metric ( $\mu$ ) provides a quantitative measure of how well models integrate modalities, highlighting that adding a second modality does not always improve performance and can sometimes degrade it.

The authors conclude that VGGSounder facilitates the development of mitigation strategies for modality confusion and serves as a more reliable foundation for evaluating multi-modal understanding.