🤖 AI

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

This paper introduces the CASU benchmark and a scalable data construction pipeline to evaluate and advance the ability of Large Audio Language Models to holistically understand and reason about complex, multi-layered real-world auditory scenes by integrating speech, sound events, and background environments.

Original authors: Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif, Yutong Song, Wenjun Huang, Henry Peng Zou, Pinxin Liu, Honghui Xu, Amir M. Rahmani

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif, Yutong Song, Wenjun Huang, Henry Peng Zou, Pinxin Liu, Honghui Xu, Amir M. Rahmani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a busy coffee shop. You hear a barista shouting an order, the hiss of the espresso machine, the clatter of cups, and the low hum of a refrigerator.

Older AI models are like a person who only listens to the barista. They are great at writing down exactly what the barista said (transcription), but they might miss why the barista is shouting. Is it because the shop is empty? Or is it because the espresso machine just exploded? They hear the words, but they don't "get" the scene.

This paper introduces a new way to test AI, called CASU (Context-Aware Auditory Scene Understanding). It asks a simple question: Can the AI understand the whole story by listening to the words, the background noise, and the sudden sounds all at once?

Here is a breakdown of what the researchers did and found, using simple analogies:

1. The Problem: The "Single-Track" vs. The "Orchestra"

Think of audio as a piece of music.

Old Benchmarks: These tested the AI on just one instrument at a time. "Can you hear the violin?" (Speech). "Can you hear the drums?" (Sound events).
The Real World: In real life, everything happens at once. The violin might be playing a sad song, but if the drums suddenly crash (like a siren), the meaning of the whole scene changes.
The Gap: The paper found that even the smartest AIs, which are perfect at transcribing speech (like a super-fast stenographer), often fail when they have to figure out how the background noise changes the meaning of the conversation. They hear the notes, but they don't understand the song.

2. The Solution: Building a "Sound Lab"

To test this properly, the researchers couldn't just use random recordings from the internet because those are messy and don't have clear answers. Instead, they built a semi-synthetic sound lab.

The Recipe: They wrote a "script" (like a movie script) that described three layers:
1. The Background: (e.g., a busy airport).
2. The Event: (e.g., a sudden announcement).
3. The Speech: (e.g., two people arguing about a flight).
The Mix: They used computers to mix these layers together perfectly, like a DJ blending tracks. This allowed them to create thousands of unique "scenes" where they knew exactly how the layers were supposed to interact.
The Test: They then asked the AI questions that required connecting the dots.
- Example: "Speaker A says the flight is on time. Speaker B says it's delayed. The airport announcement just said it's delayed. Who is right?"
- To answer this, the AI can't just listen to the people; it must listen to the announcement.

3. The Four Challenges (The "Gym" for AI)

The researchers created four specific tasks to see if the AI could handle the "whole scene":

Contextual Reasoning (The Detective): Can the AI spot when background noise contradicts what people are saying? (e.g., Someone says "It's quiet," but a siren is blaring).
Entity Extraction (The Detective): Can the AI figure out what is happening just by the sounds, even if no one says it? (e.g., Hearing a crack of thunder and a power outage hum, the AI should know it's a storm, not a flood).
Role Inference (The Social Butterfly): Can the AI guess who the people are based on the setting? (e.g., If the background is a hospital and someone says "Code Blue," the AI should guess they are doctors, not family members).
Counterfactual Reasoning (The "What If" Game): This is the hardest. The researchers asked: "If we replaced the sound of a dog barking with a car horn, how would the story change?" This tests if the AI truly understands cause and effect, not just memorizing patterns.

4. The Results: The "Perception-Understanding Gap"

When they tested the best AI models available today, they found a surprising result:

The "Perception-Understanding Gap": Many models are amazing at hearing words (99% accurate transcription) but terrible at understanding the scene. It's like having a perfect dictionary but no common sense.
The "All-or-Nothing" Rule: The researchers tested what happens if they remove one layer of sound (like muting the background noise).
- If they muted the speech, the AI failed completely (it had nothing to talk about).
- If they muted the background/events, the AI's performance dropped significantly. It proved that the AI needs the background noise to make logical sense of the speech. It can't just guess; it needs the clues.
The "Cascaded" Failure: They tried a method where one AI writes down a description of the sound, and a second AI reads that description to answer the question. This failed. Why? Because the first AI missed subtle clues (like the echo of a room) that are hard to write down but crucial for understanding the scene. The best models are the ones that listen to the raw audio directly.

5. The Takeaway

The paper concludes that for AI to truly understand the world of sound, it needs to stop treating background noise as "static" or "noise." Instead, it needs to treat every sound—whether it's a voice, a siren, or a coffee machine—as a vital piece of a puzzle.

Just like a human doesn't just listen to a speaker in a noisy room but also listens to the room itself to understand the context, the next generation of AI needs to learn to listen to the whole orchestra, not just the soloist.

Technical Summary: From Sounds to Scenes (CASU Benchmark)

Problem Statement

Recent Large Audio Language Models (LALMs) have demonstrated significant progress in isolated acoustic tasks, such as Automatic Speech Recognition (ASR), sound event detection, and audio captioning. However, existing benchmarks predominantly evaluate these capabilities in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world environments.

Current evaluation paradigms often treat mixed-layer audio (speech, events, and background) as loosely coupled signals, failing to require models to reason about their logical interactions. In contrast, human auditory understanding is inherently scene-level and context-aware; it integrates speech with surrounding events and background sounds to infer higher-order meaning. For instance, the phrase "It's time to go" implies a routine dismissal if accompanied by a school bell but becomes a critical evacuation warning if paired with an emergency siren. This gap motivates the need for Context-Aware Auditory Scene Understanding (CASU), which formalizes auditory scene understanding as a reasoning problem over structured interactions between speech, events, and background sounds.

Methodology

Dataset Construction: A Semi-Synthetic Pipeline

To rigorously evaluate CASU, the authors introduce a scalable, semi-synthetic data generation pipeline consisting of three stages:

Script Generation and Semantic Blueprint: A GPT-4o agent generates structured JSON scripts defining complex auditory scenarios. Each script specifies three semantic layers:
- Background Sound ( $T_B$ ): Continuous environmental sounds (e.g., traffic, room reverberation).
- Speech ( $T_P$ ): Human dialogue content.
- Acoustic Events ( $T_E$ ): Discrete, transient sounds (e.g., sirens, glass breaking).
  Scripts include text descriptions, start/end timestamps, and explicit narrative relationships between layers.
Audio Layer Retrieval and Synthesis:
- Speech Synthesis: Naturalistic speech is generated using Zonos (a high-fidelity TTS tool), conditioned on random speaker embeddings from the People's Speech dataset to ensure diversity.
- Non-Speech Retrieval: Background and event sounds are retrieved from source datasets (Clotho for backgrounds, ARCA23K for events) using a Matching Score-Based Retrieval system. This system computes a weighted matching score ( $S_{match}$ ) combining semantic similarity (via Sentence Transformers) and keyword-level similarity (via categorical tags) to select the most appropriate audio candidates for the script's descriptions.
Time-Aligned Audio Composition: The final audio stream is constructed by treating the background sound as the base layer and aligning the synthesized speech and retrieved event sounds at their designated temporal intervals. A human filtering stage ensures balanced mixing levels and logical causality.

Task Suite

The benchmark defines four tasks designed to probe scene understanding beyond simple perception:

Contextual Reasoning: Evaluates the model's ability to reconcile spoken assertions with broader acoustic context (e.g., determining if an event confirms or contradicts a speaker's claim).
Entity Extraction: Requires identifying entities implied by the scene's sounds rather than explicitly stated in speech (e.g., inferring a "storm" from thunder and rain sounds).
Role Inference: Assesses the ability to infer social roles or interpersonal relationships by combining speech content with scene context (e.g., distinguishing a doctor-patient interaction from a family conversation based on background cues).
Counterfactual Reasoning: Tests causal understanding by asking how a conclusion would change if a specific acoustic element were hypothetically altered (e.g., replacing a sound with a different one).

Questions are generated using an agent-based framework where LLMs propose candidates, which are then validated by critic agents and human filters to ensure logical consistency and difficulty.

Key Contributions

CASU Paradigm: The introduction of Context-Aware Auditory Scene Understanding, shifting the focus from signal filtering to reasoning over structured interactions among speech, events, and background.
Scalable Data Pipeline: A novel semi-synthetic generation pipeline that enables precise control over cross-layer contextual relationships, facilitating systematic evaluation of scene understanding.
Comprehensive Benchmark: A benchmark suite with four carefully designed tasks and a comprehensive evaluation of state-of-the-art LALMs, revealing persistent limitations in multi-layer auditory understanding.

Experimental Results

Main Findings

The authors evaluated a diverse set of models, including open-source LALMs (Qwen-series, Audio Flamingo, LTU, SALMONN, Voxtral) and closed-source models (GPT-4o Audio, Gemini 2.0 Flash).

The "Perception-Understanding Gap": Models achieving near-human performance on single-layer tasks (e.g., high-fidelity speech transcription with low WER) frequently fail on scene understanding tasks. For example, Audio Flamingo 3 and Mistral Voxtral achieve high transcription accuracy ( $1 - \text{WER} \geq 0.95$ ) but lag significantly (by ~15%) behind state-of-the-art models in CASU tasks. This indicates that high-fidelity transcription is insufficient; effective scene understanding requires the holistic integration of all auditory dimensions.
Joint Processing vs. Cascaded Reasoning: Omni-modal models (e.g., Qwen3-Omni, GPT-4o Audio) consistently outperform cascaded pipelines (e.g., Captioner + Reasoner). Cascaded systems suffer from information loss when converting audio to text, discarding subtle acoustic cues (e.g., reverberation) that are critical for logical reasoning.
Layer Roles: Different layers play distinct roles. Speech serves as the semantic foundation, but Events and Background provide logical constraints. Ablation studies show that removing speech causes a performance collapse (to near-random levels), but removing non-speech layers also leads to significant degradation, particularly in Entity Extraction and Counterfactual Reasoning.
Scene Complexity: Multi-person conversations introduce significant difficulty compared to monologues, with Role Inference accuracy dropping by approximately 12% in conversational settings.
Impact of Hints: Providing explicit textual scene hints (e.g., "This is a hospital") improves performance, especially for open-source models, suggesting that robust internal representations of auditory scenes in stronger models (like GPT-4o) make such hints redundant.

Performance Metrics

Even the most capable models (GPT-4o Audio, Gemini 2.0 Flash, Qwen3-Omni) achieve only 70–74% accuracy on CASU tasks, indicating that current LALMs still struggle to fully resolve the complex inter-layer reasoning required for true scene understanding.

Significance and Claims

The paper claims that CASU represents a critical stepping stone for advancing complex audio understanding in LALMs. The authors argue that true auditory intelligence requires models to actively discern the semantic utility of non-speech sounds rather than treating them as mere noise. By formalizing scene-level interaction and reasoning, CASU highlights the necessity for future models to move beyond isolated perception toward holistic, context-aware understanding of real-world auditory scenes. The work underscores that effective auditory scene understanding relies on the integration of all auditory layers, not just speech or sound alone.