SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

This paper introduces SCENEBench, a comprehensive benchmark suite designed to evaluate Large Audio Language Models on critical non-speech and cross-lingual audio understanding tasks relevant to assistive and industrial applications, revealing significant performance gaps and latency challenges in current state-of-the-art models.

The Gist

Imagine you have a very smart robot friend who can listen to the world. You ask it, "What did that person say?" and it replies perfectly. But what if you ask, "What else is happening in this room?" or "Is that siren getting closer or moving away?" or "Is that person coughing because they are sick or just clearing their throat?"

Most of today's super-smart AI listening robots are like obsessive note-takers. They are incredible at writing down exactly what words were spoken (like a court stenographer), but they often miss the rest of the story. They might tell you a person said "Hello," but they completely ignore the fact that a dog was barking, a car was screeching, or the person was whispering in fear.

This paper introduces a new "report card" called SCENEBench to test if these AI robots can actually understand a scene, not just transcribe words.

Here is a breakdown of the paper using simple analogies:

1. The Problem: The "Blind Note-Taker"

Current AI models are like a student who is so focused on copying the teacher's lecture that they don't notice the fire alarm going off, the window breaking, or the teacher sounding sad.

• The Old Tests: Previous tests only asked, "Did you get the words right?"
• The New Reality: In the real world (like helping a deaf person or monitoring a factory), knowing what is said is less important than knowing what is happening around it.

2. The New Test: SCENEBench

The authors created a four-part obstacle course to see if AI can handle real-life chaos. Think of it as a "Driver's Ed" test for AI ears.

Task A: The Background Noise Detective

• The Scenario: Imagine a person talking at a busy coffee shop.
• The Test: Can the AI hear the person and the espresso machine grinding, the clinking of cups, and the rain outside?
• The Result: The AI is great at hearing the person. But if you don't specifically ask, "What else do you hear?", the AI usually ignores the background completely. It's like a student who only writes down the teacher's voice and misses the fire drill.

Task B: The "Siren Sense" (Noise Localization)

• The Scenario: A siren is wailing. Is it getting closer (danger!) or moving away (safety)?
• The Test: The AI has to listen to how the volume changes to guess the direction.
• The Result: The AI is terrible at guessing this on its own. It's like a person trying to guess which way a car is driving just by listening to the engine, but only if they are told to look for that specific clue. If you ask directly, "Is it coming or going?", they get better, but they still struggle with complex movements (like a siren passing by and swinging back and forth).

Task C: The Polyglot Translator

• The Scenario: A person is speaking in English but suddenly switches to Spanish, then Hindi, then back to English.
• The Test: Can the AI write down the whole sentence, keeping the different languages intact?
• The Result: The AI often gets confused. Instead of writing "I have a fifteen-day vacation quiero un viaje," it just rewrites the whole thing in English, erasing the foreign words. It's like a translator who refuses to let you speak your native tongue, forcing everything into English even when you didn't ask.

Task D: The "Vibe" Checker (Vocal Characterizers)

• The Scenario: A person is speaking, but they also cough, laugh, sneeze, or whisper.
• The Test: Can the AI tell the difference between a cough and a laugh, even if the words are the same?
• The Result: The AI is actually pretty good at this! It can tell when someone is laughing or coughing. However, it sometimes gets confused between similar sounds (like a yawn vs. a sigh).

3. The Big Surprise: The "Synthetic" vs. "Real" Gap

The researchers built these tests using "fake" audio (mixing two recordings together on a computer). They were worried this wouldn't reflect real life.

• The Analogy: It's like practicing driving in a video game and then taking the test on a real highway.
• The Finding: Surprisingly, the results were similar. The AI that did well in the "video game" (synthetic data) also did well in the "real highway" (real recordings). This means the test is a fair way to judge the AI's true abilities.

4. The Verdict: Why Does This Matter?

The paper concludes that current AI is optimized for the "What," not the "How" or the "Where."

• Why? Because most training data is "clean." It's like teaching a chef only how to cook perfect, isolated ingredients, but never how to cook a messy, noisy family dinner.
• The Fix: We need to train these AIs on messy, real-world audio where background noise, movement, and mixed languages happen naturally.

The Takeaway

SCENEBench is a wake-up call. It tells us that while our AI listening robots are brilliant at being stenographers, they are still clumsy at being context-aware observers.

If we want AI to help deaf people hear approaching cars or help factories detect dangerous machine noises, we can't just teach them to read words. We have to teach them to listen to the whole room. This new benchmark is the first step in making sure they actually learn that skill.

Technical Summary

Here is a detailed technical summary of the paper "SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases."

1. Problem Statement

While Large Audio Language Models (LALMs) have achieved state-of-the-art performance in Automatic Speech Recognition (ASR), current evaluation benchmarks largely fail to assess audio understanding beyond simple transcription. Existing benchmarks often focus on "clean-room" scenarios, single-modality inputs, or music classification, neglecting critical real-world complexities.

The paper identifies a gap in evaluating LALMs on:

• Background sound understanding: Detecting non-speech events (e.g., sirens, alarms) layered under speech.
• Spatial/Noise localization: Tracking motion or proximity of sound sources.
• Cross-linguistic robustness: Handling code-switching and multilingual spans without normalizing them to a single language.
• Vocal characterizers: Recognizing non-speech vocal traits (coughs, whispers, sobs) crucial for situational awareness in assistive and industrial settings.

Current models often prioritize "what is said" over "how it is said" or "what else is happening," leading to failures in high-stakes domains like accessibility (e.g., hearing aids for the deaf) and industrial safety (e.g., machine anomaly detection).

2. Methodology

Benchmark Construction (SCENEBench)

The authors propose SCENEBench (Spatial, Cross-lingual, Environmental, Non-speech Evaluation), a benchmark suite comprising four distinct tasks. To ensure reproducibility and control, the dataset is largely synthetically constructed by overlaying natural audio samples, though it is validated against natural recordings.

• Task 1: Background Sound Understanding
  • Data: 2,000 clips created by overlaying ESC-50 environmental sounds (e.g., traffic, rain) onto DailyTalk speech utterances.
  • Evaluation: A hierarchical scoring system:
    1. FR1: Free-response description (spontaneous salience).
    2. FR2: Targeted follow-up asking specifically for background noise (if missed in FR1).
    3. MC1: 4-way forced-choice classification.
• Task 2: Noise Localization
  • Data: 2,000 ESC-50 clips transformed with three amplitude envelopes: Approaching (10%→100%), Receding (100%→10%), and Oscillating (sinusoidal). Total 6,000 samples.
  • Evaluation: Free-text description of motion (FR1) and explicit direction query (FR2).
• Task 3: Cross-Linguistic Speech Understanding
  • Data: 2,541 DailyTalk turns where contiguous spans (~30%) were translated into Mandarin, Spanish, Hindi, or Portuguese using Google Translate, then synthesized via ElevenLabs TTS.
  • Evaluation: Transcription of multilingual sentences, measuring similarity to the reference code-mixed text.
• Task 4: Vocal Characterizers
  • Data: 4,006 clips of non-speech vocalizations (cough, cry, laugh, sneeze, yawn, mumble, whisper) aggregated from public repositories.
  • Evaluation: Free-text description followed by 7-way classification.

Models Evaluated

Five state-of-the-art LALMs were tested:

1. GPT-4o (OpenAI)
2. Gemini 1.5 (Google)
3. Qwen2-Audio-7B (Alibaba)
4. Audio-Flamingo-3 (NVIDIA)
5. DeSTA2-8B-beta (NTU + NVIDIA)

Metrics

• Accuracy: Measured via Free Response (FR) and Multiple Choice (MC) scores.
• Latency: Recorded for local models (median and interquartile range) to assess real-time viability.
• Ecological Validity: A subset of 20 natural (non-synthetic) clips per task was used to validate that synthetic trends hold in real-world data.

3. Key Contributions

1. New Benchmark Suite: SCENEBench is the first benchmark specifically designed to evaluate audio understanding in accessibility and industrial monitoring contexts, moving beyond ASR.
2. Task-Specific Failure Analysis: The paper categorizes specific failure modes (e.g., "noise override," "omission," "language normalization") that are currently overlooked by standard metrics.
3. Synthetic-to-Natural Validation: The authors demonstrate that while absolute accuracy differs between synthetic and natural data, the relative ranking of models remains consistent, validating the use of synthetic data for comparative benchmarking.
4. Latency Reporting: Inclusion of latency metrics provides a practical dimension for assessing deployability in real-time assistive devices.

4. Key Results

• Background Sound: Models rarely spontaneously mention background noise in free text (FR1). Performance improves significantly with explicit prompting (FR2) or multiple-choice formats (MC1).
  • Top Performer: Audio-Flamingo-3 on MC1; Qwen2 on FR1.
• Noise Localization: Models struggle to detect motion in free text (FR1 accuracy <10% for oscillating). Explicit questioning (FR2) boosts performance significantly (e.g., Qwen2 jumps from ~7% to ~57%), but oscillating patterns remain a major failure point.
• Cross-Linguistic: Models tend to "normalize" code-switching, dropping non-English spans and replacing them with fluent English.
  • Top Performer: GPT-4o and Gemini trade leads depending on the language; none achieve ceiling performance on Mandarin.
• Vocal Characterizers: Models perform well on common sounds (cough, laugh) but struggle with subtle distinctions (yawn vs. sigh).
  • Top Performer: Audio-Flamingo-3 leads on most categories but fails completely on "yawn."
• Latency: Local models vary widely. Audio-Flamingo-3 is the fastest (median ~0.8s–2.3s), while DeSTA2 is significantly slower (median ~6s–15s).
• Error Analysis:
  • Background: 75% of errors are "noise override" (model transcribes speech but ignores background).
  • Localization: 43% omission of motion; 25% collapse of oscillation patterns.
  • Cross-Lingual: 64% of errors involve dropping non-English spans.
  • Vocal: 94% of errors are misattributions between similar non-speech categories.

5. Significance and Future Directions

• Diagnostic Value: SCENEBench reveals that current LALMs are optimized for lexical fidelity (ASR) rather than holistic audio comprehension. A model can score well on existing benchmarks while failing critical safety tasks (e.g., missing a siren behind speech).
• Training Implications: The authors suggest targeted improvements, such as:
  • Training on speech-noise mixtures with explicit background labels.
  • Using contrastive pairs for multilingual spans to prevent normalization.
  • Integrating temporal reasoning objectives for motion detection.
• Ethical Impact: By focusing on paralinguistic cues (coughs, sirens) rather than "emotion AI," the benchmark avoids reductive labeling while addressing real safety needs for the Deaf/Hard-of-Hearing community and industrial safety.

The paper concludes that while LALMs show promise, significant gaps remain in their ability to parse the "scene" of an audio recording, necessitating new training objectives and evaluation standards grounded in real-world use cases.