PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

This paper introduces PolyBench, a new benchmark designed to evaluate compositional reasoning in polyphonic audio across five distinct tasks, revealing that current Large Audio Language Models consistently struggle with the complexity of concurrent sound events.

The Gist

Imagine you are at a busy, chaotic street festival. There is a band playing, a food truck sizzling, a dog barking, and a crowd cheering all at the same time.

Now, imagine you have a super-smart robot friend (an Audio Language Model) who is supposed to listen to this noise and tell you exactly what is happening.

If the robot listens to just the band playing alone, it's easy. It says, "That's a guitar!" But when everything is happening at once, the robot gets confused. It might think the sizzling food truck is a drum, or it might miss the dog barking entirely because the music is too loud.

This paper introduces a new test called PolyBench to see just how good these robots are at handling that "street festival" chaos.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Cocktail Party" Effect

Current AI models are great at listening to one thing at a time (like a monologue). But real life is polyphonic—meaning many sounds overlap.

• The Old Way: Previous tests asked the AI, "What sound is this?" when the sound was clear and alone.
• The New Reality: The authors realized that when sounds overlap, the AI starts "hallucinating" (making things up) or getting confused. It's like trying to read a book while someone is shouting in your ear; the AI can't separate the text from the noise.

2. The Solution: PolyBench (The "Chaos Test")

The authors built a new benchmark called PolyBench. Think of it as a "driving test" for AI, but instead of driving on an empty road, they are driving in heavy rush-hour traffic.

They created 5 specific challenges (tasks) to test the AI:

• Counting: "How many different things are making noise?" (e.g., Is it just a car and a bird, or is there also a siren?)
• Duration: "Which sound lasted the longest?"
• Concurrency: "Did two things happen at the exact same time?"
• Classification: "While the train was passing, what other sound was happening?"
• Detection: "At what exact moment did the chaos start?"

3. The Experiment: Who Passed the Test?

The researchers tested the smartest AI models available today (like Qwen3-Omni and Audio Flamingo) using these chaotic audio clips.

The Results were surprising:

• The "Easy" Stuff: The AI was okay at simple questions like, "Did a train and a bird overlap?" (Concurrency).
• The "Hard" Stuff: The AI failed miserably at complex questions like, "How many distinct sounds are there?" (Counting) or "When exactly did the overlap start?" (Detection).
• The Analogy: It's like a student who can answer "Yes/No" questions about a messy room but fails completely when asked to count how many specific items are in the pile.

4. The "Cheat Code" Discovery

One of the most interesting findings was that some AI models were cheating.

• When the test only included messy, overlapping sounds, some models would just guess "Yes, there is overlap" every single time. They got high scores, but they weren't actually listening; they were just guessing based on the pattern of the test.
• The researchers fixed this by adding "quiet" sounds to the test. Suddenly, the models' scores dropped, revealing they hadn't actually learned to hear the overlap; they had just memorized the test format.

5. The Conclusion: The Bottleneck

The paper concludes that while AI is getting better at understanding language, it is still terrible at listening to complex, overlapping sounds.

• The Bottleneck: The AI can't separate the "signal" (the specific sound) from the "noise" (the other sounds).
• What's Needed: To fix this, AI needs to get better at "un-mixing" the audio first, before it tries to reason about it. It needs to learn to pick out the dog's bark from the band's music before it can tell you when they happened together.

In a Nutshell

PolyBench is a new, harder test that proves current AI is still "tone-deaf" when it comes to complex, overlapping sounds. It's not just about hearing that a sound exists; it's about understanding the relationship between multiple sounds happening at once. Until AI can solve this, it will struggle to understand the real, noisy world we live in.

Technical Summary

Here is a detailed technical summary of the paper "PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio."

1. Problem Statement

While Large Audio Language Models (LALMs) have advanced significantly in audio understanding, existing benchmarks primarily focus on monophonic (single-source) or sequential audio events. They fail to adequately evaluate polyphonic audio, where multiple sound sources overlap simultaneously.

• The Gap: Current benchmarks often assess chronological ordering or isolated event detection but neglect the compositional reasoning required to parse overlapping sounds, distinguish concurrent events, and understand their temporal relationships.
• The Challenge: Polyphony introduces acoustic masking and complex structural dependencies, causing LALMs to frequently hallucinate, confuse sound sources, or fail to identify concurrent events, even when they perform well on single-source tasks.

2. Methodology: PolyBench Construction

The authors introduce PolyBench, the first benchmark specifically designed to evaluate compositional reasoning in polyphonic audio. The construction follows a three-stage pipeline:

Stage 1: Problem Observation

The authors observed that while LALMs accurately identify events in monophonic audio, they exhibit severe confusion and hallucination when processing overlapping (polyphonic) sounds. This observation drove the creation of a benchmark tailored to overlapping scenarios.

Stage 2: Data and Question Preparation

• Data Sources: PolyBench is constructed from real-world recordings sampled from three datasets: DataSED, DESED, and MAESTRO-Real.
  • Note: To ensure valid evaluation for the "Concurrency" task (determining if overlap exists), the authors also included monophonic clips from AudioTime to create a balanced mix of positive (overlap) and negative (no overlap) samples.
• Task Design: The benchmark utilizes Multiple-Choice Question Answering (MCQA) with five distinct reasoning categories:
  1. Counting: Identifying the number of distinct sound event categories.
  2. Duration: Determining which event has the longest duration.
  3. Detection: Identifying the time interval where multiple sounds first coincide.
  4. Classification: Identifying the category of a sound event overlapping with a specified "anchor" event.
  5. Concurrency: Determining whether any other event overlaps with a specific event.
• Generation: Questions were generated using Qwen3-Max to create semantic variants, followed by rigorous human-in-the-loop quality checks to ensure logical coherence and eliminate semantic bias in distractors.

Stage 3: Metric and Evaluation

• Protocol: Models receive audio inputs and textual instructions. For reasoning models, a Chain-of-Thought (CoT) prompt is used to elicit internal reasoning before selecting an answer.
• Metrics: Evaluation uses Accuracy (ACC) and F1 Score.
• Error Analysis: An LLM-based judge is employed to classify errors as either perceptual (incorrect event recognition) or reasoning (incorrect inference from correct perception).

3. Key Contributions

1. First Polyphonic Reasoning Benchmark: PolyBench fills a critical gap by explicitly targeting compositional reasoning in overlapping audio scenes, moving beyond simple event detection.
2. Structured Task Taxonomy: It defines five specific reasoning tasks (Counting, Duration, Detection, Concurrency, Classification) that progressively test the model's ability to handle temporal structure and event relations.
3. Real-World Data Curation: Unlike synthetic datasets, PolyBench is built from real-world recordings, ensuring the challenges (noise, complex overlaps) reflect realistic acoustic environments.
4. Diagnostic Framework: The inclusion of an error analysis protocol allows researchers to distinguish between perception failures and higher-level reasoning failures.

4. Experimental Results and Analysis

The authors evaluated state-of-the-art LALMs, including Qwen3-Omni-30B-A3B, R1AQA, Audio Flamingo 3, and a cascaded system (TimeAudio + Qwen3-8B).

• Performance Degradation: All models showed significant performance drops in polyphonic settings compared to monophonic baselines.
  • Hardest Tasks: Counting and Detection were the most challenging. Even the best model (Qwen3-Omni) achieved only 57.5% on Counting and 63.4% on Detection. This indicates that unstable perceptual evidence under polyphony severely impacts structured decisions like cardinality estimation and interval localization.
  • More Stable Tasks: Concurrency and Classification yielded higher scores (e.g., Qwen3-Omni: 83.1% on Concurrency, 77.9% on Classification), suggesting models can sometimes detect overlap existence or identify concurrent categories under anchor constraints.
• Shortcut Learning & Bias:
  • Models exhibited a "performance illusion" on the Concurrency task when tested only on polyphonic data (where the answer is always "Yes").
  • When tested on a balanced mix of monophonic and polyphonic data, accuracy dropped sharply.
  • Biases Identified: Some models (e.g., R1AQA) showed a strong "Yes" bias (relying on priors rather than audio), while others (e.g., Qwen3-Omni) showed an overly conservative "No" bias (only predicting overlap when evidence was extremely salient).
• Architecture Comparison:
  • Qwen3-Omni-30B-A3B demonstrated the most robustness across tasks.
  • Cascaded Systems (TimeAudio + Qwen3-8B) performed surprisingly well on Detection (51.7%), suggesting that separating temporal localization (perception) from text-based decision-making (reasoning) can mitigate some polyphonic challenges.

5. Significance and Conclusion

• Fundamental Bottleneck: The study reveals that current LALMs lack robust compositional reasoning capabilities for complex acoustic scenes. The inability to reliably parse overlapping events is a fundamental bottleneck, not just a data scarcity issue.
• Future Directions: Improving LALMs for polyphonic audio requires:
  1. Strengthening low-level event perception and temporal-structure modeling to reduce masking-induced instability.
  2. Enhancing cross-modal alignment to ensure high-level reasoning is grounded in robust acoustic evidence rather than statistical priors or shortcuts.
• Impact: PolyBench provides a necessary diagnostic tool to drive the development of next-generation audio models capable of operating in dynamic, real-world environments where multiple sound sources interact.