MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models

This paper introduces MUGEN, a comprehensive benchmark revealing that Large Audio-Language Models struggle with multi-audio understanding as input scaling increases, and demonstrates that combining training-free strategies like Audio-Permutational Self-Consistency with Chain-of-Thought can significantly improve performance.

The Gist

Imagine you are at a busy cocktail party. You aren't just listening to one person talking; you are trying to understand a conversation happening between three people, while also noticing the music playing in the background, the sound of clinking glasses, and the emotional tone of a joke being told.

This is the challenge that MUGEN (Multi-audio Grounding and Understanding Benchmark) was built to test.

Here is a simple breakdown of the paper, using everyday analogies:

1. The Problem: The "Single-Channel" Blind Spot

For a long time, AI models (Large Audio-Language Models, or LALMs) have been like excellent solo listeners. If you play them one song or one speech, they are great at understanding it. They can tell you if a singer is happy or if a speaker is angry.

But real life isn't a solo performance. It's a chaotic mix.

• The Gap: Current AI is mostly tested on "solo" audio. The researchers realized that when you throw multiple sounds at these AIs at once (like a speaker talking over music, or two people arguing), the AI gets confused. It's like asking a person who is great at reading a book to suddenly read three books printed on top of each other simultaneously.

2. The Solution: The "MUGEN" Party Game

To fix this, the researchers created MUGEN, which is essentially a giant, difficult party game for AI.

• The Setup: The AI is given a text instruction, like: "Find the clip where the speaker sounds the most angry."
• The Twist: Instead of giving the AI one clip to analyze, they give it five different audio clips playing at once (or in a list).
• The Goal: The AI has to listen to all five, compare them, and pick the one that fits the description.
• The Difficulty: The game gets harder in two ways:
  1. More Players: They increase the number of audio clips from 2 up to 5.
  2. Tricky Clues: Some clues aren't about what is being said (the words), but how it sounds (the emotion, the accent, the background noise). This stops the AI from just "reading the transcript" and forces it to actually listen.

3. The Results: The AI Gets Overwhelmed

When they played this game with the smartest AI models available (including some from Google and open-source projects), they found some surprising weaknesses:

• The "Crowded Room" Effect: As soon as they added more audio clips (going from 2 to 5), the AI's performance crashed. It's like trying to find a specific face in a crowd of 5 people; it's easy. Try finding that face in a crowd of 50, and you might miss it. The AI struggles to "scale up" its listening power.
• The "Meaning vs. Feeling" Gap: The AIs were okay at understanding the words (semantics), but terrible at understanding the vibe (non-semantic stuff like emotion, pitch, or background noise). They could tell you what someone said, but they often failed to tell you how they felt when saying it, especially when other sounds were present.
• The "Proprietary" Advantage: The most expensive, closed-source models (like Google's Gemini) did better than the free, open-source ones, but even they weren't perfect. They still got confused in the "crowded room."

4. The Fix: The "Shuffling" Trick

The researchers wanted to see if they could help the AI without retraining it (which is like trying to teach an adult a new language from scratch). They tried a clever trick called Audio-Permutational Self-Consistency (APSC).

• The Analogy: Imagine you are trying to guess the winner of a race by looking at the runners. If you always look at them from left to right, you might get biased by who is standing first.
• The Trick: The researchers told the AI: "Don't just look at the list of 5 audio clips once. Look at them in 10 different random orders."
  • Order 1: Clip A, B, C, D, E
  • Order 2: Clip C, A, E, B, D
  • ...and so on.
• The Result: By shuffling the order, the AI stops relying on "position" (e.g., "I always pick the first one") and starts actually comparing the sounds. When they combined this shuffling with a "Chain of Thought" (asking the AI to think step-by-step), the accuracy jumped significantly.

The Big Takeaway

This paper is a wake-up call. It tells us that while our AI audio assistants are getting smarter, they are still bad at multitasking. They struggle when the world gets noisy and complex.

However, the researchers found a "cheat code": Shuffling the input order helps the AI focus better. This is a simple, free way to make current AI models much better at understanding the messy, multi-sound reality of our daily lives.

In short: We built a test to see if AI can handle a noisy party. It mostly failed, but we found a trick (shuffling the playlist) that helps it listen a little bit better.

Technical Summary

Here is a detailed technical summary of the paper "MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models."

1. Problem Statement

While Large Audio-Language Models (LALMs) have shown promise in single-audio tasks, their ability to perform multi-audio understanding—reasoning over multiple concurrent audio segments simultaneously—remains underexplored.

• Real-world Gap: Practical applications (e.g., speech retrieval-augmented generation, multi-speaker analytics, in-context learning) require models to compare, aggregate, and reconcile information across multiple audio clips.
• Benchmark Limitations: Existing benchmarks primarily focus on single-audio settings or narrow multi-audio scenarios (typically 2–3 clips) that emphasize semantic content while neglecting non-semantic attributes (e.g., emotion, prosody) and larger input scales.
• Core Challenge: Current LALMs lack systematic evaluation for handling increasing numbers of audio inputs and distinguishing between semantic and non-semantic auditory features.

2. Methodology: The MUGEN Benchmark

The authors introduce MUGEN (Multi-audio Grounding and Understanding Benchmark), a comprehensive evaluation suite designed to test LALMs across diverse auditory dimensions.

• Task Design:

  • Format: 35 multiple-choice tasks where the model must select the audio candidate that best satisfies a textual constraint from a set of options.
  • Audio-as-Option: Unlike text-based multiple-choice questions, all options are audio signals. This forces the model to perform direct cross-audio comparison rather than relying on text shortcuts.
  • Scale: The benchmark includes 1,750 test instances derived from 9,250 audio clips.
  • Input Scaling: Tasks range from 5 candidates to 6 candidates (5 candidates + 1 reference audio), allowing for the study of performance degradation as input count increases.

• Evaluation Dimensions (7 Categories):

  1. Semantics & Pragmatics: Speech content and context.
  2. Speaker & Demographics: Identity, accent, and speaker cues.
  3. Affective & Paralinguistic: Emotion, prosody, and non-semantic vocal signals.
  4. Temporal Awareness: Duration and pacing.
  5. Acoustic Scene & Event: Environmental sounds and events.
  6. Music Analysis: Genre and instrumentation.
  7. Compositional Acoustic Reasoning: Integrated reasoning across multiple attributes.

• Data Construction:

  • Data is sourced from public corpora, specialized academic datasets, and high-quality scripted speech.
  • Synthetic generation (TTS) is used only when precise attribute control is required.
  • Distractors are systematically selected to exhibit contrasting variations of the target attribute, preventing superficial shortcuts.

3. Experimental Setup

• Baselines: Evaluated 7 advanced LALMs, including open-source models (DeSTA2.5-Audio, Qwen2.5-Omni, Audio Flamingo 3, Voxtral variants, Phi-4) and proprietary models (Gemini-3-pro).
• Cascaded Baseline: An ASR (Whisper-large-v3) + LLM (Gemini-3-pro) pipeline was used to determine if tasks could be solved via text transcription alone.
• Metrics: Accuracy evaluated via an LLM-as-a-judge (Claude Haiku 4.5) with 99% agreement with human annotators.

4. Key Results

A. Performance Limitations

• Overall Weakness: Even state-of-the-art models struggle with multi-audio understanding. Open-source models perform comparably to cascaded ASR+LLM systems, suggesting they fail to leverage acoustic signals effectively in multi-input settings.
• Semantic vs. Non-Semantic Imbalance: Models perform significantly better on Semantic tasks (e.g., content) than on Non-Semantic tasks (e.g., emotion, speaker identity). This indicates a reliance on semantic shortcuts rather than true acoustic reasoning.
• Input Scaling Bottleneck: Performance degrades sharply as the number of concurrent audio inputs increases.
  • Example: Qwen2.5-Omni retained only 48% of its 2-candidate accuracy when moving to 5 candidates (with reference), whereas Gemini-3-pro retained ~80%.
  • Increasing "thinking depth" (Chain-of-Thought) did not mitigate this scaling degradation.

B. Improvement Strategies (Training-Free)

The authors investigated inference-time strategies to improve performance without retraining:

1. Chain-of-Thought (CoT): Provided minimal or negative gains, confirming that the bottleneck is perceptual/acoustic, not logical reasoning.
2. Self-Consistency (SC): Generating multiple responses and voting improved performance slightly.
3. Audio-Permutational Self-Consistency (APSC):
   • Method: Randomly permuting the order of audio candidates before inference to reduce positional bias, then aggregating results via majority voting.
   • Result: APSC yielded the most significant gains, improving accuracy by up to 6.28% (Gemini-3-pro Low) and 5.37% (Gemini-3-pro High).
   • Combination: Combining APSC with CoT achieved the peak performance, with a total gain of 6.74% for Gemini-3-pro (Low).

5. Key Contributions

1. MUGEN Benchmark: A comprehensive, large-scale benchmark covering 7 dimensions and 35 tasks specifically designed to evaluate multi-audio grounding and reasoning.
2. Identification of Blind Spots: Systematic evidence that current LALMs suffer from severe performance degradation with increased input scaling and struggle significantly with non-semantic attributes.
3. Inference Optimization: Demonstration that Audio-Permutational Self-Consistency (APSC) is a highly effective, training-free strategy to mitigate positional bias and improve multi-audio comprehension.

6. Significance

• Foundational Insight: The paper exposes that current LALMs are not yet ready for complex, real-world auditory environments involving multiple simultaneous speakers or sound sources.
• Evaluation Standard: MUGEN provides a necessary standard for future research, moving beyond single-audio evaluation to test robustness against input scaling and non-semantic features.
• Practical Guidance: The findings suggest that simply prompting models to "think harder" (CoT) is insufficient; instead, architectural or inference strategies that address positional bias and acoustic aggregation (like APSC) are critical for advancing the field.

The benchmark and code are available at: https://huggingface.co/Multi-Audio-Grounding.