Technical Summary: Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)

Problem Statement

The field of auditory intelligence is undergoing a paradigm shift from specialized, unimodal systems (cascaded pipelines with separate encoders like Whisper and text-based LLMs) toward integrated, "audio-native" Large Language Models (LLMs). These next-generation models, such as Gemini 3 and GPT-5, integrate audio processing directly into their neural fabric to enable end-to-end reasoning. However, as the industry moves toward these unified architectures, there is a lack of a rigorous framework to measure progress across the full spectrum of sound understanding. Existing benchmarks often focus on specific tasks or English-language semantics and fail to capture the breadth required for true multimodal auditory intelligence. This work addresses the need to evaluate whether audio-native LLMs can achieve functional parity with specialized systems and traditional cascaded approaches across a variety of audio tasks.

Methodology

The authors conducted a rigorous empirical evaluation of leading LLMs (including members of the Gemini and GPT families) using the Massive Sound Embedding Benchmark (MSEB). The study involved adapting multimodal LLMs to solve eight distinct "Super-tasks" defined by the MSEB:

1. Transcription: Converting speech to text.
2. Retrieval: Speech search and querying of knowledge bases (In-language and Cross-language).
3. Reasoning: Directly answering questions based on audio input.
4. Classification: Identifying intent, speaker gender, and sound events.
5. Reranking: Refining transcription hypotheses.
6. Segmentation: Locating relevant terms (keywords) with timestamps.
7. Clustering: Unsupervised grouping of audio samples.
8. Reconstruction: Measuring embedding fidelity through waveform regeneration.

Experimental Setup:

• Evaluated Models: The study compared proprietary audio-native LLMs (Gemini 2.5/3 Flash, GPT-4o-mini-audio) with specialized and cascaded baselines (Whisper large-v3, GPT-4o-transcribe, ElevenLabs, LAION Clap, and Gemini Embedding).
• Prompt Engineering: To unify the evaluation, the authors standardized various tasks using specific prompt templates. For non-generative tasks (e.g., Retrieval, Segmentation), they employed structured JSON outputs and, where necessary, utilized Retrieval-Augmented Generation (RAG) pipelines to handle context window limitations.
• Metrics: Performance was quantified using task-specific metrics, including Word Error Rate (WER), Mean Average Precision (MAP), Normalized Discounted Cumulative Gain (NDCG), F1-Scores, and Mean Reciprocal Rank (MRR).
• Datasets: The evaluations utilized the SVQ (Simple Voice Questions), Speech-MASSIVE, and FSD50K datasets, covering 17 languages and 26 locales with varying noise conditions.

Key Contributions

1. Adaptation of LLMs: The authors successfully adapted multimodal LLMs to solve a variety of MSEB audio tasks encompassing both generative (Reasoning) and non-generative (Retrieval, Classification) challenges via task-specific prompting.
2. Comprehensive Evaluation: A direct comparison was conducted between modern multilingual LLMs and state-of-the-art specialized/cascaded systems across all eight MSEB capabilities.
3. Audio-Text Parity Analysis: The study analyzed the performance gap between direct audio processing and processed transcribed text, examining dimensions such as modeling approach (native vs. cascaded), acoustic robustness, and locale-specific variance.
4. Open Contribution: The results were added to a public leaderboard, and the implementation was open-sourced by extending the MSEB toolkit.

Results

The empirical results indicate a complex landscape in which no single "optimal" modeling approach emerges as universally superior:

• Modality Gap: A significant performance gap persists between audio-native models and their text-based counterparts (or Ground-Truth transcripts). While some models (e.g., Gemini 3 Flash) show promise in closing this gap on Reasoning tasks, others struggle to follow instructions when processing raw audio (e.g., GPT-4o-mini-audio failing to provide literal transcriptions without conversational fillers).
• Task-Specific Performance:
  • Transcription: Specialized ASR models (GPT-4o-transcribe, ElevenLabs) significantly outperformed audio-native LLMs in Word Error Rate (WER).
  • Classification: Gemini models demonstrated strong performance in intent and sound classification, often surpassing specialized models like LAION Clap, although the authors suspect this may be partially due to data contamination (the models may have seen the benchmark data during pre-training).
  • Retrieval & Reranking: Audio-native models showed moderate success in reranking but generally lagged behind cascaded approaches using specialized embeddings (Gemini Embedding) in the initial retrieval task.
  • Segmentation: LLMs performed poorly in segmenting relevant terms, particularly regarding timestamp accuracy, compared to ASR-based models.
• Robustness and Locales: Performance variance across different locales was substantial. Several models produced outliers with WERs exceeding 100% in specific locales. Acoustic robustness varied by task; Reasoning scores remained stagnant regardless of noise levels, suggesting that model capacity, rather than environmental noise, is the bottleneck.
• Data Contamination: The authors identified potential data contamination, noting that high zero-shot intent classification scores and retrieval success rates in locales with extremely high WER suggest that models may have memorized the benchmark data or that retrieval is decoupled from literal transcription accuracy.

Significance and Claims

The work concludes that the choice between audio-native and cascaded architectures is not a binary decision of superiority but depends heavily on specific use-case requirements regarding latency, cost, and reasoning depth.

• Parity Status: The authors claim that while audio-native models are rapidly evolving, a significant modality gap regarding performance and robustness remains. The modalities "I type" and "I speak" have not yet achieved full functional equivalence across all tasks.
• Architectural Future: The work argues that regardless of the coupling between language and speech components, these modules must be co-designed and jointly optimized to achieve optimal downstream performance.
• Modest Perspective: The authors remain modest regarding the current state of the field, noting that empirical evidence for an "optimal" modeling approach remains inconclusive. They emphasize that the transition to audio-native standards requires rigorous frameworks like the MSEB to prevent overestimation of capabilities due to data contamination or task-specific biases.

Future work is proposed to extend evaluations to high-parameter frontier models (e.g., Gemma 3, Qwen2.5-Omni) and to integrate newer multimodal embedding models into RAG pipelines.