💬 NLP

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

This paper introduces PolySpeech-100, a large-scale benchmark covering 110 linguistic variants that evaluates speech understanding across diverse languages and dialects, revealing that open-source end-to-end models excel at preserving paralinguistic cues in heavy dialects while highlighting significant performance gaps in low-resource languages and the counter-intuitive negative impact of Chain-of-Thought prompting on speech comprehension.

Original authors: Sicheng Yang, Shulan Ruan, Shiwei Wu, Yu Liu, Lu Fan, Zhi Li, You He

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sicheng Yang, Shulan Ruan, Shiwei Wu, Yu Liu, Lu Fan, Zhi Li, You He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Testing the "Ear" of AI

Imagine you are teaching a robot to understand human speech. For a long time, we only tested if the robot could hear English or standard Mandarin clearly. But the real world is messy. People speak with thick accents, use local slang, and talk in dialects that sound nothing like the "textbook" version of a language.

The authors of this paper built a massive testing ground called PolySpeech-100. Think of it as a "global driving test" for AI, but instead of driving cars, the AI has to listen to people speaking in 110 different languages and dialects (including 19 different Chinese dialects like Cantonese, Sichuanese, and Shanghainese).

The Problem: The "Translator" Bottleneck

Before this paper, most AI systems worked like a two-step process:

Step 1: A robot listens to the voice and writes it down as text (like a stenographer).
Step 2: A smart brain (a Large Language Model) reads that text and answers the question.

The paper argues that Step 1 is the weak link. When you force a robot to turn a thick dialect into text first, it often loses the "flavor" of the speech—the tone, the stress, and the unique sounds that give the meaning away. It's like trying to understand a joke by reading a translation of it; you might get the words, but you miss the punchline.

The Solution: A "Native Speaker" Test

The researchers created a benchmark where they didn't just ask, "Can you read this?" They asked, "Can you understand this?"

To build this test, they faced a huge problem: Data Scarcity. There are no recordings of native speakers for many rare dialects.

The Creative Fix: They used a "hybrid" approach. For common languages, they used real human recordings (the gold standard). For rare dialects, they used advanced AI to synthesize (create) speech that sounds like a native speaker.
The Safety Check: They proved their synthetic speech was good enough by having real humans listen to it. The AI's performance on the fake speech matched its performance on real speech almost perfectly (a correlation of 0.83). This means the test is fair, even for languages where no human recordings exist.

The Results: What They Found

They tested 22 different AI models (both free/open-source and paid/commercial) on this massive test. Here are the three big discoveries:

1. The "Direct Listener" Wins on Dialects

The Finding: When it came to heavy dialects (like thick Sichuanese), the End-to-End (E2E) models (which listen directly and answer) beat the Cascaded models (which listen, write text, then answer).
The Analogy: Imagine trying to understand a song.

The Cascaded Model tries to write down every note on sheet music first. If the singer has a unique style, the sheet music looks wrong, and the model gets confused.
The End-to-End Model just listens to the melody and feels the rhythm. It captures the "vibe" and the unique sounds that the sheet music (text) would have thrown away.
Result: Open-source models that listen directly actually did a better job on dialects than the traditional text-based systems.

2. The "Rich vs. Poor" Gap

The Finding: Commercial models (like Google's Gemini) are incredibly robust; they handle rare languages (like Zulu or Lao) almost as well as common ones. However, open-source models crash hard on these rare languages.
The Analogy: Think of the Commercial models as a universal polyglot who has traveled the world and heard every accent. The open-source models are like local experts who are great at their home city (common languages) but get lost the moment they step into a foreign village (low-resource languages).

3. The "Thinking Aloud" Trap

The Finding: In text-based AI, asking the model to "think step-by-step" (Chain-of-Thought) usually makes it smarter. But for these speech models, asking them to "think aloud" often made them dumber.
The Analogy: Imagine you are listening to a complex story in a noisy room.

If you just listen and pick the answer, you do well.
If you try to stop, write down a summary of the story, explain your logic, and then pick the answer, you get distracted. You lose the thread of the audio because you are too busy focusing on your own writing.
Result: For most speech models, "thinking aloud" broke their connection to the audio, causing them to hallucinate or guess wrong.

The Conclusion

The paper concludes that to build truly inclusive AI that understands everyone—not just people with perfect accents—we need to stop relying on "transcribing first, understanding later." We need models that can listen directly to the acoustic "flavor" of speech.

They also warn that simply adding "reasoning steps" to speech AI doesn't work yet; the technology needs to learn how to reason while listening, not after.

Where to find it: The data, code, and a demo where you can listen to these dialects yourself are available on their GitHub page.

Technical Summary: PolySpeech-100

Problem Statement

The rapid evolution of End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) has outpaced the development of robust evaluation methodologies. Existing benchmarks suffer from three critical limitations:

Resource Bias: They are heavily skewed toward high-resource languages (primarily English and Standard Mandarin), neglecting the vast majority of the world's languages.
Task Limitation: They focus on low-level recognition (Automatic Speech Recognition, ASR) or simple instruction following rather than complex semantic reasoning and content comprehension.
Dialect Neglect: They treat languages as monoliths, ignoring regional dialects (e.g., Cantonese, Sichuanese, Maghrebi Arabic) which pose significant challenges for real-world deployment.

Consequently, there is a lack of rigorous standards to assess whether Speech-LLMs possess "native-level" listening capabilities or merely overfit to standard broadcast speech.

Methodology

The authors introduce PolySpeech-100, a large-scale benchmark covering 110 linguistic variants, including 19 distinct Chinese regional dialects and over 80 low-resource languages. The benchmark is built upon a hybrid data construction pipeline designed to solve data scarcity issues:

Data Foundation: The textual backbone is derived from the Belebele dataset, which provides parallel reading comprehension passages and multiple-choice questions (MCQs) across 122 languages, ensuring cross-lingual comparability.
Hybrid Audio Construction:
- Track 1 (High-Resource): Utilizes the 2M-BELEBELE corpus, providing professionally recorded human speech for 73 core languages.
- Track 2 (Dialect Adaptation): Addresses the "long-tail dialect gap" using a Rewrite-then-Synthesize strategy. An LLM (Qwen3-Instruct) rewrites standard text into dialectal vernacular (lexical adaptation), which is then synthesized using CosyVoice 3.0 with instruction-aware prompts to inject specific phonological characteristics (e.g., tonal variations, retroflexion) for 19 Chinese dialects.
- Track 3 (Long-Tail Languages): For languages lacking human recordings, high-fidelity Neural Text-to-Speech (TTS) via Edge-TTS is employed. This track enforces speaker diversity (Male/Female) to mitigate overfitting.
Quality Assurance: A three-step validation protocol ensures reliability:
- Physical Integrity: Automated checks for file corruption and duration.
- Semantic Consistency: An ASR loop (using tools like Qwen3-ASR, Whisper) transcribes synthetic audio back to text to calculate Word/Character Error Rates (WER/CER), filtering out hallucinations.
- Human-in-the-Loop: Native speakers verify prosody and lexical usage. The study reports a strong correlation ( $r=0.83$ ) between model performance on synthetic dialect data and real human recordings, validating the synthetic data as a reliable proxy.

Key Contributions

Scale and Diversity: PolySpeech-100 is the most linguistically diverse speech comprehension benchmark to date, covering 110 variants with a specific focus on under-represented dialects and low-resource languages.
Hybrid Construction Pipeline: The paper proposes a validated methodology for augmenting gold-standard human recordings with instruction-driven synthetic speech, effectively bridging the data scarcity gap for dialects without sacrificing evaluation rigor.
Comprehensive Evaluation: The benchmark evaluates 22 state-of-the-art models, including proprietary APIs (Gemini-3, GPT-Audio), open-source E2E models (Qwen2.5-Omni, Fun-Audio-Chat, MiMo-Audio), and traditional Cascade systems (ASR + LLM).

Experimental Results

The evaluation of 22 models yields several pivotal insights:

E2E vs. Cascade on Dialects: For heavy dialects, open-source E2E models significantly outperform traditional ASR pipelines. For instance, Qwen2.5-Omni achieved 78.61% accuracy on Chinese dialects, surpassing the Whisper-v3 + Qwen2.5 pipeline (62.62%). This suggests that direct audio processing preserves critical paralinguistic cues (intonation, stress) often lost during transcription, which are essential for semantic disambiguation in dialects.
Low-Resource Performance Gap: While commercial models like Gemini-3-flash maintain robustness across low-resource languages (84.61% accuracy), open-source models suffer catastrophic degradation (e.g., dropping to ~30-40%). This highlights a critical frontier for the open-source community.
Chain-of-Thought (CoT) Degradation: Counter-intuitively, applying standard Chain-of-Thought prompting frequently degrades performance for most Speech-LLMs in zero-shot settings. Models like Qwen2.5-Omni and Fun-Audio-Chat saw drops of ~10% in high-resource languages when forced to generate reasoning steps. This indicates a modality alignment gap, where intermediate text generation decouples the model from acoustic cues, leading to reasoning hallucinations.
Robustness: Models generally handle speed variations better than environmental noise. However, high noise levels (0-5dB SNR) cause significant performance drops, particularly for open-source models on low-resource languages.

Significance

PolySpeech-100 establishes a rigorous standard for the next generation of inclusive, omni-capable Speech-LLMs. The paper argues that to achieve truly inclusive speech understanding, the field must transition from a "transcribe-then-read" paradigm to native acoustic reasoning. The findings suggest that current architectures struggle with modality alignment when forced to generate intermediate text reasoning, pointing to a need for training curricula that explicitly align audio representations with complex reasoning paths. By providing a comprehensive dataset and evaluation framework, PolySpeech-100 aims to guide the community toward models that can handle the full spectrum of human linguistic diversity, from standard languages to regional dialects and low-resource tongues.