🤖 AI

Breaking the Curse ofMultilinguality inMany-to-Many Speech-to-Text Translation via a Resource-AwareMixture of Speech Encoders

The paper introduces MSRT, a novel framework utilizing a resource-aware Mixture of Speech Encoders and a five-stage curriculum learning strategy to overcome the curse of multilinguality in speech-to-text translation, achieving state-of-the-art performance across 45 languages with minimal data requirements while simultaneously improving low-resource capabilities without compromising high-resource performance.

Original authors: Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Chengpeng Fu, Yu Wang, Ming Liu

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Chengpeng Fu, Yu Wang, Ming Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a bustling, noisy international train station. This is the world of Speech-to-Text Translation, a field of computer science dedicated to teaching machines to listen to someone speaking one language and instantly write down what they said in another. For a long time, these machines were like clumsy tour guides who spoke perfect English but stumbled over everything else. They worked by first transcribing the speech into text (like a stenographer) and then translating that text (like a dictionary), but this two-step process often led to errors piling up, like a game of "telephone" gone wrong. Recently, scientists have tried to build "Multimodal Large Language Models" (MLLMs)—super-smart AI brains that can hear and speak. The dream is a single, magical brain that can handle any language. But here's the catch: when you try to teach one brain to speak 45 different languages at once, it gets confused. It's like trying to cram 45 different dialects into a single backpack; the heavy, well-practiced languages (like English or Spanish) take up all the space, leaving the lighter, less-practiced languages (like rare dialects) with no room to breathe. This problem is called the "curse of multilinguality," and it means the AI gets really good at some languages but terrible at others.

Enter MSRT, a new framework proposed by researchers Yexing Du and their team, designed to fix this backpack problem. Instead of forcing one brain to do all the heavy lifting, they built a smart system with two specialized "ears" (called speech encoders) and a traffic controller (a router). Think of it like a high-end restaurant kitchen. Usually, one head chef tries to cook every dish on the menu, from simple toast to complex soufflés. If the kitchen gets too busy, the soufflés might burn because the chef is too focused on the toast. MSRT changes the game: it has a "Master Chef" who is frozen in time, perfect at cooking the popular, high-resource dishes (like English), and a "Trainee Chef" who is eager to learn and specializes in the tricky, low-resource dishes. A smart waiter (the router) looks at the order ticket (the language being spoken) and instantly sends the request to the right chef. If the order is for a common language, the Master Chef handles it without changing a thing. If it's a rare language, the Trainee Chef steps up, adapting their skills specifically for that dish.

The researchers tested this idea on 45 different languages, covering everything from common giants like Chinese and Spanish to smaller languages like Khmer and Lao. They didn't just test a few; they checked every possible combination, meaning they tried translating from every one of the 45 languages to every other one (a total of 1,980 different directions). The results were impressive: their 4-billion-parameter model (which is relatively small compared to giants with 27 or 30 billion parameters) achieved the highest average scores, beating much larger models and even a top-tier commercial API. Crucially, this system didn't just help the rare languages; it actually made the common languages perform even better, proving that you don't have to sacrifice quality in one area to fix another. They also found that they only needed about 10 hours of paired speech-and-text data per language to train the system effectively, which is a tiny amount compared to what other models usually require.

The paper explicitly argues against the idea that a single, shared speech encoder is the best way to handle many languages. They show that when you force one encoder to do everything, it creates a "zero-sum game" where improving low-resource languages often hurts high-resource ones. By using their "Mixture of Speech Encoders" (MoSE), they broke this curse. The system uses a five-stage training strategy, kind of like a video game with levels that get harder one by one, starting with simple speech recognition and slowly adding translation tasks. This step-by-step approach helped the model learn without getting overwhelmed.

In their experiments, the MSRT model showed consistent gains across the board. For low-resource languages, the improvement was the most dramatic, lifting scores significantly where other models struggled. But unlike previous attempts that might have boosted the rare languages at the cost of the common ones, MSRT strengthened the high-resource languages too. The authors suggest that this happens because the "frozen expert" preserves the strong knowledge of the popular languages, while the "trainable expert" focuses its energy on the languages that need more help. They also noted that while the model is very good, its translation quality is ultimately limited by the underlying "brain" (the Large Language Model) it uses, meaning if the base model doesn't know a language well, the speech system can't magically invent that knowledge. However, for the languages it does cover, the results suggest a new, more efficient way to build multilingual AI that treats all languages with respect, rather than letting the loud ones drown out the quiet ones.

Technical Summary: Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via MSRT

1. Problem Statement

Multimodal Large Language Models (MLLMs) have advanced speech-to-text translation (S2TT), yet they suffer from the "curse of multilinguality" when processing inputs from diverse languages. Current architectures typically employ a single shared speech encoder for all input languages. As language coverage expands, this design forces languages with varying resource levels (high, medium, and low) to compete for limited representation capacity.

This competition leads to a performance imbalance:

High-resource languages (e.g., English, Chinese) often retain strong performance due to preserved pretrained acoustic knowledge.
Low-resource languages suffer substantial degradation because the joint optimization overwrites established representations or fails to fit scarce-language features.
Consistency Gap: Models often exhibit strong English-centric performance (English-to-X) but degrade significantly in many-to-many translation (Non-English-to-Non-English), creating a pronounced cross-lingual performance gap.

2. Methodology: The MSRT Framework

The authors propose MSRT (Multilingual Speech-to-Text Translation), a framework designed to break this curse by decoupling resource preservation from resource adaptation. The core innovation is a Resource-Aware Mixture of Speech Encoders (MoSE).

A. Mixture of Speech Encoders (MoSE)
Instead of a single shared encoder, MSRT utilizes two homogeneous Whisper-based encoders with asymmetric roles, routed explicitly by the source language:

Frozen Expert ( $E_f$ ): Handles high-resource languages. Its parameters are fixed to preserve general speech representations and prevent catastrophic forgetting of pretrained acoustic knowledge.
Trainable Expert ( $E_t$ ): Handles medium- and low-resource languages. Initialized with the same weights as $E_f$ , this expert is fine-tuned to adapt to underrepresented accents and phonological patterns.
Explicit Language Router: A source-language token in the prompt determines which expert processes the utterance. This design avoids the need for learned gating mechanisms or balancing losses, keeping inference costs equivalent to a single encoder.

B. Five-Stage Curriculum Learning
To achieve effective alignment with minimal data (only 10 hours of paired S2TT data per language), the authors introduce a five-stage training strategy:

ASR Alignment: Training the speech adapter (Q-Former + MLP) on multilingual ASR data while keeping encoders and the LLM frozen.
Expert Specialization: Activating MoSE routing. The frozen expert preserves high-resource capabilities, while the trainable expert adapts to medium/low-resource speech.
Translation Activation: Introducing Speech-Guided Machine Translation (SMT) with ground-truth transcriptions to connect speech features to the frozen LLM's translation knowledge.
End-to-End SRT: Removing ground-truth transcriptions from the input, forcing the adapter to generate both transcription and translation directly from routed speech.
Joint Adaptation: Unfreezing the LLM (via LoRA) to jointly optimize with the adapter, adapting multilingual generation to acoustic inputs without updating the full pretrained LLM weights.

C. Architecture Overview
The system consists of the MoSE module, a speech adapter (Q-Former for feature compression and MLP for dimension alignment), and a pretrained LLM (MiLMMT-4B). The prompt includes explicit source and target language tokens to control encoder selection and generation direction, respectively.

3. Key Contributions

MoSE Architecture: A resource-aware mixture-of-experts design that breaks the curse of multilinguality by preserving high-resource knowledge while enabling targeted adaptation for under-resourced languages.
MSRT-4B Model: A compact 4-billion-parameter model capable of supporting all translation directions among 45 languages (covering 11 language families) using only 10 hours of paired S2TT data per language.
Comprehensive Evaluation: A systematic evaluation of all 1,980 translation directions (45 source × 44 target) on the FLEURS benchmark, demonstrating broad and reliable many-to-many translation capabilities.
Open Source: Release of code and models to facilitate reproducible research.

4. Experimental Results

The model was evaluated on the FLEURS dataset (45 languages) and the CoVoST-2 benchmark.

Overall Performance: MSRT-4B achieved a state-of-the-art average COMET score of 83.3, outperforming significantly larger baselines including MCAT-27B (80.9), Qwen3-Omni-30B (77.5), and SeamlessM4T-V2-Large (72.5). It also surpassed the Gemini-3.5-Flash-Lite API (80.0).
Directional Consistency: MSRT-4B achieved COMET scores of ≥80 on 1,552 out of 1,980 directions (78.4%). In comparison, the Gemini API achieved this on 928 directions, and SeamlessM4T on only 117.
Resource-Level Gains:
- Low-Resource: MoSE provided the largest gains for low-resource languages (+3.0 COMET points), significantly narrowing the performance gap with high-resource languages.
- High-Resource: Unlike previous methods that trade high-resource performance for low-resource gains, MoSE improved high-resource performance (+1.4 COMET points) while simultaneously boosting low-resource results.
Parameter Efficiency: Despite having only 4B parameters, MSRT-4B outperformed 27B and 30B parameter models, demonstrating efficient use of model capacity.
Generalization: The model showed robust cross-dataset generalization on CoVoST-2 and strong zero-shot transfer capabilities.

5. Significance and Claims

The paper claims that MSRT successfully breaks the curse of multilinguality without compromising high-resource performance. By explicitly routing languages based on resource levels to specialized encoders, the framework resolves the interference caused by competing resource demands in a single shared encoder.

The authors emphasize that future speech translation models should prioritize comprehensive multilingual optimization rather than focusing predominantly on English-centric directions. They note that their approach achieves robust many-to-many translation with minimal data requirements, challenging the notion that massive data scales are strictly necessary for high-quality multilingual S2TT.

Limitations: The authors acknowledge that MSRT's translation quality is ultimately bounded by the machine translation capabilities of the underlying pretrained LLM, particularly for language pairs weakly represented during the LLM's pretraining phase.