⚡ electrical engineering

Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling

This paper proposes a projector-based LLM-ASR framework that leverages a Mixture of Experts architecture for cross-lingual adaptability and a Continuous Integrate-and-Fire mechanism for dynamic downsampling, achieving substantial performance improvements over strong baselines in multilingual speech recognition.

Original authors: Guodong Lin, Ziqi Chen, Yuxiang Fu, Ke Li, Wei-Qiang Zhang

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guodong Lin, Ziqi Chen, Yuxiang Fu, Ke Li, Wei-Qiang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, multilingual translator (a Large Language Model, or LLM) who can speak and understand almost any language in the world. However, this translator is "deaf" to sound; they only understand text. Your goal is to build a system that lets this translator listen to spoken words and write them down accurately. This is the challenge of Automatic Speech Recognition (ASR).

The paper you shared describes a new way to connect the "ears" (the audio processor) to the "brain" (the LLM) so they work together perfectly, especially when dealing with many different languages and speaking speeds.

Here is the breakdown of their solution using simple analogies:

The Problem: A Rigid Connection

In previous attempts, the connection between the audio processor and the translator was like a rigid, fixed-size conveyor belt.

The Issue: Speech is messy. Sometimes people talk fast, sometimes slow. Sometimes a sentence is short, sometimes long.
The Old Way: The old system tried to chop up the audio into equal, fixed chunks (like cutting a loaf of bread into slices of exactly the same size) before handing it to the translator. If the speaker talked fast, the slices were too small and lost information. If they talked slow, the slices were too big and wasted space. This made the system struggle with different languages and speeds.

The Solution: Two Upgrades

The authors introduced two clever upgrades to fix this:

1. The "Specialist Team" (Mixture of Experts / MoE)

Instead of using one single, generic translator to handle all the audio, they built a team of specialists.

How it works: Imagine a busy airport. Instead of one tired agent trying to check in passengers for 11 different countries, you have a team of agents. One is an expert in French, another in Japanese, another in Spanish.
The Magic: A smart "gatekeeper" (a gating mechanism) looks at the incoming sound and instantly routes it to the right specialist. If the audio sounds like Korean, it goes to the Korean expert. If it sounds like German, it goes to the German expert.
The Result: Because each "expert" only focuses on the specific patterns of their language, the system becomes much better at understanding different accents and languages than a single "jack-of-all-trades" model.

2. The "Smart Timer" (Continuous Integrate-and-Fire / CIF)

This fixes the problem of the "rigid conveyor belt" mentioned earlier.

How it works: Instead of chopping the audio into fixed sizes, the system uses a smart timer that listens to the flow of speech.
The Mechanism: Think of it like filling a bucket with water. The system drops "drops" of audio information into a bucket. As soon as the water level hits a specific line (a threshold), the system says, "Okay, that's enough for one word!" and passes that word to the translator. Then it starts filling the bucket again for the next word.
The Benefit: If someone speaks quickly, the bucket fills up fast, and words are passed quickly. If they speak slowly, the bucket fills slowly. This creates a perfect, flexible match between the sound and the text, regardless of how fast the person is talking.
The Twist: The authors found that the original "Smart Timer" was sometimes too eager, filling the bucket too fast and losing details. So, they tweaked the rules (a "relaxed" version) to ensure the bucket fills at just the right pace, keeping all the important details while still matching the text length perfectly.

The Results

The authors tested this new system on a massive dataset covering 11 different languages (including English, French, Japanese, Korean, etc.).

The Baseline: The standard system struggled, making many mistakes.
The New System: By combining the "Specialist Team" (MoE) and the "Smart Timer" (CIF), the system made significantly fewer mistakes.
Scaling Up: When they fed the system even more data (8,000 hours of speech instead of 1,500), it became even better at understanding languages it hadn't seen as much before, proving it is very good at generalizing to new situations.

In Summary

The paper claims that by giving the AI a team of language specialists and a flexible, smart timer to match sound to text, they created a speech recognition system that is more accurate, robust, and better at handling many different languages than previous methods. They did not claim this is for medical use or specific future applications, but simply that it is a major step forward in building better, more reliable speech-to-text systems.

Technical Summary: Enhancing Multilingual LLM-Based ASR with Mixture of Experts and Dynamic Downsampling

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) presents a new frontier for Automatic Speech Recognition (ASR), offering superior semantic reasoning and contextual understanding compared to traditional statistical language models. However, effectively integrating LLMs into ASR systems remains challenging. While early approaches utilized LLMs merely as post-processors or external language models, recent "Encoder–Projector–LLM" paradigms have shown promise but suffer from two critical limitations in multilingual settings:

Multilingual Generalization: Aligning speech encoders with multilingual LLMs requires flexible mapping to handle diverse phonetic and linguistic patterns. Naive integration often fails to achieve robust cross-lingual adaptability.
Modality Alignment: Existing projector designs typically rely on fixed downsampling rates to compress acoustic features into text tokens. This rigidity makes performance highly sensitive to speech rate variations and lacks explicit alignment modeling, leading to hallucinations and degraded robustness when audio and text sequence lengths do not match naturally.

2. Methodology

The authors propose a novel projector-based LLM-ASR framework that addresses these challenges through two primary architectural enhancements: a Mixture of Experts (MoE) projector and a Continuous Integrate-and-Fire (CIF) based dynamic downsampling mechanism.

2.1. Baseline Architecture

The system follows the standard Encoder–Projector–LLM paradigm:

Speech Encoder: Uses the pre-trained Whisper-large-v3 encoder to extract acoustic features.
LLM: Employs the frozen Qwen-2.5 7B model as the core decoding module, leveraging its semantic capabilities. Language-specific prompts are used to accelerate convergence.
Projector: Bridges the encoder and LLM, mapping acoustic features to the LLM's embedding space.

2.2. MoE-Enhanced Projector

To improve cross-lingual adaptability, the standard linear projector layers are replaced with an MoE architecture.

Structure: The original convolutional layers serve as a shared backbone. A set of expert sub-networks is added, with the number of experts matching the number of training languages.
Mechanism: A gating mechanism computes activation weights for each expert based on input acoustic features. The final output is a weighted sum of the selected experts:
$y = \sum_{k=1}^{K} g_k(x)E_k(x)$
where $g_k(x)$ is the gating weight and $E_k(x)$ is the expert output. This allows the model to dynamically route inputs to specialized networks, modeling diverse linguistic patterns more effectively.

2.3. CIF-Based Modality Alignment

To replace rigid fixed downsampling, the authors integrate a Continuous Integrate-and-Fire (CIF) mechanism.

Mechanism: CIF assigns a weight (0 to 1) to each acoustic frame. These weights are accumulated, and when the sum exceeds a threshold (e.g., 1), a "fire" event occurs, signaling the detection of a new token. This dynamically determines the output length based on the input audio.
Modification: Standard CIF often leads to over-compression (aggressive downsampling) compared to the baseline's fixed factor of 4, causing information loss. To mitigate this, the authors introduce a relaxed CIF variant. The training objective is modified to encourage the predictor to generate an output sequence $n$ times longer than the target token sequence. In experiments, $n=4$ was chosen to match the baseline's effective downsampling rate while retaining CIF's adaptability to speech rate variations.

3. Experimental Setup

Datasets: Training was conducted on a 1,500-hour multilingual dataset (Nexdata/MLC-SLM Challenge) covering 11 languages (English, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, Vietnamese). An expansion experiment utilized an additional 6,500 hours from CommonVoice, GigaSpeech2, LibriSpeech, MLS, and VoxPopuli (totaling 8,000 hours).
Evaluation: Performance was measured using Word Error Rate (WER) on in-domain (MLCSLM-dev) and out-of-domain (FLEURS-test, CommonVoice-test) benchmarks.
Baselines: Comparisons were made against Whisper-large-v3 and a standard LLM-ASR baseline (frozen encoder/LLM with a simple linear projector and fixed downsampling).

4. Key Results

The experimental results demonstrate that the proposed components significantly enhance performance:

MoE Impact: The MoE-Enhanced Projector alone reduced the WER on the MLCSLM-dev set from 23.26% (Baseline) to 16.10%, outperforming Whisper-large-v3 (21.48%) and showing consistent improvements across most languages.
CIF Impact: Replacing fixed downsampling with standard CIF initially degraded performance (18.95%) due to over-compression. However, the Modified CIF (relaxed variant) restored performance, achieving 15.27% WER on MLCSLM-dev.
Combined Performance: The full proposed method (MoE + Modified CIF) achieved the best results across all datasets:
- MLCSLM-dev: 15.27%
- CommonVoice-test: 13.87%
- FLEURS-test: 10.46%
Data Scaling: Expanding training data to 8,000 hours yielded substantial gains on out-of-domain datasets (CommonVoice-test dropped to 9.86%; FLEURS-test to 8.65%), though it resulted in a slight increase in in-domain WER (15.45%), suggesting the method prioritizes cross-domain generalization.

5. Significance and Claims

The paper claims that the proposed framework represents a significant step toward building more accurate, robust, and generalizable LLM-based ASR systems. The authors emphasize that:

MoE Architectures are effective for handling the complex, diverse mappings required in multilingual ASR, offering superior cross-lingual generalization compared to standard projectors.
Dynamic Downsampling via CIF is crucial for modality alignment, provided that the compression rate is carefully calibrated (via the relaxed variant) to prevent information loss.
Well-Designed Projectors and Alignment Strategies are as critical as the LLM backbone itself for achieving robust performance in multilingual scenarios.

The work concludes that these enhancements successfully address the limitations of naive LLM integration, yielding substantial performance improvements over strong baseline models and paving the way for more reliable multilingual speech recognition systems.