Technical Summary: SMEAR-MoE for Multilingual ASR

1. Problem Statement

Recent advancements in Automatic Speech Recognition (ASR) have established a paradigm where frozen speech encoders are connected to Large Language Models (LLMs) via lightweight projectors. While effective for monolingual tasks, this architecture faces a significant bottleneck in multilingual settings. A single monolithic projector struggles to align the diverse acoustic-to-semantic mappings required for typologically distinct languages (e.g., Indo-Aryan vs. Dravidian), forcing representational compromises.

Existing alternatives present their own trade-offs:

• Language-specific projectors improve per-language accuracy but hinder cross-lingual knowledge sharing.
• Standard Mixture-of-Experts (MoE) designs enable specialization but often suffer from instability and expert collapse, where poor gradient flow causes only a few experts to dominate training, rendering the model ineffective.

2. Methodology: SMEAR-MoE

The authors propose SMEAR-MoE (Stabilized Mixture-of-Experts for ASR), a projector architecture designed to ensure dense gradient flow to all experts, thereby preventing collapse while enabling cross-lingual sharing.

Architecture Overview

Building on the SLAM-ASR framework, the system connects a frozen Whisper large-v3 encoder to a frozen Gemma-2-9B LLM. The trainable component is the projector, which is redesigned as follows:

1. Shared Downsampler: A shared convolutional layer ($D$) reduces the sequence length of the encoder's hidden states ($H_S$) to an intermediate representation ($Z_S$).
2. Expert MLPs: A set of lightweight Multi-Layer Perceptron (MLP) experts ($\{P_m\}_{m=1}^M$) operate on $Z_S$.
3. Stabilized Routing (The Core Innovation):
   • Unlike standard MoEs that use hard gating (selecting top-$k$ experts per token or utterance), SMEAR-MoE employs soft gating.
   • A gating network produces token-level probabilities ($G$). These are averaged to obtain utterance-level weights ($\bar{g}$).
   • Instead of routing to specific experts, the model constructs a differentiable virtual expert by merging all expert parameters weighted by $\bar{g}$:
     $$\bar{W} = \sum_{m=1}^M \bar{g}_m W_m, \quad \bar{b} = \sum_{m=1}^M \bar{g}_m b_m$$
   • This virtual expert is applied to the input. Crucially, this ensures every expert receives a dense gradient signal proportional to its gating weight, eliminating the "expert collapse" phenomenon common in sparse MoEs.

Experimental Setup

• Languages: Four mid-resource Indic languages: Hindi, Marathi, Tamil, and Telugu.
• Data: Trained on ~250 hours per language from IndicVoices and IndicSUPERB. Evaluated on the VISTAR Benchmark (Kathbath, MUCS, IndicTTS, Fleurs).
• Baselines: Compared against a monolithic projector, static multi-projector variants (Language-Specific, Tied, Dense Ensemble), and standard top-$k$ MoE strategies (Utterance-Level and Token-Level).

3. Key Results

Experiments across four distinct test sets demonstrate that SMEAR-MoE achieves the best overall performance:

• Performance Gains: SMEAR-MoE delivers up to a 7.6% relative Word Error Rate (WER) reduction compared to the single-projector baseline. It consistently outperforms hard-gating MoE models and static ensembles.
• Efficiency: Despite its multi-expert design, SMEAR-MoE maintains runtime efficiency comparable to the single projector baseline.
  • Real-Time Factor (RTF): SMEAR-MoE achieves an RTF of 0.198, nearly identical to the single projector (0.196).
  • In contrast, the Dense Ensemble (static averaging of all experts) is slower (RTF 0.243) due to higher parameter counts.
• Parameter Efficiency: The MoE design totals 52M parameters, which is more efficient than the Dense Ensemble (72M) while offering dynamic specialization.

4. Analysis of Routing Behavior

A critical finding of the paper is the interpretability of the learned routing. The gating mechanism spontaneously learns linguistically meaningful specializations without explicit instruction:

• Indo-Aryan Languages (Hindi & Marathi): These languages, which share the Devanagari script and linguistic family, predominantly favor the same expert (e.g., Expert 4), indicating the model exploits their acoustic and structural commonalities.
• Dravidian Languages: Tamil is consistently routed to a distinct, specialized expert. Telugu, while also Dravidian, exhibits a more distributed routing pattern, reflecting its distinct script and phonological divergence from Tamil.

5. Significance and Claims

The paper claims that SMEAR-MoE addresses the critical bottleneck of projector stability in multilingual LLM-based ASR. Its primary contributions are:

1. Mitigation of Expert Collapse: By ensuring dense gradient flow through parameter merging, SMEAR-MoE prevents the instability and under-utilization of experts that plague conventional MoEs in low-to-mid resource settings.
2. Scalability and Robustness: It offers a scalable solution that balances expert specialization with cross-lingual sharing, outperforming both monolithic and static ensemble approaches.
3. Interpretability: The model learns routing strategies that align with known linguistic families, suggesting that stabilized multi-expert projectors can provide interpretable insights into how ASR systems process diverse languages.

The authors conclude that stable multi-expert projectors are a key direction for developing scalable, efficient, and robust multilingual ASR systems powered by LLMs.