Technical Summary: MoLGE – Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition

Problem Statement

Scaling automatic speech recognition (ASR) to hundreds of languages within a unified model faces a fundamental tension between language scalability and resource accessibility. Current approaches rely on uniform scaling of model capacity (billions of parameters) and training data (millions of hours), which is computationally prohibitive and often impractical. Furthermore, these dense models suffer from the curse of multilinguality, where fixed model capacity is diluted across diverse languages, leading to performance degradation, particularly for low-resource languages. Existing Mixture-of-Experts (MoE) and Parameter-Efficient Fine-Tuning (PEFT) methods typically assign experts to individual languages, a strategy that becomes unscalable when dealing with hundreds or thousands of languages. Additionally, prior work often neglects the structured nature of linguistic relationships, treating languages as independent entities rather than leveraging their phonological, morphological, and genealogical similarities.

Methodology: MoLGE

The authors propose MoLGE (Mixture of Language Group Experts), a framework designed to enable structured language specialization by integrating linguistic relationships directly into the expert allocation mechanism. The approach is built upon Speech Self-Supervised Models (S3Ms) and incorporates two primary design principles:

1. Hierarchical Speech-Oriented Adaptation

MoLGE decomposes the S3M architecture into lower acoustic layers and upper linguistic layers, applying distinct adaptation strategies to each:

• Lower Layers (Shared Acoustics): Since lower layers capture universal, language-agnostic acoustic features, MoLGE applies shared Low-Rank Adaptation (LoRA) modules here. This models global acoustic variability without introducing routing overhead.
• Upper Layers (Language-Specific Experts): Higher layers encode language-specific information. MoLGE deploys a Mixture of LoRA Experts (MoLE) exclusively in these layers.
• Shared Expert for Paralinguistics: To handle paralinguistic factors (e.g., speaker identity, speaking rate) that remain invariant across languages, a shared expert is introduced. This allows language-specific experts to focus more effectively on distinct linguistic representations.
• Attentive Pooling Router: Given that speech signals contain temporally imbalanced information (e.g., silences), the framework employs an Attentive Statistical Pooling (ASP) classifier as the routing mechanism. This emphasizes salient acoustic regions to ensure the router identifies informative representations for expert assignment.

2. Structured Language Grouping Strategies

Instead of assigning experts to individual languages, MoLGE clusters related languages and assigns shared experts to each group. The paper investigates six grouping strategies across three categories to determine the optimal injection of linguistic priors:

• Implicit Grouping: Groupings emerge naturally from the router's learned behavior without predefined labels. Variants include frame-level (token-level) and utterance-level routing.
• Embedding-Guided Grouping: Clusters are derived from high-dimensional representations of pretrained speech models. Two variants are tested:
  • SSL-based: Using embeddings from a general self-supervised model (OmniASR-W2V 1).
  • LID-based: Using embeddings from a language identification model (MMS-LID-4017).
  • Clustering is performed using Agglomerative Hierarchical Clustering (AHC) with Ward's linkage.
• Knowledge-Guided Grouping: Clusters are constructed using URIEL+ language vectors, which integrate curated linguistic databases. Two variants are tested:
  • Geographic: Based on spatial proximity and regional contact.
  • Genetic: Based on genealogical relationships (language families and subfamilies).

3. Two-Stage Dense-to-Sparse Upcycling

The framework utilizes a two-stage training strategy inspired by LLM upcycling:

1. Dense Training: A dense ASR model is trained on top of a pretrained S3M to establish robust global representations.
2. Sparse Upcycling: The MoLGE framework is introduced to transform the dense model into a sparse, parameter-efficient architecture. LoRA is applied to all linear layers (attention projections and Feed-Forward Networks) to enable expert specialization across the predefined language groups.

Key Contributions

1. MoLGE Framework: An efficient, scalable framework for massively multilingual ASR that incorporates language structure as an architectural prior for expert specialization, moving beyond uniform scaling.
2. Speech-Aware Architecture: Integration of speech modeling principles into LoRA and MoE, specifically through hierarchical adaptation, shared experts for paralinguistic factors, and attentive pooling routers.
3. Systematic Study of Grouping: A comprehensive evaluation of six language grouping strategies across 495 languages. The study demonstrates that explicit linguistic priors (knowledge-guided and embedding-guided) consistently outperform implicit or random priors, and that their benefits scale with the complexity of the target linguistic space.

Results

Experiments were conducted on a benchmark encompassing 495 languages using two backbone models: LAMA-UT (phonetic modeling via Romanized transcription) and Omnilingual ASR (orthographic modeling via language-specific scripts).

• Performance Gains: MoLGE consistently outperforms dense multilingual baselines with a minimal increase in trainable parameters.
  • Phonetic Domain (LAMA-UT): Achieved a 11.5% relative reduction in Character Error Rate (CER).
  • Orthographic Domain (OmniASR): Achieved a 19.0% relative reduction in CER.
• Grouping Effectiveness: Explicit grouping strategies (particularly LID-based and Geographic knowledge-guided) yielded substantial improvements over implicit and random grouping. The benefits were more pronounced in the orthographic domain, suggesting that structured language specialization is crucial for modeling complex linguistic structures.
• Low-Resource Generalization: MoLGE significantly narrowed the performance gap between low-resource and high-resource languages. In the orthographic domain, CER for low-resource languages dropped from 42% to 30%, and confidence intervals became narrower, indicating more consistent performance across resource levels.
• Ablation Studies: Removing shared experts or hierarchical adaptation led to performance degradation, confirming the necessity of these components for effective acoustic modeling and expert specialization.

Significance and Claims

The paper claims that structured language specialization provides an effective pathway for massively scaling the language coverage of multilingual ASR systems. By shifting from uniform capacity scaling to structured architectural design informed by language relationships, MoLGE alleviates the tension between scalability and resource accessibility.

The authors posit that:

1. Linguistic Priors Matter: Explicitly leveraging linguistic structure (via grouping) is more effective than relying solely on data-driven router learning, especially in massively multilingual settings.
2. Complexity-Dependent Benefits: The efficacy of language-aware modeling increases with the complexity of the target linguistic space (e.g., orthographic vs. phonetic).
3. Knowledge Transfer: Grouping languages by shared linguistic properties facilitates knowledge transfer, allowing low-resource languages to benefit from the training signals of related, higher-resource languages within the same expert group.

Ultimately, the work presents a unified framework that enables scalable, parameter-efficient multilingual speech recognition without the prohibitive computational costs of scaling dense models to billions of parameters.