Lamer-SSL: Layer-aware Mixture of LoRA Experts for Continual Multilingual Expansion of Self-supervised Models without Forgetting

1. Problem Statement

Self-supervised learning (SSL) models (e.g., Wav2vec 2.0, HuBERT) have achieved remarkable success in speech processing. However, extending these models to new languages via continual training faces two critical challenges:

• Catastrophic Forgetting: Adapting a model to new languages often degrades performance on previously learned languages.
• Language Interference & Scalability: Traditional multilingual training (joint training on all languages) is computationally expensive, requires reprocessing entire datasets, and suffers from interference where high-resource languages suppress low-resource ones.
• Inefficiency of Existing Adaptation: While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA reduce forgetting, they often lack the capacity to handle complex multilingual interference. Conversely, standard Mixture-of-Experts (MoE) approaches often allocate experts uniformly across layers, ignoring the hierarchical nature of SSL representations where shallow layers capture acoustic features and deep layers capture semantics.

2. Methodology: Lamer-SSL

The authors propose Lamer-SSL, a parameter-efficient framework that integrates a Layer-Aware Mixture of LoRA Experts (Lamer) module with a Replay Strategy.

A. Architecture: Layer-Aware Mixture of LoRA Experts

Instead of replacing the entire Feed-Forward Network (FFN) in Transformer blocks, Lamer-SSL replaces the FFN with a specialized module containing:

1. Frozen Backbone: The original weights ($W_0$) remain frozen to preserve pre-trained knowledge.
2. LoRA Experts: A set of low-rank adapters ($E_k$) where each expert specializes in distinct linguistic patterns.
3. Router: A mechanism that dynamically selects the top-$K$ experts for each input token based on routing probabilities.
4. Layer-Aware Allocation: Unlike uniform MoE, Lamer-SSL assigns experts hierarchically:
   • Shallow Layers: Assigned fewer experts (capturing language-independent acoustic features like pitch/energy).
   • Deep Layers: Assigned more experts (capturing language-dependent semantic and linguistic information).
   • Configuration: In the experiments, experts are allocated in groups of 6 layers with a progressive count of 2, 4, 6, and 8 experts from shallow to deep layers.

B. Replay Strategy

To mitigate catastrophic forgetting without storing the entire historical dataset, the framework employs a replay strategy:

• During continual training on new languages, a small subset of samples (e.g., 100 hours of English) from previously learned languages is interleaved with new data.
• This allows the model to revisit prior language distributions, maintaining stability without requiring access to the full original training corpus.

C. Training Objective

The training loss combines two components:

1. Masked Prediction Loss ($L_{mask}$): Standard SSL objective predicting cluster indices for masked frames.
2. Load Balancing Loss ($L_{lb}$): An auxiliary loss to prevent "expert collapse" (where the router assigns all tokens to a single expert). It encourages consistency between routing probabilities and actual token assignments, ensuring all experts are utilized effectively.

3. Key Contributions

1. Lamer-SSL Framework: A novel parameter-efficient continual training framework that combines LoRA efficiency with MoE flexibility for multilingual expansion.
2. Layer-Aware Expert Allocation: A design that aligns expert capacity with the hierarchical nature of SSL representations, assigning more specialized capacity to deeper semantic layers.
3. Scalable Continual Learning: The ability to extend models to new languages incrementally with only 2.14% of parameters being trainable, avoiding the need for full retraining or massive data upsampling.
4. Comprehensive Validation: Extensive experiments demonstrating that the method outperforms both multilingual pretraining baselines and standard LoRA baselines.

4. Experimental Results

The authors evaluated Lamer-SSL on Automatic Speech Recognition (ASR) and Language Identification (LID) tasks using Mandarin, Cantonese, and English.

• ASR Performance (Character Error Rate - CER):
  • Lamer-SSL achieved an average CER of 10.50% on CommonVoice and 10.13% on Fleurs.
  • It outperformed the HuBERT-LoRA baseline by ~0.6–0.7% and significantly surpassed the mHuBERT-147 (multilingual pretraining) baseline, despite starting from an English-only model and using far fewer resources.
• LID Performance (Accuracy):
  • Lamer-SSL achieved a state-of-the-art average accuracy of 99.22%, surpassing HuBERT-LoRA (98.80%) and mHuBERT-147 (97.21%).
• Parameter Efficiency: Only 2.14% of the total parameters were trainable (LoRA weights + Router), while the backbone remained frozen.
• Ablation Studies:
  • Removing the Replay Strategy caused severe performance degradation on the English language (CER jumped from 9.9% to 26.9%), confirming its critical role in preventing forgetting.
  • Removing Layer-Aware Allocation or Load Balancing Loss also resulted in noticeable performance drops, validating the necessity of both components.
• Expert Activation Analysis: Visualizations showed that shallow layers exhibited shared activation across languages (acoustic features), while deep layers showed distinct, language-specific expert activation patterns, confirming the effectiveness of the hierarchical allocation strategy.

5. Significance

Lamer-SSL addresses the scalability and efficiency bottlenecks in multilingual speech processing. By proving that incremental, parameter-efficient adaptation can outperform massive joint pretraining, the paper offers a practical solution for continuously expanding SSL models to new languages and dialects as they emerge. The approach minimizes computational costs, eliminates the need for reprocessing historical data, and effectively solves the trade-off between retaining old knowledge and acquiring new language capabilities.