⚡ electrical engineering

MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting

MiLorE-SSL is a lightweight framework that enables efficient continual multilingual training in self-supervised speech models by combining LoRA modules with a soft mixture-of-experts mechanism and limited replay data to achieve strong performance across new and existing languages while mitigating catastrophic forgetting with only 2.14% trainable parameters.

Original authors: Jing Xu, Minglin Wu, Xueyuan Chen, Xixin Wu, Helen Meng

Published 2026-01-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jing Xu, Minglin Wu, Xueyuan Chen, Xixin Wu, Helen Meng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, multilingual robot named HuBERT. This robot is amazing at understanding English because it spent years listening to thousands of hours of English radio and podcasts. However, it doesn't speak a word of Mandarin or Cantonese.

Usually, if you wanted to teach this robot a new language, you'd have two bad options:

The "Hard Reset": Throw away all its English knowledge, feed it a massive mix of English, Mandarin, and Cantonese data, and retrain it from scratch. This is like burning down a library to build a new one. It's expensive, slow, and you lose the old books.
The "Cramming Session": Try to teach it Mandarin while it's still studying English, but without any special care. The robot gets confused, mixes up the languages, and ends up forgetting how to speak English properly. This is called "catastrophic forgetting."

The paper introduces a new, clever solution called MiLorE-SSL. Think of it as a "smart upgrade kit" that lets the robot learn new languages without forgetting the old ones, using very little extra energy.

Here is how it works, broken down into simple analogies:

1. The "LoRA" Modules: Adding Sticky Notes

Instead of rewriting the robot's entire brain (which is huge and frozen), the researchers attach tiny, lightweight "sticky notes" to specific parts of its brain.

The Analogy: Imagine the robot's brain is a giant, locked encyclopedia. You can't change the printed pages. But, you can stick small, writable notes (called LoRA) on the pages.
How it helps: When the robot encounters Mandarin, it only writes on these specific notes. It doesn't need to relearn the whole encyclopedia. This saves a massive amount of computing power (only 2.14% of the brain is actually being trained).

2. The "Soft Mixture-of-Experts" (MoE): The Smart Switchboard

The robot has a special "switchboard" (a router) that decides which sticky notes to use for which language.

The Analogy: Imagine a busy restaurant kitchen. Instead of having one chef try to cook every dish (which leads to mistakes), you have a team of specialists.
- Hard Routing (The old way): The manager points at Chef A and says, "You only cook Italian." If the customer orders Chinese, Chef A is confused.
- Soft Routing (MiLorE's way): The manager says, "Chef A, you handle 70% of this order, and Chef B, you handle 30%."
How it helps: This "soft" approach allows the robot to share knowledge. Maybe the robot uses the same "sticky note" for the sound of a vowel in English and Mandarin because they sound similar. It lets the robot be flexible, borrowing ideas from one language to help with another, rather than keeping them in separate, rigid boxes.

3. The "Replay" Strategy: The Flashback

Even with the sticky notes and the switchboard, the robot might still start to forget its English skills while learning Mandarin.

The Analogy: Think of it like studying for a new exam. If you only study the new material, you might forget the old stuff. So, you take a few minutes every day to quickly review a few old flashcards.
How it helps: The researchers feed the robot a tiny, random sample of English audio (about 100 hours) while it learns Mandarin. This is a "flashback" that reminds the robot, "Hey, don't forget how to speak English!" Crucially, they don't need the entire history of English data, just a small, manageable sample.

The Results: What Happened?

The researchers tested this on a robot that originally only knew English. They taught it Mandarin and Cantonese using the MiLorE-SSL kit.

The Robot got better at English: Surprisingly, by using this method, the robot actually got better at English than before, not worse.
The Robot learned new languages: It became very good at Mandarin and Cantonese, far outperforming other models that were trained from scratch on all three languages at once.
Efficiency: It did all this while only "training" about 2% of its total brain.

The Bottom Line

MiLorE-SSL is a way to teach speech models new languages continuously. It uses tiny, efficient add-ons (LoRA), a flexible sharing system (Soft MoE) to mix knowledge, and small memory refreshes (Replay) to prevent forgetting. It proves you don't need to burn down the library to add new books; you just need a smart way to annotate the existing ones.

Technical Summary: MiLorE-SSL

Problem Statement
Self-supervised learning (SSL) has revolutionized speech representation learning, yet multilingual SSL models face significant constraints. Existing models are typically limited to the languages encountered during pretraining. Incorporating new languages usually requires computationally expensive retraining from scratch. Alternatively, sequential training without mitigation strategies often results in "catastrophic forgetting," where performance on previously learned languages degrades severely. While prior solutions like data replay require large historical corpora and storage, and adapter-based modules scale poorly with increasing language counts, there is a need for a framework that enables efficient, continual multilingual training without discarding prior knowledge or incurring prohibitive costs.

Methodology
The authors propose MiLorE-SSL (Mixture of LoRA Experts), a lightweight framework designed for the continual extension of SSL models. The architecture modifies standard Transformer-based SSL models (specifically HuBERT) by replacing the standard Feed-Forward Network (FFN) in each block with a MiLorE module. This module consists of three components:

Frozen Backbone: The original FFN weights ( $W_0$ ) remain frozen to preserve prior multilingual knowledge.
LoRA Experts: A set of $N$ experts, each parameterized as low-rank matrices ( $\Delta W = BA$ ) using Low-Rank Adaptation (LoRA). These introduce a minimal number of trainable parameters.
Soft Router: A learnable mechanism that computes soft routing weights ( $p$ ) based on input hidden states. Unlike hard routing, this allows inputs to be distributed across multiple experts, promoting flexible knowledge sharing and specialization without requiring explicit language identifiers.

The output of the module is a weighted sum of the frozen backbone output and the expert outputs: $o = W_0h_{in} + \sum_{i=1}^{N} p_i \cdot E_i(h_{in})$ .

To further mitigate catastrophic forgetting, the framework employs a replay strategy. Instead of relying on massive historical corpora, it utilizes a small, randomly sampled subset of data (e.g., 100 hours) from existing languages during the training of new languages. This allows the model to revisit prior linguistic distributions while adapting to new patterns.

Key Contributions

Novel Framework: MiLorE-SSL is the first framework to combine LoRA with a soft Mixture-of-Experts (MoE) mechanism specifically for the continual multilingual extension of SSL models.
Soft Expert Routing: The design introduces a soft routing mechanism that balances knowledge sharing and specialization across languages, reducing cross-lingual interference.
Efficient Forgetting Mitigation: The integration of a replay strategy using minimal existing-language data reduces forgetting without the need for access to full historical datasets.
Parameter Efficiency: The approach achieves high performance with only 2.14% of the total parameters being trainable.

Experimental Results
Experiments were conducted on the ML-SUPERB benchmark, focusing on monolingual Automatic Speech Recognition (ASR) and Language Identification (LID) tasks. The study utilized a HuBERT-Large model pretrained on English and continually trained it on Mandarin and Cantonese, using English data for replay.

Performance Gains: MiLorE-SSL significantly outperformed both the baseline English-only HuBERT-Large and the multilingual mHuBERT-147. On the CommonVoice dataset, it reduced Character Error Rates (CER) to 10.3% (English), 10.7% (Mandarin), and 11.0% (Cantonese), compared to mHuBERT-147's 18.5%, 15.5%, and 14.8% respectively.
Language Identification: The model achieved an average LID accuracy of 99.40%, surpassing mHuBERT-147 (97.21%) and HuBERT-Large (94.94%).
Ablation Studies: Removing the MoE component or the replay strategy resulted in performance degradation across all tasks. Specifically, the removal of MoE confirmed the necessity of expert specialization, while removing replay primarily harmed English performance, validating its role in preventing forgetting.
Expert Activation: Analysis of expert activation patterns revealed that lower layers encode language-independent features, while middle and higher layers exhibit language-specific divergence, validating the utility of the MoE structure for modeling both shared and distinct linguistic features.

Significance and Claims
The paper claims that MiLorE-SSL effectively addresses the trade-off between computational efficiency and multilingual capability in SSL models. By combining low-rank adaptation with soft MoE and a lightweight replay strategy, the framework enables models to incrementally learn new languages while retaining strong performance on existing ones. The authors emphasize that this is achieved without the rigid data balancing requirements of static training or the massive storage needs of traditional replay methods. The results demonstrate that it is possible to extend SSL models to new languages with minimal trainable parameters (2.14%) while avoiding catastrophic forgetting, offering a scalable solution for global linguistic diversity.