⚡ electrical engineering

MEUSLI: a Multilingual Projector for LLM-based ASR and Beyond

The paper introduces MEUSLI, the first open-science multilingual projector family that connects a Whisper speech encoder with open-source LLMs to enable scalable, end-to-end automatic speech recognition and other speech understanding tasks across 28 European languages.

Original authors: Lorenzo Concina, Seraphina Fong, Marco Matassoni, Alessio Brutti

Published 2026-07-27

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Lorenzo Concina, Seraphina Fong, Marco Matassoni, Alessio Brutti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot brain that can read and write any text in the world, but it's completely deaf. It can't hear a single word spoken. On the other hand, you have a super-ear that can hear sounds perfectly but doesn't understand what they mean. For a long time, to make them talk to each other, engineers had to build a clunky middleman: the sound would go to a translator, then to a text converter, and finally to the brain. This was slow, and if the translator made a mistake, the brain would get confused.

Recently, scientists figured out how to build a direct bridge between the "super-ear" and the "super-brain." This bridge is called a projector. Think of it like a universal adapter plug. It takes the raw electrical signals from the sound and instantly translates them into the language the brain understands, allowing the two to chat directly. This is the world of Speech Language Models (SLMs). The big question researchers are asking is: Can we build one of these adapters that works for everyone, not just English speakers? If we can, we could finally give voice to thousands of languages that have been left out of the digital conversation, especially those with very few speakers or not enough recorded data.

Enter MEUSLI, a new project that acts like a master key for 28 European languages. The researchers built a family of these "adapter plugs" that connect a powerful sound-ear (called Whisper) to open-source, multilingual robot brains. Their main finding is that this system works incredibly well, not just for common languages like Spanish or German, but also for rare ones like Breton or Maltese. They discovered that if you start with their pre-made, multilingual adapter, you can teach it a brand new language with just a tiny amount of data—sometimes as little as 46 minutes of audio.

However, there's a catch. If you try to teach this adapter a new language without any care, it tends to "forget" everything it knew about the original 28 languages, like a student who studies for a new test and instantly forgets their old homework. The paper suggests that to fix this, you need to use a technique called "data replay," where you occasionally practice the old languages while learning the new one, keeping the memory alive.

Beyond just writing down what people say (transcription), the team showed that this adapter can be tweaked to do other jobs, like translating spoken words into English or guessing the topic of a conversation (like sports or politics). They found that with only a few hours of specific training for each new task, the system could learn to do these things too.

The paper explicitly rules out the idea that you need massive amounts of data or proprietary (secret) information to build these systems. They argue against the notion that low-resource languages are impossible to support, showing instead that a good starting point (their multilingual projector) makes it possible to "bootstrap" new languages from very little data. While they don't claim to have solved every problem in the world, they provide strong evidence that this approach is scalable, open, and effective. They measured these results using standard tests on real-world data, showing that their method consistently beats starting from scratch, especially for the hardest, rarest languages.

In short, MEUSLI is a toolkit that proves we can build inclusive, open-source voice technology that doesn't just work for the few, but for the many, turning a deaf robot brain into a polyglot listener with the help of a clever, lightweight adapter.

Technical Summary: MEUSLI – A Multilingual Projector for LLM-based ASR and Beyond

1. Problem Statement

While Large Language Models (LLMs) have revolutionized text-based NLP, extending these capabilities to speech for natural multimodal interaction remains challenging. Traditional speech-enabled LLMs rely on cascaded pipelines (ASR $\to$ LLM $\to$ TTS), which suffer from error propagation, loss of paralinguistic cues, and increased latency. Although recent unified Speech Language Models (SLMs) attempt to connect speech encoders directly to LLMs via trainable projectors, existing solutions are predominantly English-centric, limited to high-resource languages, or rely on proprietary data and weights. There is a significant gap in open-science, multilingual SLMs that can effectively support low-resource languages and offer reproducible, scalable architectures for diverse European languages.

2. Methodology

The paper introduces MEUSLI (Multilingual EU Speech LInear projector), a framework designed to bridge pre-trained speech encoders with open-source multilingual LLMs.

Architecture: MEUSLI adopts the SLAM-ASR architecture. It utilizes a frozen Whisper-large-v3-turbo encoder to convert audio into high-dimensional acoustic features. These features are mapped into the LLM's embedding space via a lightweight linear projector. The LLMs used are EuroLLM 1.7B-Instruct, EuroLLM 9B, and Apertus-8B. Both the encoder and the LLM remain frozen during training; only the projector (and optionally LoRA adapters on the LLM) are trained.
Projector Design: The projector consists of a single hidden layer with ReLU activation followed by a regression layer, containing approximately 17.31M trainable parameters. Low-Rank Adaptation (LoRA) is applied to the LLM with rank $r=8$ and scaling factor $\alpha=32$ to introduce an additional 1.38M tunable parameters.
Training Data: The model is trained on 7,622 hours of audio spanning 28 European languages (including English, French, German, Italian, and low-resource languages like Breton, Maltese, and Basque). Data is sourced from Common Voice 17.0, FLEURS, and VoxPopuli. To manage data imbalance, audio samples are capped at 100K per language per dataset.
Training Configuration: Audio is converted to 128-bin mel spectrograms. Speech embeddings are downsampled by a factor of $k=5$ . Training uses the Adam optimizer with an initial learning rate of $1 \times 10^{-4}$ , a linear warmup of 1,000 steps, and cross-entropy loss on LLM outputs.
Continual Learning: To address catastrophic forgetting when adding new languages, the authors employ data replay strategies, interleaving a buffer of samples from original languages during fine-tuning on new ones.

3. Key Contributions

First Open Multilingual Projector Family: MEUSLI is the first fully open-source projector family linking Whisper to open-source multilingual LLMs, supporting 28 European languages. The projectors are released on HuggingFace.
Low-Resource Adaptation: The paper demonstrates that MEUSLI serves as a robust initialization point for fine-tuning on low-resource languages. It shows that multilingual pretraining allows for effective bootstrapping of new languages with minimal data (e.g., 2.5 hours for Breton, 46 minutes for Albanian).
Continual Learning for New Languages: The authors show that new languages can be added using state-of-the-art continual learning methodologies (data replay) without collapsing performance on previously learned languages.
Beyond ASR: The framework is extended to Speech Translation (ST) and Topic Identification (TID). The authors demonstrate that the multilingual projector can be adapted to these tasks with only a few hours of task-specific supervision per language-task pair.

4. Results

Multilingual ASR Performance: MEUSLI achieves strong Word Error Rate (WER) results across 28 languages. High-resource languages show low WERs, while medium-resource languages show consistent improvements. While very low-resource languages (e.g., Breton, Irish) remain challenging, MEUSLI outperforms monolingual systems trained from scratch in these scenarios.
Fine-tuning Efficiency:
- Breton (2.5h data): Fine-tuning MEUSLI reduced WER from 95.7% (multilingual baseline) to 24.5% (with a specialized encoder), outperforming a monolingual system trained from scratch (27.7%).
- Albanian (46min data): Fine-tuning MEUSLI achieved 75.61% WER, whereas a projector trained from scratch failed to converge (389.12% WER).
Catastrophic Forgetting Mitigation: Naive fine-tuning on Ukrainian caused WER on base languages (e.g., Spanish, Polish) to degrade significantly (e.g., Spanish from 4.09% to 90.34%). Using a data replay buffer (1,000 samples per language) restored base language performance to near-original levels while maintaining Ukrainian performance (17.56% WER).
Multitask Transfer: When extended to ST and TID with less than 5 hours of data per language-task pair, MEUSLI-bootstrapped fine-tuning significantly outperformed training from scratch. For example, on Finnish, ASR WER dropped from 53.0% (scratch) to 18.6% (MEUSLI), and Topic ID accuracy rose from 57.5% to 81.7%.

5. Significance and Claims

The paper claims that MEUSLI provides a solid foundation for multilingual speech understanding tasks, supporting scalable and inclusive open-source SpeechLLMs.

Scalability: The approach proves that multilingual pretraining enables effective knowledge transfer, allowing functional ASR systems to be bootstrapped for unseen languages with extremely limited data.
Openness: By using fully open-source models (Whisper, EuroLLM, Apertus) and open datasets, the work addresses the lack of transparency and reproducibility in current SLM literature.
Versatility: The system is not limited to transcription; it serves as a strong starting point for other speech processing tasks (translation, classification) in data-scarce environments.

The authors conclude that while language-specific fine-tuning remains advantageous for ASR, multilingual fine-tuning provides robust shared semantic representations for downstream understanding tasks, laying the groundwork for universal, LLM-powered speech technologies.