💬 NLP

In-context Language Learning for Endangered Languages in Speech Recognition

This paper demonstrates that large language models can effectively learn unseen endangered languages for speech recognition through in-context learning, where probability-based methods with relevant text samples outperform traditional instruction-based approaches and achieve performance comparable to dedicated models without sacrificing original capabilities.

Original authors: Zhaolin Li, Jan Niehues

Published 2026-01-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaolin Li, Jan Niehues

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, multilingual robot (a Large Language Model, or LLM) that knows thousands of languages like a human encyclopedia. But, there's a catch: this robot has never heard of a few specific, rare languages that are on the verge of disappearing. If you ask it to translate or recognize speech in these rare languages, it usually fails because it hasn't "read" enough books about them.

This paper asks a simple question: Can we teach this robot a new language on the fly, just by showing it a few examples, without retraining its entire brain?

The answer is yes, and here is how they did it, explained with some everyday analogies.

The Problem: The "Blank Slate" Robot

Think of the robot as a chef who knows how to cook Italian, Chinese, and French perfectly. But if you hand them a recipe for a rare, endangered dish from a tiny village they've never visited, they are clueless. They might guess, but they'll likely get it wrong.

In the world of speech recognition, this means the robot can hear the sounds of a rare language but doesn't know what words those sounds represent.

The Solution: "In-Context Language Learning" (ICLL)

Instead of sending the robot back to culinary school for years (which is called "fine-tuning" and takes a lot of time and data), the researchers tried In-Context Learning.

The Analogy: Imagine you are taking a test on a language you don't speak. The teacher gives you a cheat sheet with 50 sentences from that language and their meanings right next to them. You haven't studied the language for years, but by looking at those 50 examples, you can suddenly guess the meaning of a new sentence much better than if you had no cheat sheet at all.

The researchers did exactly this. They gave the robot a "cheat sheet" (a few hundred sample sentences) right before asking it to recognize speech in the rare language.

The Secret Sauce: Choosing the Right Cheat Sheet

The researchers found that how you choose the cheat sheet matters more than just having any cheat sheet.

Random vs. Relevant: If you give the robot random sentences, it's like giving a chef random ingredients from a grocery store. It doesn't help much.
The "Matchmaker" Strategy: The researchers built a system that acts like a matchmaker. When the robot needs to recognize a specific sentence, the system scans the "cheat sheet" and picks the examples that look or sound most similar to that specific sentence.
- Analogy: If the robot needs to translate a sentence about "fishing," the system picks examples about "fishing" from the cheat sheet, rather than examples about "cooking."
Audio vs. Text: They tried using the sound of the voice to find matches, but it didn't work well. It was like trying to find a book in a library by humming the tune of the story; it's too vague. Using the text (the written words) was much more effective.

The Results: The Robot Learns Fast

They tested this on four very different, endangered languages (Khinalug, Kichwa, Mboshi, and Japhug). Here is what happened:

Better than Guessing: The robot, with its "cheat sheet," became much better at understanding the language than it was on its own.
Beating the Specialists: In some cases, this "on-the-fly" learning was just as good as, or even better than, building a brand-new, specialized robot from scratch just for that one language.
The "Probability" Trick: They found that instead of asking the robot, "Which of these words is correct?" (which is like asking a student to raise their hand), it was better to let the robot calculate the math probabilities of the words itself. This "math-based" approach worked much better than the "instruction-based" approach.

The Limitations (The Fine Print)

The paper is careful to note a few boundaries:

Open Source Only: This trick works best with robots where you can see the "gears" (the code and math). It's harder to do with "black box" robots where you can't see how they think.
Specific Languages: They only tested four languages. While it worked well for those, we don't know if it works for every rare language in the world.
One Robot: They tested this on one specific model (Llama 3). Other models might behave differently.

The Bottom Line

This paper shows that you don't need a massive library of data to teach a super-smart AI a new, rare language. You just need to show it the right few examples at the right time. It's like giving a genius a quick reference guide instead of making them read a whole encyclopedia, and it turns out to be a very powerful way to help preserve and understand endangered languages.

Technical Summary: In-context Language Learning for Endangered Languages in Speech Recognition

Problem Statement
State-of-the-art Automatic Speech Recognition (ASR) systems perform well on high-resource languages but struggle with endangered languages due to data scarcity and limited linguistic representation in pre-trained models. While Large Language Models (LLMs) have shown promise in learning new tasks via In-Context Learning (ICL) without parameter updates, their application to low-resource languages remains challenging. Existing approaches like Cross-lingual ICL (X-ICL) suffer from translation biases, and standard ICL often fails due to insufficient pre-training data for specific endangered languages. Furthermore, the integration of language information into ASR for these languages is under-explored, particularly regarding how LLMs can learn entirely new languages (rather than just new tasks) through in-context examples to improve speech transcription.

Methodology
The authors propose In-context Language Learning (ICLL), a framework where LLMs acquire linguistic knowledge of a target endangered language using only a few hundred sample sentences provided within the prompt, rather than through fine-tuning. The study focuses on ASR as a re-ranking task:

Hypothesis Generation: An initial ASR system (Wav2Vec2 mms-300m) generates an $n$ -best list of candidate transcriptions for target audio.
Sample Retrieval Strategies: To construct the in-context prompt, the authors investigate various strategies for selecting training samples:
- Corpus-level vs. Example-specific: Comparing a static set of samples for the entire test set versus selecting unique samples for each test instance.
- Query Types: Using the best ASR hypothesis (hyp), the top- $k$ hypotheses (topk), or the ground-truth transcript (oracle) as the query for similarity search.
- Embedding Modalities: Utilizing text embeddings (SONAR model) and audio embeddings to find similar training examples based on cosine similarity.
Hypothesis Selection: The LLM is used to re-rank the $n$ $n$ -best list. Two approaches are compared:
- Instruction-based: Explicitly instructing the LLM to choose the best option.
- Probability-based: Calculating the log-probability of each hypothesis given the in-context samples ( $S = \sum \log P_{AM} + \sum \log P_{LM}$ ) and selecting the highest scoring one. This approach integrates acoustic model probabilities with the LLM's language model probability.

Key Contributions

ICLL for ASR: The paper demonstrates that LLMs can learn a new, unseen endangered language using only a few hundred sample sentences to perform ASR re-ranking, without requiring supervised training data or parameter updates.
Superiority of Probability-based Approaches: The study finds that probability-based hypothesis selection significantly outperforms traditional instruction-based methods for low-resource languages, likely because the latter struggles to internalize linguistic knowledge from limited examples.
Sample Selection Importance: The research highlights that example-specific sample selection (tailoring samples to each test instance) yields better performance than corpus-level selection. Furthermore, using the initial ASR hypothesis as a retrieval query is nearly as effective as using the ground-truth oracle transcript, despite the high error rates of the initial ASR system.
Performance vs. Dedicated Models: The proposed ICLL approach achieves language modeling and ASR performance comparable to, or in some cases surpassing, dedicated language models (n-gram and Transformer-based) trained specifically on the target language corpora, while preserving the LLM's original capabilities.

Results
Experiments were conducted on four linguistically diverse endangered languages: Khinalug, Kichwa, Mboshi, and Japhug.

Language Modeling: Example-specific retrieval strategies consistently reduced perplexity compared to corpus-level selection and random baselines. Probability-based selection with acoustic integration yielded the best results.
ASR Performance: ICL-based re-ranking outperformed acoustic-only selection across all languages. For Kichwa and Mboshi, ICLL surpassed dedicated n-gram and Transformer language models. For Khinalug and Japhug, performance was comparable to these dedicated models.
Sample Size: Performance generally improved with more in-context samples (up to 200-300), though computational constraints limited the upper bound in some experiments.
Modality: Text-based embeddings proved superior for sample retrieval; acoustic embeddings or combined audio-text embeddings did not improve performance and sometimes degraded it.

Significance and Claims
The paper claims that ICLL offers a viable, resource-efficient pathway for adapting powerful LLMs to endangered languages where traditional fine-tuning is infeasible due to data scarcity. By leveraging in-context examples, LLMs can effectively learn the phonetic and syntactic structures of unseen languages to improve speech recognition. The authors emphasize that this approach allows for dynamic adaptation (selecting different samples for different test cases) and avoids the computational cost of training new models. However, the paper modestly notes limitations: the probability-based method requires access to model parameters (limiting it to open-source LLMs), and the findings are based on a single LLM (Llama-3-8B) and four specific languages, leaving generalizability to other models and languages as an open question.