💬 NLP

Bootstrapping Embeddings for Low Resource Languages

This paper demonstrates that leveraging large language models through adapter composition and cross-lingual finetuning (XL-LoRA) to generate synthetic triplet data effectively bridges the resource gap, enabling the creation of high-performing embedding models for low-resource languages where supervised data is scarce.

Original authors: Merve Basoz, Andrew Horne, Mattia Opper

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Merve Basoz, Andrew Horne, Mattia Opper

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the meaning of sentences in different languages. To do this well, the robot needs a "dictionary of feelings" called embeddings. Think of these embeddings like a giant, multi-dimensional map where sentences with similar meanings are parked right next to each other, and sentences with opposite meanings are parked far apart.

For languages like English, we have huge, expensive libraries of human-written examples (triplets) to teach the robot how to draw this map. But for hundreds of other languages (low-resource languages), these libraries simply don't exist. It's like trying to teach someone to drive a car in a country where no one has ever built a road map before.

This paper asks: Can we use super-smart AI (Large Language Models) to draw these missing maps for us?

The authors tried three different ways to use AI to "fake" the training data, and here is how they did it, using some everyday analogies:

The Three Strategies

1. The "Tourist with a Phrasebook" (In-Context Learning)

The Idea: You ask the AI, "Here is a sentence in English. Please write a similar sentence and a different sentence in Swahili," and you show it a few examples first.
The Result: It's like giving a tourist a phrasebook and asking them to navigate a foreign city. They can get by, but they often sound awkward, mix up words, or get the tone wrong.
The Paper's Finding: This method was okay, but the AI struggled to speak the target language fluently. It often produced sentences that sounded robotic or grammatically broken, like a tourist trying to order food but accidentally ordering a shoe.

2. The "Specialized Intern" (Adapter Composition)

The Idea: Instead of just asking the AI to guess, we give it a tiny, specialized "training module" (an adapter). We train one module to understand the logic of the task (what makes a sentence similar or different) using English data, and another module to understand the grammar of the target language. Then, we "merge" these two modules together.
The Result: This is like hiring a brilliant logic professor and a native speaker, then merging their brains into one super-intern.
The Paper's Finding: This worked much better! The AI could generate high-quality sentences. However, because the logic and language parts were just glued together, the final map wasn't perfectly smooth. The AI sometimes struggled to realize that two very different-looking sentences actually meant the same thing.

3. The "Translator's Secret" (XL-LoRA) — The Winner!

The Idea: This is the cleverest trick. The authors realized that AI models are actually "bilingual" deep down, even if they only speak English out loud. They decided to train the AI using a hybrid approach:

The Anchor (The Prompt): The sentence is in the target language (e.g., Hindi).
The Answer (The Generation): The AI is allowed to generate the "similar" and "different" sentences in English.

The Analogy: Imagine you are teaching a student who speaks Hindi. You give them a Hindi sentence and ask, "What is a similar sentence?" But instead of forcing them to answer in Hindi (where they might struggle), you let them answer in English (where they are experts). You then use that English answer to teach the robot.
Why it works: It bypasses the AI's weakness (generating fluent text in rare languages) and leverages its strength (understanding complex logic in English). It's like using a high-quality English blueprint to build a house in a remote village, rather than trying to draw the blueprint in a language the architect doesn't know well.

The Paper's Finding: This method (XL-LoRA) was the clear winner. It produced the best maps, beating even the methods that used human-annotated data from other languages. It created embeddings that were accurate, smooth, and ready for real-world use.

The Big Takeaway

For a long time, if you wanted to build a smart AI for a language like Swahili or Hindi, you were stuck because you couldn't afford to hire thousands of humans to write training data.

This paper proves that we don't need to. By using a clever "translation trick" (XL-LoRA), we can use the AI's own English superpowers to generate the training data it needs to understand other languages.

In short: We found a way to bootstrap the intelligence of AI for the world's most spoken but least "digitally represented" languages, opening the door for better translation, search, and chatbots for billions of people who were previously left out.

1. Problem Statement

Modern Natural Language Processing (NLP) relies heavily on high-quality sentence embeddings for tasks like retrieval-augmented generation, semantic matching, and classification. The state-of-the-art (SOTA) embedding models are typically trained using supervised contrastive learning (e.g., SimCSE), which requires large datasets of human-annotated triplets: an anchor sentence, a positive (semantically similar) sentence, and a hard negative (semantically dissimilar but confusing) sentence.

The Gap: While such datasets are abundant for high-resource languages like English, they are virtually non-existent for hundreds of low-resource languages (LRLs), despite these languages having millions of speakers.
Limitations of Existing Solutions:
- Unsupervised methods: Perform significantly worse than supervised methods.
- Cross-lingual transfer: Transferring knowledge from English to LRLs helps but remains suboptimal compared to having native-language supervision.
The Opportunity: Can Large Language Models (LLMs) be used to generate high-quality synthetic triplet data to "bootstrap" the training of embedding models for LRLs, effectively bridging the resource gap?

2. Methodology

The authors propose a pipeline where an LLM generates synthetic triplets, which are then used to fine-tune a transformer encoder (e.g., XLM-R or mmBERT) using the supervised SimCSE objective. They investigate three distinct strategies for optimizing the LLM data generator:

A. In-Context Learning (Prompting)

Approach: Follows the SynCSE methodology (Zhang et al., 2023b). The LLM is prompted with few-shot examples (in English) and instructed to generate positive/negative pairs for a given anchor sentence in the target language.
Goal: Test if simple prompting is sufficient for LRLs.

B. Adapter Composition

Approach: Uses LoRA (Low-Rank Adaptation) to fine-tune the LLM generator. It leverages AdamergeX, a technique that composes multiple adapters to combine capabilities.
Mechanism:
1. Task Adapter: Trained on English NLI data to learn the logic of generating entailment/contradiction pairs.
2. Language Adapters: Two generic adapters trained on the Aya dataset (one source, one target) to ensure grammatical competency in the target language.
3. Composition: The adapters are merged using the formula: $T_{tgt} = T_{src} + \lambda(L_{tgt} - L_{src})$ . This removes source-language artifacts while injecting target-language fluency.
Goal: Optimize the generator specifically for the dual task of semantic reasoning and target-language fluency.

C. XL-LoRA (Cross-Lingual LoRA)

Approach: A novel method that exploits the internal cross-lingual alignment of LLMs.
Mechanism:
- The generator is trained to produce English positives and negatives while the anchor remains in the target language.
- Rationale: LLMs often struggle to generate fluent text in low-resource languages due to a bias toward English. By keeping the generation target in English (where the model is strongest) but the input anchor in the target language, the model leverages its internal shared concept space without the penalty of generating low-quality target-language tokens.
- Data Construction: Uses high-quality human-translated XNLI data. The English anchor is swapped with a target-language anchor, while the positive/negative pairs remain in English.
Goal: Bypass the "generation bottleneck" of LRLs by keeping the generation task in a high-resource language.

3. Key Contributions

Systematic Evaluation of Synthetic Data: The first comprehensive study applying synthetic data generation specifically for embedding models in low-resource languages, moving beyond standard English benchmarks.
Novel XL-LoRA Technique: Introduction of a cross-lingual training regime that decouples the anchor language from the generation language, achieving SOTA results without requiring parallel data in the target language.
Adapter Composition for Data Synthesis: Demonstrating that composing task-specific and language-specific adapters improves data quality over simple prompting.
Scalable Pathway: Proving that high-performing embedding models can be created for diverse languages using only small-scale fine-tuning datasets (10k–20k examples) and no target-language supervision for the generator.

4. Experimental Results

The authors evaluated the generated data on Semantic Textual Similarity (STS) and Information Retrieval (MTEB/MIRACL) tasks across multiple low-resource languages (e.g., Afrikaans, Hindi, Marathi, Telugu, Hausa, Korean).

Performance Hierarchy:
1. XL-LoRA: Consistently achieved the best performance, outperforming all baselines (including cross-lingual transfer) and the prompting method. It often matched or exceeded the performance of models trained on human-annotated data.
2. Adapter Composition: Showed strong, consistent gains over the cross-lingual baseline but was slightly less effective than XL-LoRA.
3. In-Context Prompting: Outperformed unsupervised baselines but failed to surpass the cross-lingual transfer baseline. It suffered from fluency issues, code-switching, and poor semantic alignment in LRLs.
Qualitative Findings:
- Prompting often resulted in ungrammatical sentences and high lexical overlap between anchors and negatives (lazy negation).
- XL-LoRA produced embeddings with better alignment (similar sentences cluster together) and uniformity (even distribution in vector space) compared to other methods.
Scaling: Increasing the training data for XL-LoRA from 10k to 20k examples yielded further performance gains, suggesting scalability.

5. Significance and Conclusion

This paper provides a scalable, cost-effective solution to the lack of embedding resources for low-resource languages.

Paradigm Shift: It challenges the notion that high-quality embeddings require massive amounts of human-annotated data in the target language.
Practical Impact: The XL-LoRA approach allows researchers and practitioners to bootstrap performant models for virtually any language, provided they have a high-quality multilingual LLM and a small set of high-quality translations (for the training set construction), without needing target-language supervision for the generator.
Future Work: The authors note that scaling to larger LLMs and exploring decoder-based embeddings could further improve results, but the current pipeline offers a robust foundation for democratizing NLP technologies for under-resourced communities.