Less is More: Adapting Text Embeddings for Low-Resource Languages with Small Scale Noisy Synthetic Data

This paper demonstrates that fine-tuning multilingual text embedding models on a small amount of noisy synthetic data is surprisingly effective for low-resource languages, achieving performance comparable to models trained on massive datasets while challenging the assumption that large-scale, high-quality data is necessary for semantic alignment.

The Gist

Imagine you want to teach a brilliant, multilingual librarian (an AI model) how to understand a specific, rare language like Armenian. Usually, to do this, you'd think you need a massive library of perfect, human-written books in that language, or you'd need to hire a team of expert translators to create millions of perfect sentence pairs. It's expensive, slow, and often impossible for languages with few speakers.

This paper says: "Actually, you don't need a library. You just need a few messy, handwritten notes."

Here is the story of their discovery, explained simply:

1. The Problem: The "Perfect Data" Myth

For years, AI researchers believed that to make a computer understand a low-resource language (like Armenian, Georgian, or others with unique scripts), you needed massive amounts of perfect data. They thought, "If the translation isn't perfect, the AI will get confused."

2. The Experiment: The "Messy Reddit" Approach

The researchers decided to test a different idea. Instead of hiring experts, they took a huge pile of English posts from Reddit (titles and their corresponding stories) and asked a standard AI translator to turn them into Armenian.

The Result? The translation was terrible.

• The grammar was broken.
• The punctuation was wrong.
• Some words were translated literally and made no sense.
• It was "noisy" data—like trying to learn a language by listening to a radio with static.

They took this "garbage" data, cleaned it up just a tiny bit to make sure the meaning was still there, and used it to teach the AI.

3. The "Less is More" Surprise

Here is the magic part:

• They trained the AI on just 10,000 of these messy pairs.
• Then, they tried training it on 1 million pairs (100 times more data).

The Result: The AI trained on just 10,000 messy pairs performed just as well (and sometimes better) than the one trained on 1 million pairs.

The Analogy:
Imagine you are trying to teach someone to recognize a specific type of dog (a Golden Retriever).

• The Old Way: You show them 1,000,000 perfect, high-definition photos of Golden Retrievers in a studio.
• The New Way: You show them 10,000 blurry, slightly tilted, poorly lit photos of Golden Retrievers taken by a shaky hand.

Surprisingly, the person learns to recognize the dog just as well from the blurry photos. Why? Because the core shape of the dog is there. Once the brain (or AI) understands the basic shape, showing it more blurry photos doesn't help much. In fact, showing too many might just confuse them.

4. Why This Matters

• It's Cheap: You don't need to hire expensive human translators. You can use free, open-source AI tools to generate the data.
• It's Fast: You only need a tiny dataset (10k examples) instead of millions.
• It Works for Others: They tested this on Georgian (another language with a unique script) and it worked there too.

5. The Big Takeaway

The paper proves that for AI, semantic alignment (teaching the AI that "dog" in English means "dog" in Armenian) doesn't require perfection. It just requires the gist to be correct.

The AI is like a resilient student. It doesn't need a perfect textbook; it just needs enough "rough drafts" to figure out the pattern. Once it figures out the pattern, it stops learning from new data and just gets confused.

In short: If you want to build AI tools for rare languages, stop waiting for perfect data. Grab some messy, imperfect data, use a little bit of it, and you'll get amazing results. Less is truly More.

Technical Summary

1. Problem Statement

Low-resource languages (LRLs), such as Armenian, suffer from a lack of high-quality, large-scale datasets required to train effective text embedding models. This scarcity hinders the deployment of critical AI applications like Retrieval-Augmented Generation (RAG) and semantic search in these languages.

• Current Limitations: Traditional adaptation strategies rely on either pre-training from scratch on massive monolingual corpora (computationally expensive) or using extensive, pristine parallel datasets to project LRL representations into high-resource semantic spaces (often unavailable).
• The Gap: There is a prevailing assumption that effective semantic alignment requires massive datasets or human-verified, high-quality translations. This paper challenges that assumption.

2. Methodology

The authors propose a minimalist, cost-effective adaptation strategy that leverages small-scale, noisy synthetic data generated via Large Language Model (LLM) translation.

A. Data Generation & Filtering

• Source: The team utilized approximately 2 million (title, body) pairs from Reddit, a dataset known for diverse semantic structures.
• Translation: These English pairs were translated into Armenian using Gemma-2-27B-it (an open-weights model). The resulting translations were intentionally "noisy," containing grammatical errors, lexical ambiguities, and literal translations.
• Quality Control: Instead of manual correction, the team applied a rigorous filtering pipeline using the multilingual E5-base (mE5) model to ensure semantic consistency:
  1. Semantic Drift Check: Ensured the cosine similarity between the original English pair and the translated Armenian pair remained within a threshold (difference < 0.05).
  2. Translation Drift (Query/Passage): Ensured the similarity between the original English text and its Armenian translation remained high (> 0.85).
• Result: A curated corpus of ~1 million samples, from which subsets (10k to 1M) were drawn for training.

B. Training Setup

• Base Model: multilingual-e5-base (mE5) was selected as the primary base model due to its strong performance and size-efficiency trade-off. EmbeddingGemma was used for generalization testing.
• Fine-Tuning: Full fine-tuning was performed on 16x MI250x GPUs with a batch size of 512, learning rate of 7e-5, for 5 epochs.
• Model Averaging: A "Merged" checkpoint was created by averaging the fine-tuned weights with the original base model weights to mitigate catastrophic forgetting.

C. Evaluation Benchmark

The authors constructed a comprehensive benchmark for Armenian, including:

1. MTEB [hye]: Standard classification and bitext mining tasks.
2. Manually Curated Retrieval: 185 high-quality, native Armenian query-document pairs (Gold Standard).
3. MS MARCO (Translated): A 10k subset translated via Gemini 2.0 Flash.
4. STS (Translated): A 3k subset of Semantic Textual Similarity data.

3. Key Contributions

1. The "Less is More" Phenomenon: Demonstrated that fine-tuning a multilingual encoder on just 10,000 noisy synthetic pairs yields state-of-the-art (SOTA) results, matching or exceeding models trained on 1 million examples.
2. Robustness to Noise: Proved that semantic alignment for LRLs is highly robust to translation errors. The model learns effectively even when the data is grammatically imperfect, provided the semantic relationship (topic) is preserved.
3. Comprehensive Ablation Studies: Showed that increasing data scale (up to 1M), improving translation quality (using high-quality curated data), or diversifying domains (e.g., Wikipedia) yields no significant gains over the minimal 10k noisy baseline.
4. Generalizability: Validated the approach on Georgian (a different language family with a unique script) and EmbeddingGemma (a decoder-based architecture), confirming the pipeline's broad applicability.
5. Open-Source Release: Released the fine-tuned models, the noisy synthetic dataset, and the new Armenian benchmark to democratize LRL embedding development.

4. Key Results

• Performance Leap: On the critical Retrieval task (Manually Curated), the model improved from a baseline of 58.15 to 79.35 (Main) and 78.80 (Merged), representing a >35% relative improvement.
• Average Gains: The 10k noisy baseline achieved an 11-12% average improvement across all benchmark tasks compared to the base model.
• Data Scaling: Scaling data from 10k to 1M resulted in marginal gains (<1%) or even slight performance degradation in unmerged models, indicating that semantic alignment saturates early.
• Quality vs. Quantity: High-quality, curated data performed worse or similarly to the noisy 10k baseline on retrieval and MS MARCO tasks, suggesting that semantic diversity (found in Reddit) is more critical than grammatical perfection.
• Architecture Independence: While mE5 (bidirectional encoder) outperformed EmbeddingGemma (decoder-based) in absolute scores, EmbeddingGemma also showed significant gains with the 10k noisy data, confirming the method's architecture-agnostic nature.

5. Significance and Implications

• Democratization of AI: This work lowers the barrier to entry for LRL communities. Researchers no longer need massive computational resources or expensive human-annotated parallel corpora to build SOTA retrieval systems.
• Paradigm Shift: It challenges the dogma that "clean" data is a prerequisite for high performance. The findings suggest that for strong multilingual baselines, the semantic space is already structured; the adaptation task is merely a surface-level alignment of tokens, which noisy data can achieve efficiently.
• Practical Application: The approach enables the rapid creation of high-performance text embedders for resource-constrained communities using only open-weights LLMs and public English datasets.
• Future Research: The paper opens new avenues for investigating how much "noise" different language families can tolerate and how this applies to languages with complex morphology or shared scripts with high-resource neighbors.

In conclusion, the paper establishes that for Low-Resource Languages, minimal, imperfect, synthetic data is sufficient to achieve state-of-the-art semantic alignment, rendering the pursuit of massive, pristine datasets unnecessary for many adaptation tasks.