💬 NLP

SEA-Embedding: Open and Reproducible Text Embeddings for Southeast Asia

This paper introduces SEA-Embedding, a fully open and reproducible text-embedding pipeline trained exclusively on publicly available data to address the lack of robustness and reproducibility in Southeast Asian language models, while systematically analyzing key design factors to achieve state-of-the-art performance on the SEA-BED benchmark.

Original authors: Peerat Limkonchotiwat, Raymond Ng, Sarana Nutanong, Jian Gang Ngui

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Peerat Limkonchotiwat, Raymond Ng, Sarana Nutanong, Jian Gang Ngui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library where every book is written in a different Southeast Asian language. Now, imagine you want to build a super-smart librarian who can instantly understand that a story about "rain" in Thai means the same thing as "rain" in Vietnamese, even though the words look completely different. This librarian is called a Text Embedding.

For a long time, the best librarians in the world were built by big tech companies using secret recipes and private data. You couldn't see how they worked, and they often struggled with Southeast Asian languages, treating them like second-class citizens compared to English or Chinese.

The paper you shared introduces SEA-Embedding, a new, completely open-source librarian built specifically for Southeast Asia. Here is how they built it and why it works, explained simply:

1. The Problem: The "Black Box" Librarians

Most top-tier librarians today are "black boxes." We know they are smart, but we don't know exactly what books they read or how they learned. Because their training data is secret, we can't fix them if they make mistakes, and they often fail when asked to understand the diverse dialects and languages of Southeast Asia.

2. The Solution: A Transparent, Open Kitchen

The authors built SEA-Embedding like a recipe book that anyone can use. They didn't use any secret ingredients. They only used data that is already public and free on the internet.

The Goal: To create a librarian that is not only smart but also reproducible. If you want to build your own version, you can follow their exact steps and get the same result.

3. The Three Secret Ingredients (The "Recipe")

The researchers tested three main things to see what makes a librarian truly robust for this region. Think of these as the three pillars of their construction:

A. The Reading List (Data Composition)

To be a good librarian, you need to read a lot of different things.

The Mix: They didn't just read one type of book. They combined 245 million general text pairs (like news and stories to understand the languages broadly) with 14 million instruction texts (like Q&A pairs to understand how to do tasks).
The Analogy: Imagine training a chef. If you only give them a cookbook, they know recipes but can't improvise. If you only give them a list of orders, they know what people want but don't know how to cook. SEA-Embedding gives the model both the cookbook and the orders, so it understands the language and how to use it.

B. The Training Drills (Objective Design)

How do you teach the librarian to be consistent?

Symmetric Contrastive Learning (SCL): This is like a game of "Find the Match." The model is shown two sentences that mean the same thing and told, "These are twins!" It also sees sentences that are different and told, "These are strangers!" They added a special twist called "focal reweighting," which is like giving the teacher extra attention to the students who are struggling the most, rather than just the easy ones.
Similarity Distribution Matching (SDM): This is the "Master Class." The model (the student) tries to copy the behavior of a very strong, pre-existing expert model (the teacher). The student doesn't just copy the answers; it tries to copy the way the teacher thinks about how similar two things are.
The Result: This combination forces the model to create a very organized mental map where similar ideas are always close together, no matter what language they are in.

C. The Starting Point (Base Encoder)

You can start with a smart student or a less experienced one.

The team tested their recipe on different "base" models (some small, some large).
The Finding: No matter which student they started with, the recipe worked. It improved every single one. However, starting with a slightly larger, smarter student (the E5-Large model) gave the best final results.

4. The Results: A New Champion

When they tested their new librarian against the current champions (the "black box" models):

SEA-Embedding won. It achieved the highest average score on a tough test called SEA-BED, which covers 10 different Southeast Asian languages and 9 different types of tasks.
It's especially good at the hard stuff: It performed significantly better on lower-resource languages (like Lao and Khmer) compared to other models that usually favor big languages like Indonesian or Thai.
It's open: Unlike the winners of the past, you can see their code, download their data, and run the experiment yourself.

Summary

The paper claims that by being transparent, using a mix of general and instruction data, and using a specific two-step training method (learning from matches and copying a master teacher), they created the best text embedding model for Southeast Asia to date. It's a model that works better, is fairer to smaller languages, and is open for everyone to inspect and improve.

Technical Summary: SEA-Embedding

Problem Statement

Text embeddings are critical for downstream natural language processing (NLP) applications, yet current state-of-the-art models face two significant limitations regarding Southeast Asian (SEA) languages:

Lack of Reproducibility: Most high-performing models rely on closed or undisclosed training data, preventing systematic analysis and re-training.
Insufficient Robustness: Models that excel on global benchmarks (e.g., MMTEB) often underperform on SEA-specific evaluations (e.g., SEA-BED). Linguistic diversity in the region is high, and performance gains from high-resource settings do not always transfer effectively to SEA languages.

Methodology

The authors introduce SEA-Embedding, a fully open and reproducible text-embedding pipeline trained exclusively on publicly available data. The pipeline is designed to systematically investigate three core factors of robust embedding design: Data Composition, Training Objective, and Base Encoder Initialization.

1. Data Composition (RQ1)

The pipeline utilizes a dual-category dataset totaling 259 million samples:

General Datasets (245M samples): Sourced from resources like FineTranslations and CCMatrix to ensure broad regional coverage across SEA languages.
Instruction Datasets (14M samples): Sourced from multilingual downstream datasets (e.g., MIRACL, SEA-Instruct) with task-specific instructions prepended to enhance task awareness and generalization.
Format: Data is formatted as triplets (anchor, positive, negative) or text pairs (anchor, positive).

2. Training Objective Design (RQ2)

The authors propose a two-stage objective to balance local discrimination with global consistency:

Symmetric Contrastive Learning (SCL): Unlike standard contrastive learning, this stage employs Symmetric InfoNCE with focal reweighting. It treats the anchor-positive and positive-anchor pairs symmetrically and uses a focal weight ( $w_i = (1 - j_i)^\gamma$ ) to emphasize hard examples.
Similarity Distribution Matching (SDM): This stage performs knowledge distillation to match the student model's similarity distribution against a teacher model's distribution over a large memory queue (65,536 entries).
- Teacher Model: A strong pre-trained encoder (Wang et al., 2024c).
- Mechanism: The student mimics the teacher's similarity distribution using KL Divergence ( $L_{sdm} = KL(D_t \parallel D_s)$ ).
- Temperature Design: The student temperature ( $\tau_s$ ) is set lower than the teacher's ( $\tau_t$ ) to create a sharper student distribution, forcing the student to learn a more precise embedding space.

3. Base Encoder Initialization (RQ3)

The pipeline is instantiated on different base encoders to test transferability and efficiency-performance trade-offs:

Small Scale: SEA-LION-ModernBERT-300M.
Large Scale: E5-Large (600M parameters).
Ablation: The study also tests XLMR-Large, mmBERT-base, and SEA-LION-ModernBERT-base to verify if the pipeline's benefits generalize beyond specific initializations.

Key Results

The model was evaluated on SEA-BED, an out-of-domain benchmark covering 10 Southeast Asian languages, 9 task types, and 169 datasets.

State-of-the-Art Performance:
- SEA-Embedding-E5-Large-600M achieved a language-average score of 0.800 on SEA-BED, outperforming the previous best open model (multilingual-e5-large-instruct at 0.789) and significantly surpassing the much larger Qwen3-Embedding-8B (0.773).
- SEA-Embedding-ModernBERT-300M achieved a score of 0.760, outperforming other small models like harrier-oss-v1-270M (0.753) and embeddinggemma-300m (0.704).
Low-Resource Language Gains: The model showed particularly strong improvements on lower-resource languages such as Lao, Khmer, and Myanmar compared to models trained primarily on high-resource SEA languages (Indonesian, Thai, Vietnamese).
Ablation Insights:
- Data: Removing either general or instruction datasets caused significant performance drops. Crucially, removing non-SEA instruction data (e.g., Arabic, Thai MIRACL) reduced the score from 0.800 to 0.755, indicating the value of multilingual task diversity.
- Objectives: Removing the SDM loss reduced the score from 0.800 to 0.776. Replacing SDM with standard knowledge distillation (L2-KD) or CKD resulted in further degradation. The temperature design ( $\tau_s < \tau_t$ ) was also critical.
- Base Models: The pipeline consistently improved diverse base encoders (MLM and contrastive pre-trained), with gains ranging from +0.294 to +0.366 in language-average scores, suggesting the recipe is not tied to a single initialization.

Significance and Contributions

The paper claims three primary contributions:

Reproducible Framework: SEA-Embedding provides a transparent pipeline with open code, open data (publicly available only), and a fully reproducible environment, enabling controlled comparisons and failure analysis that are currently impossible with closed models.
Empirical Insights: The study isolates the specific effects of data composition, objective design, and base-encoder initialization on SEA robustness, providing a "design recipe" for future models.
Robust Performance: It establishes a new state-of-the-art for SEA text embeddings while demonstrating that high performance can be achieved through systematic engineering of open data and objectives rather than relying on undisclosed proprietary data.

The authors note that while the pipeline is region-agnostic and adaptable, the current study focuses on building a robust foundation rather than exhaustively covering every dialect or domain-specific application (e.g., financial text), leaving those for future work.