💬 NLP

PL-MTEB: Polish Massive Text Embedding Benchmark

The paper introduces PL-MTEB, a comprehensive benchmark consisting of 30 diverse NLP tasks designed to evaluate the performance of Polish and multilingual text embedding models.

Original authors: Rafał Poświata, Sławomir Dadas, Michał Perełkiewicz

Published 2026-04-27

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Rafał Poświata, Sławomir Dadas, Michał Perełkiewicz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to hire a world-class translator, but instead of translating spoken words, you need someone who can understand the "soul" of a sentence—the deep, hidden meaning behind it.

In the world of Artificial Intelligence, we call these "meaning-readers" Text Embeddings. They turn sentences into mathematical maps so computers can understand that "The weather is lovely" and "It is a beautiful day" are essentially saying the same thing.

However, most of these "meaning-readers" were trained primarily in English. If you ask them to work in Polish, they might struggle with the nuances, the grammar, or the cultural context. It’s like hiring a brilliant English professor and expecting them to write poetry in Polish without any training.

This paper introduces PL-MTEB, which is essentially the "Polish Olympics for AI Meaning-Readers."

The "Polish Olympics" (What they did)

The researchers created a massive, rigorous testing ground to see which AI models are actually good at understanding Polish. They didn't just give them one test; they gave them a gauntlet of 30 different challenges across five "sporting events":

Classification (The Sorting Hat): Can the AI look at a sentence and correctly label it? (e.g., "Is this a happy review or an angry one?")
Clustering (The Organizer): If you give the AI a huge pile of random Polish documents, can it group them into logical piles, like "Science," "News," or "Sports"?
Pair Classification (The Matchmaker): If you show the AI two sentences, can it tell if they are related or if they are complete opposites?
Retrieval (The Librarian): If you ask a question, can the AI dive into a massive library and find the exact right book to answer you?
Semantic Similarity (The Mirror): How closely does the "meaning map" of one sentence reflect the "meaning map" of another?

The Competitors (Who they tested)

They brought in 30 different "athletes"—some were specialized Polish models (the local experts) and some were massive multilingual models (the global superstars). They tested them in different "weight classes," from tiny, lightweight models that run fast on a phone, to "Extra Large" models that are massive, heavy-duty powerhouses.

The Results (Who won?)

The results were a bit like a real sports tournament: there is no single champion for everything.

The Heavyweight Champion: A massive model called Qwen3-Embedding-8B took the overall gold medal. It was incredibly smart and dominant in the "Sorting" (Classification) and "Organizing" (Clustering) events.
The Specialist: While the heavyweights were great, the stella-pl model was the king of the "Librarian" (Retrieval) and "Mirror" (Similarity) events. It was like a specialist who might not be the strongest overall, but is the absolute best at finding specific information.
The Underdog: They found that some "Small" models (the lightweight athletes) actually performed better than much larger, heavier models. This is great news because it means we can get high-quality results without needing a supercomputer.

Why does this matter to you?

Every time you use a search engine, a chatbot, or a smart assistant, there is an "embedding model" working behind the scenes to understand what you mean.

By creating PL-MTEB, these researchers have provided a "rulebook" and a "scoreboard." Now, developers can stop guessing and start knowing exactly which AI model will provide the best, most accurate experience for Polish speakers. It ensures that the technology speaks our language—not just the words, but the meaning.

Technical Summary: PL-MTEB: Polish Massive Text Embedding Benchmark

1. Problem Statement

Text embeddings are critical components in modern Natural Language Processing (NLP) tasks, such as semantic search, document clustering, and question answering. While the Massive Text Embedding Benchmark (MTEB) provides a standardized framework for evaluating these models, it is predominantly English-centric. Existing multilingual evaluations often fail to provide a granular or comprehensive assessment for specific languages like Polish. Consequently, researchers lack a reliable, multi-task benchmark to determine how different embedding models (monolingual vs. multilingual) perform across diverse linguistic nuances and task types in the Polish language.

2. Methodology

The authors developed PL-MTEB, a comprehensive benchmark specifically tailored for Polish. The methodology is structured around four pillars:

Task Taxonomy: The benchmark comprises 30 diverse NLP tasks categorized into five functional groups:
1. Classification: Predicting labels using logistic regression.
2. Clustering: Grouping texts using mini-batch k-means (measured via v-measure).
3. Pair Classification: Determining relationships between text pairs (measured via average precision).
4. Information Retrieval: Finding relevant documents for a query (measured via nDCG@10).
5. Semantic Textual Similarity (STS): Measuring correlation between text pairs (measured via Spearman correlation).
Dataset Curation: The authors integrated three types of data:
- Existing MTEB tasks: Polish subtasks from multilingual datasets.
- Extended MTEB tasks: 12 new tasks adapted from existing Polish datasets (e.g., KLEJ).
- New Datasets: Two entirely new corpora—PLSC (Polish Library of Science Corpus) and Wikinews-PL—were created to address the lack of clustering tasks in previous benchmarks.
Data Quality Control: To ensure rigor, the authors implemented a strict pipeline involving the removal of empty/short strings, deduplication of near-duplicates, and verification to prevent test-train leakage.
Model Evaluation: The benchmark evaluated 30 models, ranging from small (108M parameters) to extra-large (9.2B parameters), including specialized Polish models (e.g., Stella-PL, MMLW) and massive multilingual models (e.g., Qwen3, BGE-Gemma2).

3. Key Contributions

The PL-MTEB Benchmark: The first comprehensive, multi-task evaluation framework for Polish text embeddings.
New Polish Datasets: The introduction of PLSC and Wikinews-PL, providing high-quality data for hierarchical and category-based clustering.
Standardization: Integration with the global MTEB framework, allowing for standardized comparisons between Polish-specific models and international multilingual models.
Open Science: Public release of all datasets, source code, and experimental results via GitHub and Hugging Face.

4. Key Results

Overall Performance: The Qwen3-Embedding-8B model achieved the highest overall average score across all 30 tasks.
Task-Specific Winners: No single model dominated all categories, highlighting the trade-offs in model training:
- Classification & Clustering: Dominated by large-scale models like the Qwen3 family.
- Retrieval: The Polish-specific stella-pl-retrieval-8k model performed best, demonstrating the advantage of language-specific fine-tuning.
- Pair Classification: The BGE-Multilingual-Gemma2 model achieved the highest average.
- STS: The stella-pl model achieved the highest scores.
Model Size Trends: Generally, models with $>1\text{B}$ parameters outperformed smaller models. However, the mmlw-roberta-base (a "small" model) significantly outperformed many "base" and "large" models, suggesting that high-quality, language-specific training can compensate for smaller parameter counts.
Zero-Shot Insights: Many top-performing models showed high "zero-shot" scores, indicating that while they were not explicitly trained on these Polish tasks, their massive pre-training allowed for strong cross-lingual transfer.

5. Significance

PL-MTEB fills a critical gap in the Polish NLP ecosystem by providing a standardized "yardstick" for embedding quality. It allows developers to make informed decisions—for instance, choosing a lightweight model like mmlw-roberta-base for efficiency or a massive model like Qwen3 for maximum accuracy. Furthermore, by providing new datasets and a rigorous evaluation framework, it serves as a foundation for future research into multilingual model transferability and the development of even more specialized Polish language technologies.