💬 NLP

MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese

This paper introduces MTEB-BR, the first dedicated benchmark comprising 22 native Brazilian-Portuguese tasks to evaluate text embedding models without relying on translations, revealing that top-tier performance is achievable with open-source models and demonstrating that global multilingual rankings only moderately predict Portuguese-specific effectiveness.

Original authors: Tardelli Ronan Coelho Stekel

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tardelli Ronan Coelho Stekel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Lost in Translation" Trap

Imagine you are trying to buy a high-quality car specifically for driving on Brazilian roads. You go to a global car review website, but they only have reviews for cars tested on American or European highways. They tell you, "This car is great!" because it performed well on those foreign tracks.

But here's the catch: Brazilian roads have different potholes, different traffic patterns, and different weather. A car that wins on a European track might struggle on a Brazilian one.

For a long time, this was the situation for Brazilian Portuguese in the world of AI. If you wanted to test how well an AI understands Portuguese sentences, you had to use:

Translated tests: Taking English tests and translating them into Portuguese (which often loses the "flavor" and nuance of the language).
Thin coverage: Very few tests that didn't cover the full range of how people actually speak and write in Brazil.

The Solution: MTEB-BR (The "Brazilian Driving Test")

The authors created MTEB-BR, a brand-new, native benchmark. Think of this as building a dedicated driving test track right in São Paulo, using only Brazilian traffic rules and road conditions.

No Translations Allowed: They strictly banned any test data that was translated from English. Every single question, legal document, medical note, or social media post used in the test was created originally in Brazilian Portuguese.
The Course: They built a 22-stop obstacle course covering seven different types of driving skills:
- Classification: Sorting mail into the right bins (e.g., is this tweet hateful?).
- Retrieval: Finding the right needle in a haystack (e.g., finding a specific law in a massive database).
- Clustering: Grouping similar items together (e.g., organizing medical records by topic).
- And more, including legal, medical, and tax domains.

The Race: Who Won?

They put 93 different AI models through this course. Some were free, open-source models (like a DIY car you can build yourself), and some were expensive, closed commercial APIs (like a luxury car you have to rent by the minute).

The Results:

The "Frontier" Tie: The top 6 models were so close in performance that the test couldn't statistically tell who was truly the "best." They are all in the same elite tier. It's like a race where the top six cars crossed the finish line within a millimeter of each other.
The Surprise Winner: One of the top performers was an open-source model (Qwen3-Embedding-8B). This is huge because it means you don't need to pay a commercial company to get top-tier performance; you can run this model yourself for free.
The "Global" Trap: The paper checked if the global "World Champion" AI (the one that wins on English and multilingual tests) would also win here. The answer was no.
- Analogy: Imagine a Formula 1 champion driver. If you put them in a rally car on a muddy Brazilian dirt track, they might not win.
- One model ranked 3rd globally but dropped to 49th in Brazil. This proves that being good at English doesn't automatically mean you are good at Portuguese.

The "Statistical Layer" (The Referee's Notebook)

Most benchmarks just give you a single score and a rank (1st, 2nd, 3rd). The authors felt this was too simple, like saying "Car A is faster than Car B" without mentioning the margin.

They added a statistical layer to act like a careful referee:

Confidence Intervals: Instead of just saying "Model A scored 0.68," they say "Model A scored 0.68, give or take 0.05." This tells you if the difference between two models is real or just a fluke.
Item Response Theory (IRT): This is a fancy way of asking, "Which parts of the test actually separate the good drivers from the bad ones?"
- They found that Retrieval tasks (finding information) were the best at separating the models.
- Clustering tasks (grouping things) were the worst at telling models apart.
- Analogy: It's like realizing that a math test is great at spotting geniuses, but a spelling bee isn't very good at distinguishing between a high schooler and a college student.

The Bottom Line for Users

If you are a developer or business owner in Brazil:

Don't trust the global leaderboards blindly. A model that is #1 in the world might be mediocre in Brazil.
You don't need to pay. You can get top-tier performance with free, self-hosted models.
Look at the "Frontier." Since the top 6 models are statistically tied, your choice shouldn't be based on a tiny score difference. Instead, choose based on cost (free vs. paid), speed, and license (can you use it commercially?).

In short, MTEB-BR is the first time Brazilian Portuguese speakers have a fair, native, and statistically rigorous way to judge AI models, proving that local nuance matters and that you don't need a commercial API to get the best results.

Problem Statement
Despite Brazilian Portuguese being spoken natively by over 200 million people, there is no comprehensive, native benchmark for evaluating text embedding models. Current evaluation frameworks, such as MTEB and MMTEB, rely heavily on translated corpora (e.g., mMARCO-PT, a machine translation of English MS MARCO) or offer thin multilingual coverage. The authors argue that translation introduces systematic artifacts—including translation noise, domain drift, and idiomatic flattening—that mask real performance differences between models on native text. While language-specific extensions exist for Chinese, French, German, and others, Portuguese lacked a dedicated, native benchmark that excludes machine-translated data.

Methodology
The paper introduces MTEB-BR, a benchmark designed with a single inclusion criterion: native source.

Task Suite: The benchmark consists of 22 tasks across seven MTEB categories: classification, multilabel classification, pair classification, semantic textual similarity (STS), clustering, retrieval, and reranking. These tasks are drawn from 17 distinct corpora spanning legal, medical, tax, scientific, encyclopedic, and social-media domains. Crucially, machine-translated datasets (specifically mMARCO-PT and mkqa-PT) are excluded by construction.
Model Panel: The authors evaluated 93 models, ranging from 23M to 27B parameters. This includes 73 open-weight models (decoder-LLM embedders, multilingual encoders, Portuguese-specific encoders, and Sentence-Transformer baselines) and 20 closed commercial APIs (from Google, OpenAI, Voyage, Mistral, and Amazon).
Statistical Rigor: Beyond standard point estimates, the authors introduce a "statistical layer" to complement the leaderboard. This includes:
- Per-task bootstrap confidence intervals (95% CI from 10,000 resamples).
- Paired-bootstrap significance testing to determine if one model reliably beats another.
- Item Response Theory (IRT) analysis to measure task discrimination (how sharply a task separates models) and model ability.
- Cross-leaderboard correlation analysis against the global HuggingFace MTEB multilingual leaderboard.
Evaluation Protocol: Metrics follow standard MTEB conventions (accuracy, V-measure, nDCG@10, MAP, Spearman correlation). All models were evaluated with a uniform 512-token context cap to ensure comparability.

Key Results

Model Separation: The benchmark cleanly separates approximately 78.7% of all model pairs into about a dozen distinguishable tiers. However, the top six models form a "converged frontier tier" where their performance differences are statistically indistinguishable (within a 0.020 mean score spread).
Open vs. Closed: A self-hostable, open-weight model (Qwen3-Embedding-8B, Apache-2.0) reaches the leading frontier tier, ranking second overall with a 22-task mean of 0.6704. This demonstrates that high-quality Portuguese embeddings do not strictly require commercial APIs.
Task Discrimination: Retrieval tasks are the most effective at separating models, showing the highest discrimination scores (IRT $a_t$ values between 1.49 and 2.05). Conversely, clustering tasks and certain classification probes (e.g., fact-checking) show the lowest discrimination.
Multilingual Correlation: A model's rank on the global multilingual leaderboard predicts its Portuguese rank only moderately (Spearman $\rho = 0.75$ over 55 shared models). Significant divergences exist; for instance, one model ranked 3rd globally fell to 49th on MTEB-BR. This suggests native benchmarks measure capabilities that multilingual boards miss, particularly regarding native retrieval.
Portuguese-Specific Encoders: Portuguese-specific encoders generally trailed general multilingual encoders on retrieval tasks. However, the authors attribute this to training objectives (e.g., masked language modeling vs. retrieval fine-tuning) rather than language specialization. Retrieval-tuned Portuguese models (Serafim variants) performed comparably to multilingual models.

Significance and Claims
The paper claims MTEB-BR provides practitioners with a tool to choose embedding models based on native evidence rather than translated proxies. Its primary contributions are:

A Native Task Suite: The first consolidated benchmark for Brazilian Portuguese that strictly excludes machine-translated data, offering a more accurate reflection of performance on native text.
Statistical Transparency: The introduction of a statistical layer (confidence intervals, paired significance, IRT) allows the community to distinguish between genuine performance gaps and statistical noise, particularly at the top of the leaderboard where models converge.
Cost-Quality Insights: The benchmark identifies a "Pareto frontier" where open-weight models (like Qwen3-Embedding-8B) and specific commercial APIs offer the best quality-to-cost trade-off, challenging the assumption that only large, proprietary APIs can achieve state-of-the-art results.
Limitations of Global Boards: The moderate correlation with global leaderboards serves as a caution that multilingual benchmarks are insufficient proxies for language-specific deployment, especially for retrieval tasks.

The authors release all code (Apache-2.0), the full results dataset (CC-BY 4.0), and an interactive leaderboard, aiming to establish a reusable statistical template for future language-specific MTEB extensions.

The Problem: The "Lost in Translation" Trap

The Solution: MTEB-BR (The "Brazilian Driving Test")

The Race: Who Won?

The "Statistical Layer" (The Referee's Notebook)

The Bottom Line for Users

More like this