Technical Summary: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

Problem Statement

Despite Portuguese being one of the most widely spoken languages globally, it remains significantly underrepresented in text embedding evaluation. Current assessment practices rely heavily on English-centric or broad multilingual aggregates (such as MMTEB), which fail to reveal language-specific behaviors. Consequently, it is unclear which embedding models are genuinely effective for Portuguese, how rankings shift across different task families, and whether multilingual performance reliably translates to Portuguese-specific robustness. Existing Portuguese evaluations have been limited to narrow task subsets (e.g., only Semantic Textual Similarity or specific NLI tasks), lacking a unified, multi-task framework.

Methodology

Benchmark Construction (MTEB-PT)

The authors introduce MTEB-PT, a Portuguese benchmark slice derived from the MMTEB ecosystem. It comprises 14 existing datasets spanning four distinct task families:

1. Semantic Textual Similarity (STS): 3 datasets (e.g., Assin2STS, SICK-BR-STS) evaluating pairwise semantic similarity.
2. Classification: 5 datasets covering intent detection, sentiment analysis, toxicity, and hate speech.
3. Retrieval: 3 datasets (e.g., WebFAQRetrieval, WikipediaRetrievalMultilingual) involving asymmetric query-document ranking.
4. Reranking: 3 datasets (e.g., WikipediaRerankingMultilingual) focusing on candidate reordering.

The selection criteria prioritized publicly available, open-license datasets of sufficient quality that are already integrated into the MTEB/MMTEB ecosystem. The benchmark includes both symmetric tasks (STS) and asymmetric, long-context tasks (retrieval/reranking) to stress-test model capabilities.

Evaluation Protocol

The study evaluates 17 models (open-source, closed-source, and community models) under a unified protocol:

• Input Formatting: Adherence to model-specific inference templates (e.g., query: and passage: prefixes for E5-family models) where documented; raw text otherwise.
• Metrics:
  • STS: Spearman's rank correlation on cosine similarity.
  • Classification: Logistic regression on frozen embeddings.
  • Retrieval: nDCG@10.
  • Reranking: Mean Average Precision (MAP) or nDCG@10 depending on the task.
• Dimensional Truncation: An analysis of performance degradation as embedding dimensions are reduced (from full size down to 64 dimensions) to assess storage and latency efficiency.

Portuguese Fine-Tuning and MRL

To establish strong baselines, the authors fine-tuned three representative backbone models (multilingual-e5-large, bertimbau-large, and modbertbr) using:

• Data: A combination of Portuguese STS pairs, NLI entailment pairs, and retrieval-oriented positive pairs (from MLDR).
• Objective: Contrastive supervision using CoSENT for scored pairs and Multiple Negatives Ranking Loss for positive pairs.
• Technique: Matryoshka Representation Learning (MRL) was applied, enabling the models to support progressive dimensional truncation (64, 128, 256, 512, and full dimensions) without retraining.

Key Contributions

1. MTEB-PT Benchmark: The first unified Portuguese benchmark covering STS, classification, retrieval, and reranking, constructed from 14 datasets within the MMTEB framework.
2. Systematic Evaluation: A comprehensive comparison of 17 models, revealing that Portuguese performance is highly task-dependent and that multilingual rankings do not uniformly predict Portuguese-specific outcomes.
3. Strong Portuguese Baselines: Release of three MRL-trained, Portuguese-adapted models (e5-large-matryoshka, bertimbau-large-matryoshka, modbertbr-matryoshka) that serve as reference points for future research.
4. Dimensional Efficiency Analysis: An investigation into how Portuguese embeddings retain quality under compression, demonstrating that MRL-trained models remain competitive even with aggressive truncation.

Results

Task-Dependent Performance

• STS: Portuguese-specific fine-tuning yields the most significant gains. Models like e5-large-matryoshka and serafim-900m outperform general multilingual models.
• Classification: Performance gaps between models are narrower. General-purpose multilingual encoders often perform as well as or better than Portuguese-specialized models, suggesting classification is less sensitive to fine-grained sentence alignment than STS.
• Retrieval & Reranking: These asymmetric, long-context tasks show the largest divergence from STS rankings. Closed-source models and search-oriented multilingual encoders (e.g., qwen3-embedding-0.6b, text-embedding-3-large) dominate. Models with short context limits (e.g., 128 tokens) struggle significantly here.
• Multilingual vs. Portuguese Rankings: While some structure is preserved, multilingual rankings are not reliable predictors for Portuguese performance. The discrepancy is most pronounced in retrieval and reranking, where models optimized for search outperform those optimized for symmetric semantic similarity.

Impact of Fine-Tuning and MRL

• Adaptation: Fine-tuning raw Portuguese pretrained encoders (like bertimbau-large) with contrastive supervision dramatically improves their sentence embedding capabilities, particularly on STS.
• MRL Benefits: MRL-trained models maintain robustness under dimensional truncation. While e5-large-matryoshka shows the strongest gains on STS, the adapted models also improve retrieval performance relative to their non-fine-tuned backbones, though they do not fully displace top-tier search-optimized models.

Dimensional Truncation

• Symmetric Tasks (STS/Classification): Performance remains relatively stable even with reduced embedding dimensions.
• Asymmetric Tasks (Retrieval/Reranking): These tasks are more sensitive to truncation. However, MRL-trained models degrade more gracefully than non-MRL baselines, offering a viable trade-off for storage-constrained deployments.

Significance and Claims

The paper argues that multilingual average performance is insufficient for selecting sentence encoders for Portuguese. The authors emphasize that:

• Task Diversity is Critical: A model strong in STS may fail in retrieval, and vice versa. Selection must be based on the specific application domain.
• Language-Specific Adaptation Matters: While multilingual models are strong, fine-tuning on Portuguese data (especially for symmetric tasks) provides measurable improvements.
• Context Length is a Bottleneck: Long-context support is a decisive factor for retrieval and reranking tasks, where short-context models underperform regardless of their multilingual capabilities.
• Efficiency is Achievable: Through MRL, it is possible to obtain compact, high-quality Portuguese embeddings that remain competitive under dimensional constraints, facilitating deployment in low-resource environments.

The authors conclude that MTEB-PT provides a necessary, task-diverse foundation for evaluating Portuguese sentence encoders, moving beyond the limitations of aggregated multilingual metrics. They release the benchmark, fine-tuned models, and code to support reproducible research in this domain.