Technical Summary: EuroBERT: Scaling Multilingual Encoders for European Languages

Problem Statement

General-purpose multilingual vector representations are foundational for Natural Language Processing (NLP) tasks such as information retrieval, classification, and regression. Traditionally, these representations are derived from bidirectional encoder models (e.g., BERT, XLM-RoBERTa). However, recent research progress has heavily favored unidirectional, decoder-only generative models. While decoder models have driven significant advances in architecture, data quality, and scale, these innovations have not been systematically transferred back to bidirectional encoders. Consequently, practitioners often rely on outdated encoder models that lack the performance benefits of modern training recipes, despite there being no inherent architectural barrier to applying these advances to encoders.

Methodology

The authors introduce EuroBERT, a family of multilingual encoders designed to revisit encoder development through the lens of recent decoder advancements. The methodology encompasses three core components:

1. Architecture

EuroBERT models (available in 210M, 610M, and 2.1B parameter sizes) adopt a standard dense transformer architecture but incorporate specific modernizations inspired by Llama 2 and Llama 3:

• Removal of biases: All biases are removed from the architecture.
• Attention Mechanisms: Implementation of Grouped Query Attention (GQA).
• Activation Functions: Use of Swish Gated Linear Units (SwiGLU).
• Normalization: Root Mean Square (RMS) Layer Normalization.
• Positional Embeddings: Rotary Position Embeddings (RoPE).
• Context Length: Native support for sequences up to 8,192 tokens.

2. Dataset Construction

The models are trained on a 5 trillion token multilingual corpus, comprising:

• Languages: 15 languages including major European languages (English, French, German, Spanish, Italian, Portuguese, Polish, Dutch, Russian, Turkish) and widely spoken global languages (Chinese, Japanese, Vietnamese, Arabic, Hindi).
• Domains: The corpus includes general text (FineWeb, CulturaX), parallel translation data (EuroLLM), and specialized domains: Code (38 programming languages from The Stack v2) and Mathematics (Proof-Pile-2, Open-Web-Math).
• Training Strategy: A two-phase pipeline consisting of Pre-training (4.8T tokens) and Annealing (200B tokens). The annealing phase adjusts data distribution to emphasize higher-quality datasets and specific domain balances.

3. Training Recipe

• Objective: Masked Language Modeling (MLM).
• Masking Ratios: A dynamic approach is used. Pre-training employs a 50% masking ratio (based on findings that higher ratios benefit larger models), while the annealing phase reduces this to 10% to optimize downstream performance.
• Curriculum Learning: During annealing, data distribution is tuned. The authors found that reducing the proportion of English (to 26%) and proportionally increasing other languages, while mixing quality buckets (specifically levels 3 and 4 rather than just the highest), yields better results.
• Variable Length Training: To handle fixed-length pre-processed documents, the authors employ random cropping of sequences up to 8,192 tokens during annealing, which proved superior to training on fixed lengths alone.
• Infrastructure: Training was conducted on the Adastra supercomputer using MI250X and MI300A GPUs, utilizing FlashAttention, fused cross-entropy, and hybrid sharding.

Key Contributions

1. EuroBERT Family: Release of three multilingual encoder models (210M, 610M, 2.1B) that outperform existing alternatives across diverse tasks.
2. Revised Training Recipe: Demonstration that encoder performance can be significantly improved by adopting decoder-era insights, specifically regarding high masking ratios in pre-training, lower ratios in annealing, and the inclusion of code and mathematics data.
3. Comprehensive Evaluation: Extensive benchmarking across multilingual retrieval, classification, regression, code, and mathematics tasks, showing that EuroBERT models match or exceed larger baselines (e.g., XLM-RoBERTa-3.5B) while being significantly smaller.
4. Open Release: Public release of the models, intermediate checkpoints, and the training framework to facilitate future research.

Results

The EuroBERT family demonstrates state-of-the-art or competitive performance across a wide range of benchmarks:

• Multilingual Retrieval: EuroBERT models achieve top performance on MIRACL, MLDR, Wikipedia, and CC-News. Notably, the 210M and 610M models outperform all similarly sized baselines and compete with the 3.5B XLM-RoBERTa.
• Classification & Regression: The models perform on par with or better than similarly sized alternatives on XNLI, PAWS-X, AmazonReviews, and MassiveIntent. They also show strong results in sequence regression (WMT, SeaHorse).
• Code and Mathematics: EuroBERT models significantly outperform all baselines (including English-only ModernBERT) on code retrieval (CodeSearchNet, DupStackMath) and mathematical tasks (MathFormula, MathShepherd). The inclusion of code and math data during training was found to improve retrieval capabilities.
• Long Context: Unlike XLM-RoBERTa, which degrades on longer inputs, EuroBERT maintains performance at context lengths up to 8,192 tokens.
• Token Classification (NER): While EuroBERT lags slightly behind XLM-RoBERTa on NER, the authors attribute this to tokenizer fertility (vocabulary size) rather than model capability, noting that performance equalizes when entities are split into similar token counts.

Significance and Claims

The paper claims that the EuroBERT family represents a "refreshed recipe" for training general-purpose multilingual encoders. The authors emphasize that:

• Scale and Architecture Matter: Increasing model capacity and adopting modern architectural components (GQA, RoPE, SwiGLU) effectively mitigates the "curse of multilinguality."
• Data Composition is Critical: The inclusion of code and mathematics data improves multilingual retrieval, while a balanced language distribution (reducing English dominance) enhances overall performance.
• Training Dynamics: The specific combination of high masking ratios during pre-training and lower ratios during annealing, along with variable-length training via random cropping, is crucial for optimizing both retrieval and classification tasks.
• Practical Utility: The models provide a viable, high-performance alternative for real-world encoder applications, offering native long-context support and superior performance in specialized domains like code and math, all while being more parameter-efficient than previous state-of-the-art encoders.

The authors conclude that their work bridges the gap between recent decoder advancements and encoder utility, providing a robust foundation for multilingual vector representations in retrieval, classification, and regression tasks.