💬 NLP

EuroLLM-22B: Technical Report

This paper introduces EuroLLM-22B, a large language model trained from scratch to support 35 languages (including all 24 official EU languages), addressing the underrepresentation of European languages in existing models while achieving competitive performance in reasoning, instruction following, and translation, with all related datasets, models, and code released to support future research.

Original authors: Miguel Moura Ramos, Duarte M. Alves, Hippolyte Gisserot-Boukhlef, João Alves, Pedro Henrique Martins, Patrick Fernandes, José Pombal, Nuno M. Guerreiro, Ricardo Rei, Nicolas Boizard, Amin Farajian, Ma

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Miguel Moura Ramos, Duarte M. Alves, Hippolyte Gisserot-Boukhlef, João Alves, Pedro Henrique Martins, Patrick Fernandes, José Pombal, Nuno M. Guerreiro, Ricardo Rei, Nicolas Boizard, Amin Farajian, Mateusz Klimaszewski, José G. C. de Souza, Barry Haddow, François Yvon, Pierre Colombo, Alexandra Birch, André F. T. Martins

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

EuroLLM-22B: The Multilingual Librarian of Europe

Imagine the world of Artificial Intelligence as a massive library. For a long time, this library has been dominated by books written in English. If you walked in and asked for a story in French, German, or Portuguese, you might find a few books, but they were often translations of English stories or written by authors who didn't fully understand the local culture.

EuroLLM-22B is a new, massive project designed to fix this. It is a "Large Language Model" (think of it as a super-smart, digital librarian) built from scratch specifically to speak, understand, and reason in all 24 official languages of the European Union, plus 11 other major languages like Arabic, Chinese, and Japanese.

Here is how the team built this digital librarian, explained in simple terms:

1. The Blueprint (The Architecture)

Before building the house, you need a blueprint. The team didn't just copy-paste an existing design; they tweaked the blueprint to handle a massive amount of information.

The Size: This model has 22 billion "neurons" (parameters). To use an analogy, if the previous version (9B) was a smart high school student, this one is a seasoned university professor with a very large memory.
The Long Memory: One of the biggest upgrades is the "context window." Imagine trying to read a novel. Older models could only remember the last few pages before they forgot the beginning. EuroLLM-22B can hold 32,000 words in its head at once. It can read a whole short story or a long legal document and remember the details from the first page to the last without getting lost.

2. The Training Diet (The Data)

You can't make a great chef with bad ingredients. The team was very picky about the "food" they fed the model during its training.

The Filter: They didn't just dump the entire internet into the model. They used a special filter (called EuroFilter) to grade the quality of the text, giving it a score from 0 to 5. They threw away the junk (like random code or low-quality web pages) and kept only the high-quality educational and literary content.
The Phases: They trained the model in three stages, like a student progressing through school:
1. Elementary: They started with a huge amount of general data to teach the basics.
2. High School: They introduced higher-quality data to refine its reasoning.
3. Graduate School: In the final phase, they fed it the absolute best data available, including complex math, coding, and translated books, to sharpen its intelligence.
The Language Mix: Unlike other models that focus heavily on English, EuroLLM-22B was fed a balanced diet of all European languages from day one, ensuring it doesn't favor one over the others.

3. The Final Polish (Instruction Tuning)

After the model learned the facts, it needed to learn how to behave. This is called "post-training."

The Classroom: The team used a new dataset called EuroBlocks. Imagine this as a massive collection of practice questions and answers covering everything from "Write a poem about rain" to "Solve this math problem" and "Translate this sentence."
The Teacher: They used other advanced AI models to generate high-quality answers for these questions, then selected the best ones to teach EuroLLM-22B how to follow instructions precisely. They removed any "reasoning traces" (the internal monologue) to make the final output clean and direct.

4. The Report Card (The Results)

The team tested EuroLLM-22B against other famous AI models to see how it performed.

The Competition: They compared it against other "fully open" models (models where the code and weights are free for anyone to use) and some "open-weight" models (where you can use the model but can't see how it was built).
The Outcome:
- European Champion: Among the fully open models made in Europe, EuroLLM-22B is the strongest. It outperformed other European models, even those that were much larger (like a 70-billion-parameter model).
- Translation: It is excellent at translating between languages, often matching the performance of models that are twice its size.
- Reasoning: It is very good at logic puzzles, math, and following complex instructions.
- Efficiency: The paper notes that EuroLLM-22B achieved these results with a "modest" amount of training data (4 trillion words) compared to some competitors that used 15 trillion, suggesting that high-quality data matters more than just a massive quantity.

5. The Gift to the World

The most important part of this paper is that the team didn't keep the library to themselves. They are releasing everything to the public:

The Models: Both the "base" version (the raw brain) and the "instruct" version (the helpful assistant).
The Data: The actual datasets they used to train the model (EuroWeb and EuroBlocks), so other researchers can learn from them.
The Code: The software tools they used to build and test the model.

In summary: EuroLLM-22B is a powerful, open-source AI tool designed to ensure that European languages are not left behind in the AI revolution. It proves that with careful data selection and a focus on quality, you can build a world-class AI that speaks your language fluently, without needing to be a closed, secret project.

Technical Summary: EuroLLM-22B

Problem Statement

Despite rapid advancements in large language models (LLMs), the current ecosystem remains heavily skewed toward English and a limited set of high-resource languages. Many leading models are either closed-source or only partially open, offering limited transparency regarding training data and procedures. Consequently, numerous European languages remain underserved, with few models "made in Europe" capable of natively supporting the full spectrum of official European Union (EU) languages. The EuroLLM project was launched to address this gap by developing open models that prioritize European linguistic diversity and foster AI development within the EU.

Methodology

Architecture and Modeling

EuroLLM-22B is a 22.6 billion parameter model built upon the architectural decisions of its predecessors (EuroLLM 1.7B and 9B). Key specifications include:

Tokenization: A BPE-based tokenizer with a vocabulary of 128,000 units, covering 24 official EU languages and 11 additional languages (Arabic, Catalan, Chinese, Galician, Hindi, Japanese, Korean, Norwegian, Russian, Turkish, and Ukrainian).
Architecture: Utilizes Grouped Query Attention (GQA), pre-layer normalization, RMSNorm, SwiGLU activation functions, and Rotary Positional Embeddings (RoPE).
Context Window: A significant extension to 32,768 tokens, achieved by adjusting the RoPE scaling factor ( $\theta$ ) from $1 \times 10^4$ to $1 \times 10^6$ .

Pre-Training Strategy

The model was pre-trained on approximately 4 trillion tokens using a three-phase training schedule designed to progressively expose the model to higher-quality data:

Phase 1: Trained on 3.6T tokens with a linear warmup to a peak learning rate of $1.5 \times 10^{-4}$ . This phase includes high-quality educational documents from FineWeb-edu immediately, rather than reserving them for later stages.
Phase 2: An annealing phase over 400B tokens, reducing the learning rate to 10% of its peak.
Phase 3: A final decay phase where the learning rate drops to zero. This phase incorporates long-context data (books and code) to support the 32K token window.

Data Curation:

Web Data: English data is sourced from FineWeb-edu and Nemotron-CC. Multilingual data is aggregated from RedPajama-Data-v2 (for high-resource languages) and HPLT, MADLAD-400, CulturaX, and mC4 (for others). All data undergoes rigorous filtering, including perplexity checks, heuristic filters (e.g., removing "lorem ipsum," excessive uppercase), and quality scoring via EuroFilter, a classifier assigning scores from 0 to 5.
Parallel Data: Sentence-level and document-level parallel data (xx $\leftrightarrow$ en) are collected from sources like Europarl, ParaCrawl, and WikiMatrix, filtered using Bicleaner and CometKiwi-22.
Code and Math: Data is sourced from The Stack, Algebraic-stack, and Open-web-math. The third phase introduces FineMath and synthetic math data generated by Qwen-2.5 models, filtered by LLM-as-a-Judge.
High-Quality Sources: Wikipedia, ArXiv, Books, and Apollo are used for English, with Cosmopedia (and its translated variants) added for later phases.

Post-Training

The instruction-tuned model (EuroLLM-22B-Instruct) is derived from the base model using EuroBlocks-22B, a new multilingual instruction dataset containing approximately 10.6 million examples.

Dataset Construction: Built upon previous EuroBlocks versions, incorporating instructions from diverse public sources (e.g., Hermes-3, Tülu 3, Nemotron V2). Responses are regenerated using capable open models (DeepSeek, Qwen, Llama) and selected via Skywork-Gemma2-27B as a reward model.
Filtering: Structured reasoning traces are removed to create a non-reasoning instruction-response corpus.
Fine-Tuning: The model is fine-tuned for 5 epochs using bfloat16 mixed precision, sequence packing, and a cosine learning rate scheduler ( $1 \times 10^{-5}$ max LR). Training efficiency is enhanced using Axolotl and Liger-Kernel.

Key Contributions

EuroLLM-22B Model: The release of a 22B parameter model trained from scratch to support all 24 official EU languages plus 11 additional languages, with a 32K token context window.
Improved EuroLLM-9B: An updated 9B model incorporating the same long-context extension and post-training recipe as the 22B version.
EuroWeb Dataset: The release of the multilingual web pretraining dataset used for EuroLLM 22B, including the quality-filtered data tiers.
EuroBlocks-22B: A new, expanded multilingual instruction dataset with high-quality synthetic responses.
Open-Source Ecosystem: Release of base and instruction-tuned models, along with the pre-training (Megatron-LM fork) and evaluation codebases.

Results

Evaluated across a broad set of English and multilingual benchmarks, EuroLLM-22B demonstrates competitive performance:

Scaling Trends: The model shows consistent improvements over EuroLLM-9B and EuroLLM-22B-Preview, confirming a positive scaling trend within the EuroLLM family.
Comparison with European Models: EuroLLM-22B achieves the strongest performance among fully open European models. Notably, it frequently outperforms or remains competitive with Apertus-70B (a 70B parameter model) despite having roughly one-third of the parameters, particularly in instruction following and STEM tasks.
Comparison with Global Models: While EuroLLM-22B trails the very best open-weights models (e.g., Llama-3.3-70B, Qwen-3-30B-A3B) in aggregate scores, it offers the most capable fully open alternative specifically designed for European languages.
Translation: Translation performance remains robust, with the updated post-training recipe strengthening general assistant behavior and reasoning without degrading translation quality.
Long Context: The extension to 32K tokens enables effective modeling of long-form inputs and coverage of benchmarks like WMT25, which were previously inaccessible to the 4K context models.

Significance and Claims

The paper positions EuroLLM-22B as a significant step toward democratizing AI in Europe. By providing a fully open, high-quality model that natively supports the full linguistic diversity of the EU, the authors aim to reduce reliance on closed, English-centric models. The authors claim that EuroLLM achieves strong quality with a comparatively modest token budget (4T tokens) compared to larger models trained on significantly more data (e.g., 15T tokens for Apertus-70B), suggesting that high-quality data curation and targeted post-training are effective strategies for closing performance gaps. The release of the associated datasets and codebases is intended to support future research in European language understanding and generation.