💬 NLP

EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training

This paper demonstrates that combining continued pretraining with Estonian-enriched multilingual replay and lightweight post-training alignment substantially enhances Estonian language capabilities in multilingual LLMs while effectively preserving their English performance and general reasoning skills.

Original authors: Aleksei Dorkin, Taido Purason, Emil Kalbaliyev, Hele-Andra Kuulmets, Marii Ojastu, Mark Fišel, Tanel Alumäe, Eleri Aedmaa, Krister Kruusmaa, Kairit Sirts

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aleksei Dorkin, Taido Purason, Emil Kalbaliyev, Hele-Andra Kuulmets, Marii Ojastu, Mark Fišel, Tanel Alumäe, Eleri Aedmaa, Krister Kruusmaa, Kairit Sirts

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are the engines behind many of today's most advanced artificial intelligence tools, capable of writing stories, solving problems, and answering questions. These systems are trained on vast amounts of text from the internet, learning patterns of human language by reading billions of examples. However, the data used to teach them is heavily skewed toward English. Because English dominates the digital world, these models become experts in English while often struggling with smaller languages like Estonian, which have far less digital representation. This imbalance means that while an AI might write a perfect essay in English, it could produce confusing or incorrect text when asked to do the same in Estonian. Researchers are now working to fix this gap, asking whether it is possible to teach these powerful, English-heavy models a new language without breaking the skills they already possess.

A team of researchers from Estonia set out to answer this question by creating a specialized version of a large language model tailored for the Estonian language. They started with two existing models: one that was heavily trained on English data and another that was already somewhat familiar with many languages. The researchers did not build a new model from scratch. Instead, they took these existing systems and gave them a focused "re-education" using a specific mix of data. First, they exposed the models to a large volume of Estonian text, including literature, academic writing, and web content, to build a foundation of language knowledge. To ensure the models did not forget how to reason or understand other topics, they mixed in English text, code, and mathematical problems during this training phase. This process, known as continued pretraining, allowed the models to learn Estonian while keeping their general intelligence intact.

After this initial training, the researchers refined the models to make them better at following instructions, a crucial skill for chatbots and assistants. They taught the models how to respond to user commands in both Estonian and English, using a mix of real human conversations and synthetic examples. A key step in their process involved a technique called chat vector merging. Imagine a model as a student who has learned a new language but has forgotten how to speak politely or follow complex rules. The researchers took the "knowledge" of how to follow instructions from the original English version of the model and blended it into their new Estonian-trained version. This allowed the final product to speak fluent Estonian while retaining the ability to follow instructions clearly in both languages.

The results of this experiment were surprising and significant. Before the training began, the model that was already multilingual performed better on Estonian tasks than the one that was primarily English-focused. However, after the adaptation process, the situation flipped. The model that started with a strong English foundation showed much larger improvements in Estonian, eventually outperforming the multilingual model. This suggests that a model's starting point in a specific language does not necessarily predict how well it will learn that language later. The researchers found that the adapted models could understand grammar, translate text, and solve logic puzzles in Estonian far better than before. They also tested these models in a public arena where human users could chat with them and vote for the best response. The new Estonian models ranked highly, competing successfully against much larger and more complex systems.

The study also revealed important limits to how these models learn. The researchers tested whether repeating the Estonian training data multiple times would help the models learn better, a method some other studies have used. They found that for this size of model, going through the data a second time offered no real benefit and was simply a waste of computing power. One training cycle was sufficient. Furthermore, the researchers discovered that while the models became excellent at Estonian, there was a slight trade-off: their ability to follow instructions in English dipped slightly. However, the chat vector merging technique successfully restored most of this lost English ability, proving that it is possible to specialize a model for a smaller language without sacrificing its general capabilities.

This work provides a clear path forward for improving artificial intelligence in languages that are often overlooked. By carefully mixing new language data with existing knowledge and using smart techniques to blend different model behaviors, researchers can create powerful tools for specific communities without needing to start from zero. The findings suggest that the most effective way to adapt a large language model might not be to start with a model that already knows the language, but to take a model that is strong in general reasoning and teach it the new language with a balanced diet of data. The resulting models are now available for others to use, offering a high-quality, native-level experience for Estonian speakers while maintaining the robustness of a global intelligence.

Technical Summary: EstLLM

Problem Statement

Large Language Models (LLMs) are predominantly trained on English-centric data, leading to uneven performance across languages, particularly for smaller languages like Estonian. While multilingual models exist, their training data distributions remain highly imbalanced, often resulting in suboptimal capabilities for low-resource languages. Existing adaptation efforts often struggle to balance improving target-language competence with preserving general reasoning and instruction-following abilities acquired during large-scale pretraining. The paper investigates whether Continued Pretraining (CPT) can substantially enhance Estonian capabilities in multilingual LLMs without compromising broader English performance or general reasoning.

Methodology

The authors propose EstLLM, an adaptation pipeline designed to improve Estonian language skills in pretrained multilingual LLMs. The pipeline utilizes Llama 3.1 8B and Apertus 8B as base models and consists of three primary stages:

Continued Pretraining (CPT):
- Objective: Increase Estonian exposure while mitigating catastrophic forgetting of general capabilities.
- Data Mixture: The authors constructed a balanced mixture totaling 35.7 billion tokens. It includes the Estonian National Corpus 2023 (ENC) (8.6B tokens) as the primary Estonian source. To preserve general capabilities, this is combined with English replay (Cosmopedia v2), code (Python-Edu), mathematics (FineMath-4+), and instruction-augmented corpora.
- Strategy: A "warm-up" phase precedes the main CPT to soften distribution shifts, emphasizing high-quality curated data. The main CPT runs for a single epoch on the Estonian-enriched mixture without data repetition, based on preliminary findings that multiple epochs yielded only marginal gains.
- Technique: Sequences are packed to 4096 tokens using best-fit bin packing to preserve document integrity.
Post-Training (Instruction Tuning & Preference Optimization):
- Supervised Fine-Tuning (SFT): The CPT models undergo SFT primarily on English datasets to restore instruction-following abilities, while incorporating Estonian data (expert-authored Q&A, parallel translation, and synthetic data generated by Gemma 3 27B) to enable Estonian instruction following.
- Preference Optimization (DPO): Direct Preference Optimization is applied using the HelpSteer3 dataset to align model outputs with human preferences, relying on cross-lingual transfer rather than language-specific preference data.
Chat Vector Merging:
- To further restore instruction-following behavior, the authors apply Chat Vector merging. This involves computing the parameter difference ( $\theta_{Instruct} - \theta_{Base}$ ) between the original instruction-tuned base model (e.g., Llama 3.1 8B Instruct) and its base, then adding this vector to the post-trained model ( $\theta_{CPT}$ ). This transfers instruction-following patterns without retraining.

Key Contributions

Adaptation Pipeline: A comprehensive pipeline for adapting pretrained LLMs to Estonian via CPT, post-training, and chat vector merging.
Empirical Insights:
- Evidence that English-centric models (Llama 3.1) can outperform massively multilingual models (Apertus) after adaptation, suggesting initial target-language performance is not a reliable predictor of adaptation potential.
- Demonstration that chat vector merging effectively restores English capabilities while compounding Estonian gains.
Released Models: The release of EstLLM models (8B scale) that achieve top-tier performance in Estonian-specific evaluations.
Benchmarking: The construction of a new multi-turn prompt dataset for systematic human evaluation and the deployment of an Estonian-specific Chatbot Arena-style leaderboard.

Results

The adapted models were evaluated on Estonian benchmarks, English benchmarks, and through human evaluation.

Estonian Performance: Both adapted models showed consistent improvements over their base counterparts. Notably, the Llama-3.1-EstLLM-8B model achieved the strongest overall performance among 8B-scale models, outperforming multilingual alternatives like EuroLLM-9B and Apertus-8B on Estonian discriminative benchmarks. The Llama-based model showed a relative gain of +30.4% on Estonian discriminative tasks, compared to +7.8% for Apertus.
English Performance: While CPT initially caused slight regressions in English instruction-following and reasoning for the Llama-based model, chat vector merging substantially restored these capabilities. The merged model (Llama-EstLLM-8B-Instruct-CV) matched or exceeded the original Llama 3.1 8B Instruct on most English tasks while maintaining superior Estonian performance.
Human Evaluation: Targeted pairwise human evaluation on a multi-turn Estonian benchmark confirmed that:
- Adding a chat vector significantly improved results across all categories (fluency, grammar, instruction following).
- Starting from the English-centric Llama model yielded better results than starting from the multilingual Apertus model.
- Repeating Estonian data for a second epoch provided no measurable improvement over a single epoch.
Arena Performance: In an Estonian Chatbot Arena-style evaluation, the merged Llama-based EstLLM model ranked competitively, sharing the top rank with significantly larger models (e.g., 685B and 1T parameter models) and outperforming other open 8B models.

Significance and Claims

The paper claims that Continued Pretraining with a balanced multilingual data mixture, combined with lightweight post-training alignment (specifically chat vector merging), can substantially improve single-language capabilities in pretrained multilingual LLMs.

Key takeaways emphasized by the authors include:

Adaptation Potential: The effectiveness of adaptation depends not just on the target-language competence of the base model, but also on the original pretraining distribution. English-centric models may offer greater "room for improvement" in specific low-resource languages than models already heavily tuned for multilingualism.
Efficiency: A single epoch of CPT is sufficient for significant gains at the 8B scale, challenging the common practice of repeating target-language data for multiple epochs.
Restoration of General Capabilities: Chat vector merging offers a computationally inexpensive method to recover general-purpose capabilities (like English instruction following) that may be degraded during language-specific adaptation, effectively decoupling the trade-off between specialization and generalization.

The authors maintain a modest stance, noting limitations such as the lack of systematic optimization of data mixtures due to computational constraints, the restriction of experiments to the 8B parameter scale, and the absence of dedicated safety alignment for the Estonian models. They suggest that while the findings are promising, further investigation is needed to understand the specific contributions of different data sources and to verify if these dynamics hold for substantially larger models.