💬 NLP

Enhancing Multilingual LLM Pretraining with Model-Based Data Selection

This paper introduces a transparent and efficient model-based filtering framework for multilingual datasets that, by leveraging Transformer and FastText classifiers to select diverse, knowledge-rich samples, enables a 1B-parameter LLM to match baseline performance with only 15% of training tokens while mitigating the curse of multilinguality across 20 languages.

Original authors: Bettina Messmer, Vinko Sabolčec, Martin Jaggi

Published 2026-02-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bettina Messmer, Vinko Sabolčec, Martin Jaggi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but very hungry student (a Large Language Model, or LLM) how to speak and understand the world. You have a massive library of books, websites, and articles (the internet) to feed this student.

The problem? The library is messy. It's full of spam, repetitive ads, low-quality forum posts, and garbage. If you just throw the whole library at the student, they get confused, waste time on junk, and don't learn as well as they could.

For a long time, researchers only figured out how to clean up the "English" section of this library. But what about the rest of the world? The student was getting a terrible education in other languages because no one had built a filter for them.

This paper is about building a universal, smart filter that works for 20 different languages, not just English. Here's how they did it, using some simple analogies:

1. The Problem: The "Noisy Library"

Think of the internet as a giant, chaotic flea market.

The Good Stuff: Encyclopedias, well-written news articles, and clear instructions (Structured & Knowledge-Rich).
The Bad Stuff: Spam, broken links, and random gibberish.

Previously, researchers used Rule-Based Filters (like a bouncer with a checklist: "If the text is too short, kick it out" or "If it has no punctuation, kick it out"). This works okay, but it's dumb. It might kick out a great short poem or keep a long, boring ad.

2. The Solution: The "Smart Librarian"

Instead of a checklist, the authors created a Smart Librarian (a machine learning model) that actually reads the text to decide if it's good.

They trained this librarian using a special "training manual" made of high-quality examples. They gathered:

MMLU: Like a collection of hard trivia questions and exams.
Aya Dataset: A huge collection of helpful human conversations and instructions.
OpenAssistant: Chat logs between humans and AI.

They told the librarian: "Look at these examples. These are the 'gold standard' of good text. Now, go through the messy flea market and find anything that looks or feels like these gold examples."

3. The Two Tools: The "Speedy Scanner" vs. The "Deep Thinker"

The authors built two versions of this librarian to see which worked best:

The Speedy Scanner (FastText): This is like a quick visual scan. It looks at the words and patterns to guess if a document is good. It's super fast, cheap to run, and works on regular computers. It's like a bouncer who knows the "vibe" of a good conversation.
The Deep Thinker (Transformer/MLP): This is like a professor who actually reads the sentence structure and understands the context. It's more accurate but takes more computing power. It's like a librarian who reads the first few pages of a book to judge its quality.

4. The Magic Result: "Less is More"

Here is the most surprising part. The authors took a massive dataset (FineWeb-2) and used their Smart Librarian to throw away 85% to 90% of the data.

The Old Way: Feed the student 100% of the messy library.
The New Way: Feed the student only the top 10% of the cleanest, most interesting books.

The Result: The student learned just as well (or even better!) using only 15% of the data.

Analogy: Imagine you have 100 hours of TV to watch. The old way is to watch everything, including commercials and bad reality shows. The new way is to have a smart guide who only shows you the 15 best episodes. You learn the story faster and remember it better because you didn't get distracted by the junk.

5. Why This Matters for the World

Fixing the "Curse of Multilinguality": Usually, when you teach a model many languages at once, it gets confused and performs worse in each one (like a student trying to learn 20 languages at once and forgetting them all). This paper found that by cleaning the data first, the model actually got better at all languages simultaneously. The "curse" turned into a "blessing."
Democratizing AI: By making this filter work for languages like Arabic, Danish, Chinese, and French, they are helping AI become smarter for people who don't speak English.
Open Source: They didn't keep this secret. They released the "cleaned" datasets and the code so anyone can use it.

Summary

The authors built a smart, language-aware filter that acts like a high-quality editor. They proved that you don't need more data to build a better AI; you just need better data. By throwing away the junk and keeping only the "structured and knowledge-rich" samples, they made AI models smarter, faster to train, and more capable of understanding the world in many different languages.

In one sentence: They taught the AI to ignore the noise and focus on the signal, proving that a little bit of high-quality data is worth a whole lot of garbage.

1. Problem Statement

While dataset curation is critical for Large Language Model (LLM) performance, current state-of-the-art methods are heavily skewed toward English.

The Disparity: Existing model-based filtering techniques (e.g., FineWeb-Edu, DCLM) primarily focus on English data. Multilingual datasets (like FineWeb-2) still rely heavily on heuristic-based filters (e.g., word length, punctuation) rather than semantic quality assessment.
The Gap: This lack of research on non-English languages risks widening the performance gap between English and other languages. Furthermore, "multilingual pretraining" often suffers from the "curse of multilinguality," where models trained on mixed languages perform worse on specific languages compared to monolingual counterparts due to data dilution.
Goal: The authors aim to develop a transparent, simple, and efficient model-based filtering framework for multilingual datasets that identifies structured and knowledge-rich samples across diverse language families, scripts, and resource levels.

2. Methodology

The proposed framework involves training proxy classifiers to distinguish high-quality pretraining data from low-quality web crawl data.

A. Representative Sample Selection (Training Data for Classifiers)

To train the filters, the authors curated a Multilingual Knowledge Collection (MKC) consisting of structured, knowledge-rich datasets. They defined two configurations:

MKC: Includes high-quality, curated datasets: Include-Base-44 (exams), OpenAssistant-2 (conversations), MMLU (knowledge questions), and Aya Dataset (instruction tuning).
MKC+: Includes MKC plus the Aya Collection (a larger, AI-generated prompt dataset with ~514M samples, though with lower quality guarantees).

The training process treats the task as binary classification:

Positive Class: Samples from the MKC/MKC+ datasets.
Negative Class: Randomly subsampled documents from the raw FineWeb-2 dataset (assumed to be mostly unstructured/noisy).
Strategy: Classifiers are trained separately for each language to avoid cross-lingual bias leakage.

B. Filtering Approaches

The paper evaluates three model-based filtering methods:

FastText (FT): A linear classifier using n-gram features (2-gram for most, 4-gram for Chinese). It is computationally efficient and runs on CPUs.
Transformer Embeddings + MLP (MLP): Uses XLM-RoBERTa (base model, 279M params) to generate 768-dimensional embeddings for the first 512 tokens of a document. A single-hidden-layer Multi-Layer Perceptron (MLP) is trained on these embeddings to classify quality.
Cosine Similarity (CS): Calculates the maximum cosine similarity between a document's embedding and a set of $K$ (8192) randomly sampled positive embeddings.

C. Experimental Setup

Base Dataset: FineWeb-2 (globally deduplicated).
Model: 1B-parameter Llama models trained on 70B and 119B tokens.
Languages: Focus on Chinese, German, French, Arabic, and Danish (representing diverse families/scripts), with final release covering 20 languages.
Evaluation: Uses FineTasks (a multilingual evaluation suite) and standard benchmarks (MMLU, ARC, etc.) to measure reading comprehension, reasoning, and knowledge.

3. Key Contributions

Unified Multilingual Framework: A transparent and simple framework for model-based filtering applicable to diverse language families and scripts, moving beyond English-centric heuristics.
Efficient Data Selection: Demonstrates that using MLP MKC+ (Transformer embeddings + MLP trained on the combined knowledge collection) allows models to match baseline performance with as little as 15% of the training tokens (specifically 10% retention for high-resource languages).
Mitigation of the Curse of Multilinguality: The refined datasets enable multilingual models to outperform their monolingual counterparts when trained on equivalent token counts, effectively reversing the typical performance degradation seen in multilingual training.
Open Release: The authors released FineWeb2-HQ, a refined pretraining dataset covering 20 languages, along with the codebase and precomputed embeddings.

4. Key Results

Performance Gains: The MLP MKC+ approach consistently outperformed the state-of-the-art baseline (FineWeb-2) and other filtering methods (FastText, CS) across Chinese, German, French, Arabic, and Danish.
- On high-resource languages (Chinese, German, French), the filtered model (10% retention) achieved baseline-level performance with significantly fewer tokens.
- On benchmarks like MMLU and CMMLU, the MLP MKC+ model showed clear accuracy improvements over the baseline during training.
Threshold Analysis: Retaining the top 10% of documents was found to be the optimal threshold for high-resource languages when using the MKC+ dataset. Lower-resource languages required higher retention rates (56% for Arabic, 65% for Danish) to maintain sufficient token volume.
Training Data Diversity: Ablation studies showed that combining diverse source datasets (MKC+) yielded better results than using individual components (e.g., MMLU alone or Aya alone). Interestingly, even the lower-quality Aya Collection contributed positively when combined with high-quality data.
Decontamination: The performance gains persisted even after rigorous 13-gram decontamination, proving the results were not due to benchmark leakage.
Multilingual Training: When training a multilingual model on the filtered data, the "curse of multilinguality" disappeared; the multilingual model outperformed the monolingual model trained on the same amount of language-specific data.
English Validation: The method also improved English performance, outperforming DCLM and FineWeb-Edu baselines on average rank across SmolLM tasks.

5. Significance and Impact

Democratization of LLMs: By providing a scalable, model-based filtering framework for non-English languages, this work helps reduce the disparity in LLM capabilities between English and other languages.
Efficiency: The ability to achieve state-of-the-art performance with a fraction of the training tokens (e.g., 10-15%) offers significant cost savings for training LLMs, making high-quality pretraining more accessible.
Reproducibility and Transparency: The release of the FineWeb2-HQ dataset (20 languages), the codebase, and the precomputed embeddings allows the community to replicate and build upon these findings without incurring massive computational costs for embedding generation.
Paradigm Shift: The paper challenges the reliance on heuristic filters for multilingual data, demonstrating that embedding-based semantic filtering is superior for identifying structured, knowledge-rich content across diverse linguistic contexts.

In conclusion, the paper establishes that model-based data selection is not only feasible for multilingual datasets but is essential for overcoming the limitations of current multilingual LLMs, offering a path to more efficient, equitable, and high-performing language models.