💬 NLP

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

This paper demonstrates that for German language modeling, repeatedly training on a small, high-quality subset of filtered data significantly outperforms single-pass training on larger, diverse corpora, enabling state-of-the-art performance with drastically fewer tokens.

Original authors: Ansar Aynetdinov, Patrick Haller, Alan Akbik

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ansar Aynetdinov, Patrick Haller, Alan Akbik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to speak German. You have two main strategies to choose from, and this paper is a big experiment to see which one works better.

The Dilemma: The Buffet vs. The Masterclass

Strategy A (The Buffet): You gather a massive, chaotic buffet of German text from the internet. It has everything: high-quality news, helpful tutorials, but also spam, broken sentences, and repetitive ads. You let the robot eat a little bit of everything once. The idea is: "More variety is better."
Strategy B (The Masterclass): You act like a strict editor. You throw away the spam, the broken sentences, and the fluff. You keep only the "gold"—the clear, fact-rich, educational documents. But because you threw away so much, you don't have enough food for a full meal. So, you serve this small, high-quality plate to the robot, and then you serve it again, and again, and again. You let the robot eat the same perfect meal multiple times.

The Experiment

The researchers at Humboldt-Universität zu Berlin wanted to know: Is it better to feed the robot a huge buffet once, or a small, perfect meal many times?

They built a "quality filter" (like a very smart sieve) to separate the "gold" German text from the "junk." They created a tiny, super-dense pile of the best documents (called the Dense Core).

The Results: Quality Over Quantity

The paper claims that Strategy B (The Masterclass) wins every time.

Here is the analogy:
Imagine trying to learn a complex dance.

The Buffet approach is like watching a thousand different videos of people dancing, but half of them are blurry, the music is cut off, and some people are just walking around. You see a lot of moves, but you don't really learn the steps well because the signal is weak.
The Masterclass approach is watching one perfect, crystal-clear video of a master dancer. You watch it once, then you watch it again, then again. You notice the tiny details you missed the first time. By the third or fourth time, you know the dance better than someone who watched the thousand blurry videos.

Key Findings in Simple Terms:

Repetition is Magic: Even after watching the "perfect video" (the high-quality data) 7 times, the robot kept getting smarter. It didn't get bored or confused. In fact, it learned more from the repetition than it did from seeing a huge variety of messy data just once.
Less is More: The robots trained on the small, high-quality pile (repeated many times) became better at understanding and speaking German than robots trained on 10 to 360 times more data that was less filtered.
Better at Following Orders: When they asked these robots to act as helpful assistants (like answering questions or writing emails), the ones trained on the "Masterclass" data were much more accurate and helpful. The ones trained on the "Buffet" data were more likely to make mistakes or give weird answers.
The "Translation" Problem: The researchers also noticed that many existing German tests for AI were broken because they were just machine-translated from English without fixing the grammar. They fixed these tests (like cleaning up a broken map) to ensure they were actually testing the robots fairly.

The Bottom Line

For languages like German (which have lots of data, but not trillions of words like English), the paper argues that don't just grab everything. Be picky. Filter out the noise, keep the best stuff, and let the AI study that best stuff over and over again. It's not about how much you feed the robot; it's about how good the food is and how many times it gets to digest it.

They released their "robots" (called BOLDT) and their cleaned-up tests for everyone to use, proving that you can build a very smart, small German AI without needing a massive budget or a mountain of messy data.

1. Problem Statement

The paper addresses a strategic dilemma in training Large Language Models (LLMs) for high-resource non-English languages (specifically German). While English LLMs benefit from massive, diverse web corpora where "more data" is generally better, non-English languages have significantly smaller token pools (hundreds of billions vs. trillions).

The Dilemma: Practitioners must choose between:
1. Diversity: Training on a large, lightly filtered dataset in a single pass (maximizing unique token coverage).
2. Quality/Repetition: Strictly filtering for a small, high-quality core dataset and training on it for multiple epochs (maximizing semantic density).
The Gap: Existing literature suggests that repeated exposure to data yields diminishing returns after ~4 epochs. However, it is unclear if this holds true when the alternative is a massive, noisy dataset where the signal-to-noise ratio is low. The authors investigate whether "repetition over diversity" is the optimal strategy for German.

2. Methodology

The authors constructed a hierarchical data filtering pipeline applied to 500 million German web documents from the FineWeb-2 (FW2-DE) dataset.

A. Hierarchical Quality Filters

They implemented three tiers of classifiers to score documents, creating increasingly strict subsets:

Coherence: Removes structural noise (e.g., "word salad," truncated HTML, fragments) to ensure syntactic flow.
Information Value: Selects for "signal density," retaining fact-bearing, content-rich documents (e.g., news, technical reports) while removing SEO spam and boilerplate.
Educational Quality: The most restrictive tier, prioritizing textbook-like clarity, pedagogical depth, and structured knowledge.
Dense Core: The intersection of all three filters, representing the highest semantic density subset (approx. 28 billion tokens).

B. Benchmark Cleaning

Recognizing that existing German benchmarks often suffer from poor machine translation (specifically word-order issues in German verb-final structures affecting completion tasks like LAMBADA and HellaSwag), the authors:

Re-translated major benchmarks (ARC, HellaSwag, LAMBADA, OpenBookQA, Global MMLU) using a state-of-the-art translation model (Tower+ 72B).
Manually corrected structural artifacts to ensure task integrity.

C. Experimental Design

Models: Trained decoder-only transformers from scratch at 350M and 1B parameter scales.
Budgets: Compared strategies under fixed token budgets (100B and 200B tokens).
Strategies Tested:
- Baseline: Single-pass training on diverse (Random/Coherence) data.
- Repetition: Multi-epoch training on filtered subsets (Dense Core, Information Value, etc.).
- Curricula: Hybrid approaches (e.g., "Phased" training starting with diverse data then moving to high-quality data).
Evaluation: Zero-shot performance on the cleaned benchmarks and instruction-tuning capabilities (using an LLM-as-a-judge protocol).

3. Key Contributions

Empirical Validation of Repetition: Demonstrated that for German, training on a small, high-quality subset for multiple epochs significantly outperforms training on a larger, diverse dataset in a single pass, even when the total token budget is identical.
Repetition Ceiling Analysis: Showed that performance gains from repeating high-quality data persist well beyond the 4-epoch threshold previously suggested for English models, extending up to 7+ epochs without saturation.
Benchmark Modernization: Released a cleaned, modernized suite of German evaluation benchmarks, correcting critical translation errors that previously compromised evaluation reliability.
Open Source Release: Released the BOLDT series of German Small Language Models (SLMs) and the cleaned datasets.

4. Key Results

Performance Gains:
- The 350M Dense Core model (trained on 28B tokens repeated ~3.6x) outperformed the 350M Random baseline (trained on 100B unique tokens) by 4.89 points on average across benchmarks.
- The 1B Dense Core model achieved a 5.14-point lead over its 1B Random baseline.
Efficiency: The BOLDT models achieved state-of-the-art results for their size while being trained on 10x to 360x fewer tokens than comparable multilingual models (e.g., Gemma-3, Llama-3, Qwen).
Instruction Tuning: The "density advantage" translated to instruction-tuning. Models pre-trained on high-quality cores showed higher correctness and helpfulness in assistant tasks compared to those pre-trained on diverse, noisy data.
Scaling: As model size increased (350M $\to$ 1B), the performance gap between high-quality repetition and diverse single-pass training widened, suggesting larger models are better at leveraging concentrated information.
Curriculum Failure: Hybrid "Phased" curricula (diverse $\to$ dense) underperformed pure dense-core training, suggesting that even a "warm-up" phase with low-signal data dilutes the learning trajectory.

5. Significance and Implications

Paradigm Shift for Non-English LLMs: The paper challenges the "maximize unique token volume" paradigm for non-English languages. It argues that for languages with limited data pools, semantic concentration via aggressive filtering is a more viable path to efficiency than maximizing diversity.
Resource Efficiency: It provides a blueprint for developing high-performance SLMs in resource-constrained settings, proving that high-quality data selection can compensate for a lack of compute and data volume.
Repetition is Not Harmful: Contrary to fears of overfitting or diminishing returns, the study shows that for high-signal data, repeated exposure is beneficial and necessary to extract maximum value from limited corpora.
Community Impact: By releasing the BOLDT models and corrected benchmarks, the authors provide the German NLP community with reliable tools for evaluation and high-quality base models, addressing a critical gap in non-English LLM infrastructure.

In conclusion, the paper establishes that for high-resource non-English languages, quality filtering and multi-epoch training are superior strategies to broad, single-pass training, enabling the creation of competitive language models with significantly reduced data requirements.