💬 NLP

KletterMix: Climbing Toward High-Quality German Pretraining Data

The paper introduces KletterMix, a high-quality German pretraining corpus created by translating a state-of-the-art English dataset while preserving its structure and diversity, and demonstrates through controlled experiments that models trained on this curated data achieve measurable improvements in downstream German-language tasks compared to existing resources.

Original authors: Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia, Abbas Goher Khan, Mehdi Ali, Michael Fromm, Kristian Kersting

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia, Abbas Goher Khan, Mehdi Ali, Michael Fromm, Kristian Kersting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "German Language Gap"

Imagine you are trying to teach a robot how to speak and think. To do this, you need to feed it a massive library of books, articles, and websites (this is called pretraining data).

For a long time, English has had the best library in the world: huge, diverse, and carefully organized. But the German library has been much smaller, messier, and less organized. It's like trying to build a skyscraper in English with a full toolbox, but trying to build one in German with a few scattered hammers and some rusty nails.

The authors of this paper asked: Can we fix the German library by translating the best English library?

The Solution: KletterMix (The "Climbing Mix")

The team created KletterMix (German for "Climbing Mix"). Think of it as taking the most carefully curated, high-quality English "recipe" for training AI and translating it into German.

But they didn't just use a simple "copy-paste" translator. They built a sophisticated pipeline to ensure the German version kept the same structure and quality as the English original.

The Analogy: The Architectural Blueprint
Imagine the English data is a set of blueprints for a complex, beautiful city.

Old Method: Just translating the words on the blueprints. You might end up with a city where the roads don't connect or the buildings are the wrong size.
KletterMix Method: They translated the blueprints but kept the exact layout, the street names, the zoning laws, and the metadata. They ensured that if a building in the English blueprint was a "library," the German version was also a "library," not a random house. They preserved the "soul" of the original data.

How They Built It (The Pipeline)

Building this wasn't easy. They had to solve three main puzzles:

The "Size" Problem: Some English documents are tiny tweets; others are massive technical manuals. A translator that works for a tweet might choke on a manual.
- The Fix: They sorted documents into "buckets" based on size. Short docs got one translation setting; long docs got a special setting that could handle more text without breaking.
The "Context" Problem: If you translate a long book page by page without looking at the previous page, the story might get weird.
- The Fix: They used a "context window." When translating page 2, the system was allowed to peek at the German translation of page 1 to keep the tone and style consistent.
The "Quality Control" Problem: How do you know the translation is good without reading every single word?
- The Fix: They used a "smart filter." First, they used a high-tech AI (COMETKiwi) to grade a sample of the translations. Then, they trained a cheaper, faster AI (a "proxy") to look at the German text and guess the quality score based on patterns (like sentence length, weird characters, or repetition). This allowed them to filter out the bad translations without needing to read the original English text again.

Did It Work? (The Results)

The team tested this new German data by training a small AI model (0.6 billion parameters) on it. They compared it against other existing German datasets.

The Analogy: The Training Camp
Imagine three runners training for a race:

Runner A (GermanWeb): Trained on a mix of random German websites.
Runner B (FineWeb2-DE): Trained on a filtered German web crawl.
Runner C (KletterMix): Trained on the translated, high-quality English mix.

The Race Results:

Runner C (KletterMix) didn't just run faster; they ran smarter.
When tested on general knowledge (MMLU) and physical common sense (PIQA), Runner C was competitive with the best.
The Real Win: Runner C crushed the tests that required reasoning and storytelling (HellaSwag and ARC-Challenge).
- Why? Because the original English data was full of coherent stories, logical explanations, and structured arguments. By translating that structure into German, the AI learned how to think logically in German, not just how to speak it.

They also tested "Annealing" (a technique where you train a model on one dataset and then fine-tune it on another). When they took a model trained on standard German data and gave it a "top-up" dose of KletterMix, the model's reasoning skills jumped significantly.

The Catch (Limitations)

The authors are honest about the flaws:

Cultural Bias: Since the source is English, the German data might still carry American or British cultural biases, even after translation.
"Translationese": Sometimes, the German might sound a little stiff or unnatural, like a sentence translated by a robot rather than written by a native poet.
Not a Replacement: This isn't a magic wand that replaces the need for native German data. It's a powerful supplement that fills the gaps.

The Bottom Line

KletterMix proves that you don't always need to start from scratch to build a great dataset for a non-English language. If you take a high-quality, well-organized English dataset and translate it carefully—preserving its structure and filtering out the bad parts—you can create a German dataset that helps AI models think and reason much better than they could with standard German web data alone.

It's like realizing that to build a better German library, you don't just need more German books; you need to import the best English books, translate them perfectly, and put them on the shelves in the right order.

Technical Summary: KletterMix: Climbing Toward High-Quality German Pretraining Data

Problem Statement

High-quality pretraining data is a primary determinant of language model performance. While English has benefited from large, diverse, and well-documented pretraining corpora (e.g., ClimbMix, Dolma), German-language resources remain comparatively underdeveloped. Existing German datasets are often derived from noisy web crawls, embedded as subsets of multilingual corpora, or released with limited documentation and validation through controlled training experiments. Native German web crawling alone struggles to replicate the source diversity, mixture design, and documentation standards of top-tier English datasets without aggressive filtering that may remove useful long-tail content. Furthermore, strong German model behavior cannot be assumed to emerge solely from English-centric data due to specific linguistic features such as morphology, compounding, capitalization, and regional variation.

Methodology

The authors introduce KletterMix, a 725B-token German pretraining and annealing corpus constructed by translating ClimbMix, a state-of-the-art English pretraining mixture, into German. The construction follows a five-stage pipeline designed to preserve document boundaries, metadata, source structure, and topical diversity:

Source-Record Normalization: The English source is stored as sharded JSONL files containing document identifiers, text, and metadata (source cluster, location, approximate length). This structure is preserved throughout the pipeline.
Length-Aware Routing: To handle variable document lengths efficiently, documents are routed into context buckets (short, medium, long, overflow) based on their source length. This allows for bucket-specific source and target budgets, reducing padding and scheduling inefficiencies.
Document-Preserving Translation:
- Short documents are translated in a single pass.
- Long documents are segmented into sentences and greedily packed into chunks. If a sentence exceeds the chunk budget, it is split at the token level.
- For chunked documents, the translation of chunk $t-1$ is included as a truncated context window for chunk $t$ to maintain discourse continuity and terminology consistency.
- Dynamic target-side budgeting is used, where the maximum generation length is derived from the source chunk length ( $\ell_{max}^{tgt} = \min(L_{max}, \lceil \alpha\ell_{src} + \beta\rceil)$ ) to accommodate German's tendency for surface-length expansion.
Scalable Shard-Wise Execution: Translation is performed by parallel workers on disjoint subsets of the corpus. Workers preserve original identifiers for traceability and implement resume capabilities for interrupted runs.
Quality Estimation and Filtering:
- Pilot Subset: A stratified sample of the corpus is scored using COMETKiwi, a reference-free quality estimation model, to generate high-quality labels.
- Proxy Training: A gradient-boosted regression model is trained on the pilot data to predict COMETKiwi scores using only target-side features (German text) and inexpensive metadata (e.g., GlotLID language identification, text shape, repetition ratios).
- Filtering: The proxy model scores the full corpus. Three filtered 12B-token training splits are created by retaining documents with proxy scores $\ge 0.50$ , $\ge 0.55$ , and $\ge 0.60$ .

Key Contributions

KletterMix Dataset: A 725B-token German corpus constructed from a high-quality English source while preserving document-level structure, metadata, and mixture design.
Scalable Translation Pipeline: A methodology for translating pretraining data that includes length-aware routing, document-preserving chunking, dynamic budgeting, and shard-wise execution.
Quality Diagnostics: A comprehensive documentation of the corpus using COMETKiwi-based and proxy-based quality estimation, alongside analyses of length distributions, topic coverage, and metadata preservation.
Empirical Validation: Controlled pretraining and annealing ablations comparing KletterMix against established German corpora (FineWeb2-DE and GermanWeb).

Results

The authors evaluated KletterMix using a Qwen3-0.6B model under matched 12B-token training budgets.

Training Dynamics: Models trained on KletterMix achieved lower training and validation loss compared to models trained on FineWeb2-DE and GermanWeb, suggesting improved sample efficiency.
Downstream Benchmarks: On a suite of German-language benchmarks (MMLU, PIQA, HellaSwag, ARC-Challenge), KletterMix variants generally outperformed the baselines in aggregate (Core Avg).
- The filtered KletterMix (score $\ge 0.60$ ) achieved the highest Core Avg (40.2), outperforming FineWeb2-DE (38.3) and GermanWeb (36.8).
- KletterMix showed particular strength in HellaSwag (grounded commonsense inference) and ARC-Challenge (science-style reasoning), suggesting the corpus effectively transfers coherent explanatory structures.
- Annealing Experiments: When annealing a model pre-trained on FineWeb2-DE, switching to KletterMix yielded a higher Core Avg (39.4) than switching to GermanWeb (37.6), with gains concentrated in HellaSwag and ARC-C.
Quality Analysis: Corpus diagnostics revealed a heavy-tailed length distribution consistent with the source. However, "lower tails" in long-context buckets (where long English documents resulted in unusually short German translations) were identified as potential failure modes requiring audit.

Significance and Claims

The paper argues that carefully curated translated data can substantially strengthen the German pretraining data ecosystem. The significance of KletterMix lies not in replacing native German data, but in demonstrating that the curation decisions, source diversity, and mixture structure of a strong English corpus can be transferred to German through high-quality translation and rigorous validation.

The authors claim that this approach transfers more than just surface form; it transfers useful mixture structure. The results suggest that translated data can improve model capabilities in areas requiring structured context, such as event-level plausibility and compositional reasoning. However, the paper maintains a modest stance, acknowledging that:

Translated data may inherit cultural, geographic, and licensing biases from the English source.
Machine translation can introduce artifacts (translationese, semantic drift) and fail on specialized documents.
Proxy-based filtering is a practical ranking signal for fixed-budget training but not a universal quality law (as MMLU performance did not improve monotonically with stricter filtering).
The current results are limited to a specific model size (0.6B) and token budget, necessitating future work on larger models and broader benchmarks.

Ultimately, KletterMix presents a practical route for non-English pretraining that prioritizes documentation, empirical validation, and the preservation of dataset structure over simple scale or native crawling alone.