GottBERT: a pure German Language Model

Technical Summary: GottBERT: a pure German Language Model

Problem Statement
While pre-trained language models (LMs) like BERT and RoBERTa have revolutionized Natural Language Processing (NLP), the initial research focus was predominantly on English. Although multilingual models exist, single-language models often offer advantages in pre-training efficiency, resource utilization, and downstream task performance. Despite the rise of large prompt-based LLMs, smaller, compute-efficient BERT-like models remain critical for practical deployment. At the time of this work, there was a lack of a dedicated, single-language German RoBERTa model trained exclusively on a massive, high-quality German corpus. Existing German models (e.g., GermanBERT, dbmdz BERT) were primarily based on the original BERT architecture or utilized mixed data sources, and none had leveraged the full scale of the OSCAR corpus with the specific architectural optimizations of RoBERTa.

Methodology
The authors developed GottBERT, the first German single-language RoBERTa model, utilizing the German portion of the OSCAR (Open Super-large Crawled ALMAnaCH coRpus) dataset. The methodology involved several key stages:

1. Data Curation and Filtering:

   • Source: The model was trained on the first published deduplicated version of the German OSCAR corpus (145GB, ~21.5 billion words).
   • Filtering Strategy: Inspired by FlauBERT, the authors implemented a filtering pipeline to create a "clean" version (121GB, ~18.1 billion words). This process addressed:
     • Encoding Errors: Correcting erroneous umlauts caused by UTF-8 decoding issues and removing lines with non-reproducible characters.
     • Content Quality: Removing spam, word lists, and non-German documents.
     • Syntactic Heuristics: Utilizing a language detection algorithm (n-gram based) to filter ASCII art and a single-class SVM trained on 12k documents to identify "dirty" content. The SVM leveraged German-specific linguistic features, such as the capitalization of nouns and ratios of stopwords, punctuation, unique words, and uppercase tokens.
   • Vocabulary: A new vocabulary of 52k subword tokens was computed using Byte Pair Encoding (BPE) on 40GB of randomly sampled German text, resulting in a 40% reduction in binary data size compared to the English-trained GPT-2 tokenizer.

2. Pre-training Infrastructure:

   • Hardware: GottBERT is the first published RoBERTa model pre-trained on TPUs (Tensor Processing Units). The base model was trained on a 256-core TPUv3 pod, while large models used a 128-core TPUv4 pod.
   • Architecture: The models followed the standard RoBERTa architecture (removing segment embeddings and next-sentence prediction, using dynamic masking).
   • Hyperparameters: Training utilized the fairseq framework. Base models were trained for 100k steps with a batch size of 8k and a peak learning rate of 0.0004. Large models used a peak learning rate of 0.00015.
   • Models Generated: Four models were produced: GottBERTbase and GottBERTlarge (unfiltered data), and fGottBERTbase and fGottBERTlarge (filtered data).

3. Evaluation:

   • The models were fine-tuned and evaluated on five downstream tasks:
     • Natural Language Inference (NLI): XNLI (German portion).
     • Named Entity Recognition (NER): CoNLL 2003 (German) and GermEval 2014.
     • Text Classification: GermEval 2018 (coarse and fine-grained) and 10kGNAD (news topic classification).
   • Performance was measured using F1 scores and accuracy, with results averaged over 24 runs with varying hyperparameters to ensure robustness.

Key Contributions

• First German RoBERTa: The introduction of the first single-language German RoBERTa model, trained exclusively on the German OSCAR corpus.
• Filtering Methodology: The development and application of a specific filtering method for German texts, utilizing syntactic features (like noun capitalization) and machine learning (SVM) to remove low-quality data.
• Open Release: The release of all four model variants (base and large, filtered and unfiltered) under the MIT license to support the German NLP research community.
• Empirical Analysis: A comprehensive evaluation comparing these new models against existing German BERT models (GermanBERT, dbmdz BERT, GBERT, GELECTRA) and multilingual models (mBERT, XLM-RoBERTa).

Results

• Base Models: GottBERT demonstrated competitive performance, leading among base models in 4 out of 6 tasks. Specifically, GottBERTbase and fGottBERTbase achieved top or near-top scores in NER and text classification tasks.
• Large Models: The large models showed robust performance but were generally outperformed by GELECTRAlarge and GBERTlarge in several tasks. XLM-Rlarge achieved the highest F1 score on the GermEval 2014 NER task.
• Impact of Filtering: Contrary to the authors' expectations, the filtering of the OSCAR corpus did not significantly improve results. In some cases, the unfiltered models performed slightly better or comparably to the filtered ones. The authors suggest that the cleaning process may have inadvertently removed valuable text variance, which is known to benefit model performance.
• Comparison: While GELECTRA models generally achieved the highest accuracies (likely due to the ELECTRA architecture and potentially larger/more varied training data including OPUS and OpenLegalData), GottBERT established itself as a strong baseline, particularly in the base architecture category.

Significance and Claims
The paper claims that GottBERT serves as a valuable, efficient resource for the German NLP community, offering a state-of-the-art single-language foundation that rivals multilingual models in specific tasks. The authors highlight that while the specific data cleaning strategy did not yield the anticipated performance boost, the successful pre-training of a RoBERTa model on TPUs and the release of these models provide a critical infrastructure for future research.

The authors modestly note that their results are specific to the first version of the OSCAR corpus and that the models may struggle with cultural nuances or dialects not well-represented in the training data. They acknowledge that the filtering approach was a trade-off between computational cost and data quality, and suggest that future work should explore increasing corpus variance (e.g., incorporating Wikipedia or legal data) and utilizing Whole Word Masking (WWM) to further improve performance. The work underscores the continued relevance of efficient, single-language models despite the trend toward massive LLMs.