💻 computer science

ModernBERT-TR: A Modern Encoder Foundation Model for Turkish

The paper introduces ModernBERT-TR, a 150M-parameter Turkish encoder pretrained from scratch on 144.4 billion tokens using the ModernBERT architecture and a custom tokenizer, which significantly outperforms existing Turkish baselines and matches larger concurrent models despite being trained on substantially fewer resources.

Original authors: Besher Alkurdi, Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali

Published 2026-07-29

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Besher Alkurdi, Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, chaotic library where every book is written in a different language. For a long time, the librarians (computer scientists) built one massive, super-dense encyclopedia to help computers understand all the languages at once. But when you try to find a specific fact about a tiny, specific language inside that giant book, the information gets a bit blurry. It's like trying to hear a whisper in a crowded stadium; the signal gets lost in the noise.

To fix this, researchers started building smaller, specialized dictionaries just for single languages. The most famous of these for Turkish was built back in 2020. It was a good start, but it was built with old tools, like a bicycle when everyone else was driving sports cars. Meanwhile, the rest of the world moved on to "Modern" engines that could read faster, remember longer sentences, and understand the twists and turns of grammar much better. The big question was: Could we build a brand-new, super-fast sports car specifically for Turkish, using these modern tools, without needing a billion-dollar budget or a library the size of the moon?

This paper introduces ModernBERT-TR, a new artificial intelligence model designed to be the ultimate "Turkish brain" for computers. The researchers took the latest, most advanced architectural designs (the "ModernBERT" engine) and trained it from scratch using only Turkish text. They didn't just throw a massive amount of data at it; they carefully curated a balanced mix of high-quality Turkish writing and used a custom-made dictionary (tokenizer) specifically tuned for the unique way Turkish words are built.

The results are surprisingly powerful. Even though this new model is relatively small—containing about 150 million "neurons" (parameters)—it outperformed much larger models, including some with nearly four times as many neurons. When tested on 11 different Turkish tasks, like spotting hate speech or understanding movie reviews, ModernBERT-TR scored an average of 60.2%, beating the next-best model by a significant margin. It even matched a competitor that was trained on roughly 7 times more data (1 trillion tokens vs. the model's 144.4 billion tokens) in a full fine-tuning test, proving that smart training and better architecture can beat raw data volume.

The authors suggest that the secret sauce wasn't just the size of the model, but the quality of the data mix. They found that repeating a specific, high-quality Turkish dataset five times during training was far more important than tweaking other settings like batch size. In short, they built a lean, mean, Turkish-speaking machine that proves you don't need to be the biggest to be the best; you just need to be the most modern and the most focused. They have released all their code and the model itself to the public, hoping to give a huge boost to anyone trying to build technology that speaks Turkish.

Technical Summary: ModernBERT-TR

Problem Statement

Turkish Natural Language Processing (NLP) currently relies on outdated architectures or under-optimized multilingual models. The dominant monolingual encoder, BERTurk (2020), utilizes the vanilla BERT architecture and was trained on a relatively small corpus (~4.5 billion tokens). While multilingual models (e.g., mBERT, XLM-R) offer coverage, they dilute capacity across many languages, limiting per-language performance. Furthermore, recent architectural advancements that have significantly improved encoder quality for English—such as rotary position embeddings (RoPE), Gated Linear Units (GLU), alternating local-global attention, and sequence-packed training—have not been systematically applied to Turkish monolingual pretraining.

Methodology

The authors introduce ModernBERT-TR, a 150M-parameter encoder model built upon the ModernBERT architecture but pretrained from scratch specifically for Turkish.

Architecture and Training Configuration

Architecture: The model follows the ModernBERT-base configuration with 22 transformer layers, 768 hidden dimensions, and 12 attention heads. Key architectural improvements over vanilla BERT include:
- RoPE: Rotary position embeddings replace learned absolute positions to improve generalization to varying sequence lengths.
- GLU: Gated Linear Units with GELU activation are used in feed-forward networks.
- Pre-normalization: Layer normalization is applied before attention and feed-forward sublayers to improve gradient flow.
- Attention Mechanism: A sliding window of 128 tokens is used in most layers, with full global attention applied every 3rd layer to balance efficiency and long-range dependency modeling.
- Training Efficiency: The model utilizes Flash Attention and sequence-packed training (concatenating variable-length documents into fixed-length batches) to eliminate padding waste.
Tokenizer: A custom 50K WordPiece tokenizer was trained from scratch on Turkish data. This size was selected to balance compression efficiency against embedding table size, achieving a high vocabulary efficiency compared to larger (128K) alternatives.
Training Data: The corpus consists of 144.4 billion tokens (trained over 2 epochs). The data is a balanced mixture of:
- FineWeb-2 Turkish: 41.2B tokens (web-crawled).
- BertTurk Corpus: 6.2B tokens, upsampled 5× (31.0B effective tokens) to balance the mixture and leverage higher-quality text.
Hyperparameters: Training used the Decoupled StableAdamW optimizer with a peak learning rate of $2 \times 10^{-4}$ , a global batch size of 256 sequences (262K tokens), and a cosine learning rate schedule with 1B token warmup. The model was trained on 4× NVIDIA H100 GPUs for 623 GPU-hours.

Ablation Studies

Systematic ablations were conducted on learning rate, batch size, cosine schedule final ratio ( $\alpha_f$ ), and data mixture. The results indicated that data composition (specifically the 5× upsampling of the BertTurk Corpus) and learning rate were the primary drivers of downstream performance, while batch size and $\alpha_f$ had minimal effects within the tested ranges.

Key Contributions

ModernBERT-TR Model: The first publicly available Turkish encoder leveraging ModernBERT architectural advances (RoPE, GLU, alternating attention, unpadded training).
Custom Tokenizer: A 50K WordPiece tokenizer optimized for Turkish morphology, demonstrating superior vocabulary efficiency.
Curated Corpus: A balanced training corpus combining FineWeb-2 and the BertTurk Corpus, with empirical evidence supporting the upsampling of higher-quality data.
Systematic Ablations: Identification of data mixture and learning rate as critical factors for Turkish monolingual pretraining quality.
State-of-the-Art Results: Achievement of new benchmarks on Turkish NLP tasks with a model significantly smaller than leading multilingual alternatives.
Open Release: Public release of model weights, tokenizer, evaluation code, and training configuration.

Results

Frozen-Encoder Linear Probing

Evaluated on 11 diverse Turkish NLP tasks (classification, regression, and token classification) using a frozen-encoder protocol:

Overall Performance: ModernBERT-TR achieved an average score of 60.2%, surpassing the next-best model (Turkish-E5-large, 560M parameters) by 13.1% relative and the conventional BERTurk baseline by 70.3% relative.
Efficiency: Despite having only ~150M parameters (approx. 1/4 the size of the E5-large variants), ModernBERT-TR outperformed them across all task categories.
Task Specifics: The model led in 5 of 11 tasks, including offensive language detection, hate speech classification, linguistic acceptability (TrCOLA), and Named Entity Recognition (WikiNER).

Full Fine-Tuning (TabiBench)

Evaluated on the 28-task TabiBench benchmark using full fine-tuning:

ModernBERT-TR scored 77.28, matching the concurrent TabiBERT (trained on ~1 trillion tokens with mixed languages) within 0.30 points.
ModernBERT-TR led in 5 of 8 task categories (Text Classification, Semantic Textual Similarity, NLI, Academic Understanding, Information Retrieval).
TabiBERT maintained an advantage in Code Retrieval and Question Answering, attributed to its inclusion of code and mathematical data in pretraining.

Significance

The paper demonstrates that modern architectural innovations combined with targeted monolingual pretraining can produce representations that are more linearly separable and effective than larger multilingual models, even when trained on significantly fewer tokens (144.4B vs. 1T).

The authors claim that for resource-constrained monolingual pretraining, careful data curation and mixing (specifically the upsampling of high-quality corpora) yield greater benefits than simply scaling batch sizes or tuning schedule parameters. By releasing ModernBERT-TR, the authors aim to accelerate Turkish NLP research, providing a foundation model that outperforms existing baselines while requiring only 623 GPU-hours to train.