MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum
This paper introduces MoganBert-TR, a 149M-parameter Turkish encoder foundation model trained from scratch using a novel CLM-to-MLM curriculum and specialized long-context scheduling, which achieves state-of-the-art performance on Turkish NLP benchmarks and yields a highly efficient embedding model that reaches 99.5% of its 7.57B-parameter teacher's score.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast landscape of artificial intelligence, there are two main ways computers learn to understand human language. One approach, often in the news, involves models that generate text, writing stories or answering questions from scratch. The other, quieter but equally vital approach, uses models that act as powerful search engines and classifiers. These "encoder" models read a sentence and turn it into a compact mathematical summary, allowing computers to compare ideas, find relevant documents, or sort information with incredible speed. For years, the standard method for teaching these models has been a specific type of guessing game: the computer sees a sentence with some words hidden and must predict the missing parts. This method, known as masked language modeling, has worked well, but researchers have long wondered if there is a better way to teach the machine the structure of a language, especially for languages that are complex and rich in word forms, like Turkish.
A team of researchers has now built a new foundation for Turkish language understanding, demonstrating that the way a model is taught matters just as much as the architecture it is built on. They created a model called MoganBert-TR, trained from the ground up on a massive collection of Turkish text. Instead of sticking to the traditional guessing game, they introduced a two-stage learning curriculum. First, the model learned to read text from left to right, predicting the next word in a sequence, much like a human reading a book. Only after this initial phase did they switch to the traditional method of hiding words and asking the model to fill them in. This shift in teaching strategy, combined with a carefully cleaned dataset and a new way of handling long sentences, resulted in a model that understands Turkish better than previous versions, particularly when it comes to finding relevant information in large databases.
The researchers began by gathering a massive amount of text from the internet, including recent web pages and older archives, but they did not simply dump this data into the training system. They built a sophisticated pipeline to filter out low-quality content, such as spam or repetitive advertisements, and to ensure the text was genuinely useful. Because there was no existing tool to automatically judge the quality of Turkish text, they trained a smaller, specialized model to act as a teacher, which then taught a faster system how to make the same quality judgments. This allowed them to process millions of documents efficiently, keeping only the high-quality material. They also created a new dictionary for the model, one that breaks Turkish words into smaller pieces more efficiently than previous attempts, which is crucial because Turkish words can change form significantly depending on their role in a sentence.
The core of their discovery lies in the order of operations during training. They tested whether starting with the left-to-right prediction method before switching to the word-hiding method made a difference. In a controlled experiment using the same amount of computing power and the same data, the new two-stage approach vastly outperformed the traditional method when it came to retrieving information. While both methods performed similarly on simple classification tasks, the new approach was nearly four times better at finding relevant documents in a search test. The researchers discovered that the traditional method caused the model's internal representations to collapse into a narrow, unhelpful shape, where all meanings looked too similar. The new curriculum prevented this collapse, allowing the model to keep a wider, more distinct view of language meaning.
To further refine the model, the researchers experimented with how the computer learned during the final stages of training. They split the process into two paths: one where the model continued to learn with short sentences, and another where it learned with much longer sentences. By comparing the results of these two paths, they found that finishing the training with shorter sentences actually improved the model's overall performance more than finishing with long ones. This finding was surprising, as longer contexts are often assumed to be better. They also tested a technique called "model soup," where they averaged the weights of different trained models together, hoping to get the best of both worlds. However, their simple branching method, which cost only a tiny fraction more to run, produced a better result than the complex averaging technique.
The final product, MoganBert-TR, achieved the highest score among Turkish models based on the modern architecture used in this study. It performed exceptionally well on tasks involving code retrieval, finding the right snippets of programming code from natural language descriptions, a task where it significantly outperformed older models. This success was attributed to the combination of their new dictionary, which preserved the spacing and structure of code, and the deliberate inclusion of a high amount of programming text in the training data. The team also created a companion model designed specifically for creating search-friendly summaries of text. By teaching this smaller model to mimic a much larger, more powerful teacher, they achieved a result that was nearly as good as the giant teacher but required fifty-one times less computing power to run.
The work highlights that for languages like Turkish, simply updating the structure of a model is not enough; the training method itself must be rethought. The researchers showed that a two-stage curriculum, starting with sequential prediction and moving to word masking, creates a more robust understanding of the language. They also demonstrated that the final stages of training are a critical design space where small changes, like the length of the text being processed, can have a measurable impact on performance. While the model still has some weaknesses in specific areas, such as understanding sentence structure in certain grammatical tests, it represents a significant step forward. The team has made their model, the dictionary, and the code used to build it available to the public, allowing others to build upon this foundation for Turkish language technology.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.