One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography
This paper demonstrates that pretraining multilingual language models on romanized text, rather than native orthography or IPA, significantly enhances cross-lingual knowledge transfer across diverse writing systems, suggesting that romanization should be a core pretraining design choice rather than a post-hoc adaptation.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where a single computer program could read, write, and understand every language on Earth with equal fluency. This is the promise of multilingual artificial intelligence, a field that has made incredible strides in recent years. These programs learn by reading vast amounts of text, absorbing patterns of grammar and meaning. A key part of how they work is a shared dictionary of word fragments, or subwords, that allows them to recognize similarities between different languages. If two languages use the same alphabet, like English and Spanish, they naturally share many of these fragments, making it easy for the computer to transfer knowledge from one to the other. However, the world is filled with languages that use entirely different writing systems, such as the Cyrillic script of Russian, the Devanagari script of Hindi, or the Arabic script of Urdu. When these scripts do not overlap, the computer's shared dictionary breaks down, and the ability to learn from one language to help another vanishes. This barrier has long been a stumbling block for creating truly universal language models.
To solve this, researchers have tried to force different writing systems into a common shape. One approach is to convert text into the International Phonetic Alphabet, a universal set of symbols that represents human speech sounds. Another is to transliterate text into the Latin alphabet, the script used for English, effectively rewriting every language in a single, shared script. While these methods have shown promise in smaller experiments, it remained unclear which approach was truly superior when building large, powerful models from scratch, or whether simply converting text after the model was already trained would work just as well. A team of researchers at Ohio State University and the University of Washington set out to answer these questions by building and testing a series of language models designed to handle eight distinct languages spanning four different language families.
The researchers constructed a controlled experiment using eight languages: English and Spanish, Russian and Polish, Hindi and Urdu, and Tamil and Malayalam. These pairs were chosen because they offered a mix of relationships; some shared scripts, while others shared sounds but used completely different writing systems. They built three versions of a language model for each of three different sizes, ranging from roughly 467 million to over one billion parameters. For each size, they trained one model on the original native scripts, a second on text converted to the International Phonetic Alphabet, and a third on text converted to the Latin alphabet. Crucially, they kept everything else identical: the amount of data, the computer architecture, and the training time. This allowed them to isolate the effect of the writing system itself on the model's ability to learn and transfer knowledge.
The results were striking and clear. The models trained on text converted to the Latin alphabet, or romanized, consistently outperformed the others across every test. Whether the task was answering questions, understanding the meaning of sentences, or summarizing news articles, the romanized models showed the strongest ability to transfer knowledge from one language to another. This advantage grew larger as the models became bigger. The models trained on the phonetic alphabet also performed better than those trained on the original scripts in most cases, but they generally fell short of the romanized models. The only exception was the pair of Hindi and Urdu; because these two languages are nearly identical in how they sound but use completely different scripts, the phonetic approach worked exceptionally well for them, matching the performance of the romanized models. However, when the models were tested on languages they had never seen before, the romanized approach remained the most robust.
Perhaps the most surprising finding concerned a common strategy used in the field: taking a model already trained on native scripts and fine-tuning it on romanized data later on. Many researchers had assumed this would be an efficient shortcut, allowing them to gain the benefits of romanization without the cost of retraining the entire model. The study found the opposite to be true. When the researchers applied this shortcut to their models, performance actually dropped significantly on the languages the model already knew. The intervention only helped when the model was being asked to handle a language it had no prior exposure to, and even then, the improvement was modest compared to training the model on romanized text from the very beginning. This suggests that the benefit of romanization is not just about the final output format, but about how the model learns the fundamental structure of language during its initial training phase.
The study also looked at the efficiency of these different writing systems. Languages that use complex scripts, like Tamil and Malayalam, required significantly more computer tokens to represent the same amount of text compared to languages using the Latin alphabet. This meant that training on the original scripts was not only harder for the model to learn across languages but also more expensive and slower. Both the phonetic and romanized approaches compressed this text, bringing the cost and speed of these languages in line with the others. This compression, combined with the improved ability to share knowledge across different scripts, made the romanized approach the most effective design choice for building multilingual models that span diverse writing systems.
Ultimately, the research indicates that for artificial intelligence to truly bridge the gap between the world's many languages, the choice of how to represent text is a fundamental design decision, not a minor technical detail. The evidence suggests that converting diverse scripts into a shared Latin form before training the model yields the best results, allowing the system to internalize a unified structure that works across linguistic boundaries. While converting text to phonetic symbols offers some benefits, particularly for languages that sound alike but look different, it does not replace the advantages of a unified script. The findings challenge the idea that simply adjusting a model after it is built is enough; instead, they point to the need for a deliberate, upfront strategy in how these digital minds are taught to read the world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.