Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems
This paper quantifies the significant tokenization overhead faced by Ukrainian and other Cyrillic languages in modern AI systems compared to English, demonstrating that this disparity stems from training data allocation but can be effectively mitigated through inference-time compression techniques or by training balanced byte-level BPE tokenizers.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a computer reads a sentence, it does not see words the way a human does. Instead, it breaks the text down into small chunks called tokens, which are the basic units the machine uses to understand meaning and calculate costs. For languages written with the Latin alphabet, like English, these chunks are often efficient because the letters themselves take up very little space in the computer's memory. However, for languages written in the Cyrillic script, such as Ukrainian, the situation is different. The characters used in these scripts require more digital space to store, and because most of the data used to train artificial intelligence is in English, the computer has learned to handle English text very efficiently while struggling with Cyrillic. This creates a hidden tax: to process the same amount of information, a computer might need to break a Ukrainian sentence into nearly twice as many pieces as an English one, making the process slower and more expensive.
A recent study by independent researcher Ivan Dobrovolskyi investigates this disparity, focusing specifically on Ukrainian but drawing conclusions that apply to many other languages using the Cyrillic script. The research asks a simple but critical question: why does the computer struggle so much with these languages, and can we fix it without waiting for a complete overhaul of the technology? The study suggests that the problem is not just about the shape of the letters or how much space they take up, but rather about how the computer was taught to read them. By analyzing how different modern systems break down text, the researcher found that the imbalance comes from the training data itself. The computer has seen far more English text than Ukrainian text, so it has learned fewer shortcuts for the Ukrainian language.
To measure the scale of the problem, the researcher tested nine different commercial systems used by major technology companies. They fed these systems millions of words of Ukrainian text and compared how many pieces each system created against how many pieces were created for English text. The results showed a significant gap. On the most modern systems, the Ukrainian text required between 68 percent and 121 percent more pieces than the English text. On older systems, the gap was even wider, reaching 220 percent more pieces. This means that for every dollar spent to process English text, a user might spend nearly double that amount to process the same amount of Ukrainian text, simply because the computer has to do more work to break the words down.
The study also looked at whether switching to the Latin alphabet, a process known as romanization, would solve the problem. Some have suggested that writing Ukrainian with Latin letters would make it cheaper and faster for computers to read. However, the research found the opposite to be true for modern systems. Because these systems have already learned efficient ways to handle the native Cyrillic letters found on the Ukrainian web, forcing the text into Latin letters actually confused the computer. This caused the text to be broken into even more pieces, increasing the cost by up to 19 percent on some systems. The computer works best with the script it has seen most often in its training data, and for Ukrainian, that is the native Cyrillic script.
To understand if the problem could be fixed at the source, the researcher created a new, custom system designed to treat English and Ukrainian equally. Instead of letting the computer learn from the vast, English-heavy internet, this new system was trained on a perfectly balanced mix of English and Ukrainian text. The results were striking. When this balanced system was used, the gap in efficiency shrank dramatically. The Ukrainian text no longer required twice as many pieces; instead, it required only about 30 percent more pieces than the English text. This proves that the high cost is not an unavoidable physical limit of the letters themselves, but a result of how the computer was trained. If the training data is balanced, the computer learns to be efficient for both languages.
The research also explored a way to reduce costs without changing the computer's training. By using a technique that intelligently removes unnecessary words from the text before the computer reads it, the researchers were able to cut the amount of Ukrainian text by nearly half. Crucially, this compression did not hurt the computer's ability to find the right answers in a test involving thousands of real products and customer questions. The system still retrieved the correct prices, ratings, and details just as accurately as it did with the full, uncompressed text. This suggests that while the training imbalance is a deep-rooted issue, there are practical ways to lower costs right now while the technology catches up.
Ultimately, the study reveals that the inefficiency in reading Cyrillic languages is a solvable problem of data allocation, not an inherent flaw in the scripts. The computer is not broken; it is simply biased toward the data it has seen most. By adjusting the training data to include more of these underrepresented languages, or by using smart compression tools, the digital divide in cost and speed can be narrowed. The findings offer a clear path forward for making artificial intelligence more equitable, ensuring that the benefits of these powerful tools are not limited to those who write in the most common scripts.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.