Modern computers that understand human language do not read words the way we do. They do not see a sentence as a flowing stream of meaning; instead, they see a long list of small, discrete pieces. To make this work, engineers must first break every sentence down into these tiny fragments, a process called tokenization. For decades, the standard method for doing this has been purely statistical. The computer looks at a massive library of text and counts how often certain letter combinations appear together. If two letters or small groups of letters show up together very frequently, the computer decides to glue them into a single unit. This approach is excellent at compressing data, allowing the computer to process vast amounts of text quickly. However, it treats language like a bag of random letters rather than a structured system of meaning. It ignores the deep rules of how words are built, such as how a root word changes when you add a prefix or a suffix. This blind spot becomes a major problem for languages that are rich in these structural rules, particularly many languages of India, where words are formed by combining complex syllables and grammatical markers in ways that standard computer methods often fail to respect.

Researchers at Motilal Oswal Financial Services and the Indian Institute of Technology Bombay have developed a new approach to fix this problem, calling it SuTRA. They observed that the standard statistical method often breaks words apart in ways that destroy their meaning. They named this destructive phenomenon "morphological shattering." Imagine a word that consists of a core meaning and a grammatical ending. A standard computer might split the word right in the middle of the core, separating the essential meaning from its beginning or end, or it might fuse a prefix with the root in a way that obscures their distinct roles. This is especially damaging for Indic languages, which use scripts where a consonant and a dependent vowel form a single, indivisible visual unit called an akshara. Standard tokenizers frequently split these units apart, separating the vowel from the consonant it belongs to, and they often ignore the boundaries between the root of a word and its grammatical attachments. The result is a fragmented representation where the computer struggles to understand the true semantic core of a word, making it harder for the machine to learn or translate effectively.

To solve this, the team created a new algorithm that respects the natural structure of these languages. Instead of letting the computer decide how to split words based solely on how often letters appear together, they taught the system to recognize the building blocks of the language first. They began by creating a new, verified dataset for Hindi, Marathi, and Gujarati. This was not a simple list of words; it was a detailed map showing exactly where the roots of words begin and end, and where the grammatical markers attach. Because existing resources were incomplete or relied on imperfect rules, the researchers used a large language model to verify and correct the splits, ensuring that every word in their dataset was broken down according to its true linguistic roots. This gold-standard dataset became the foundation for training their new tokenizer.

The new method, SuTRA, works in two stages. First, it looks at the text and groups the letters into their natural, indivisible syllable units, ensuring that a vowel never gets separated from the consonant it modifies. It also marks the boundaries where a root word ends and a grammatical suffix begins, based on the verified dataset. In the second stage, it builds its vocabulary by merging these units together, but with a crucial difference: it is forbidden from merging across the boundaries it just identified. If the computer tries to glue a prefix to a root in a way that violates the linguistic structure, the system penalizes that move. It forces the computer to learn the valid roots first, treating them as solid, unbreakable blocks, before it is allowed to combine them with other parts. This ensures that the final pieces the computer uses to understand language always contain a complete, meaningful root.

The results of this structural approach were significant. When tested against the standard methods, the new tokenizer showed a much better ability to keep words intact. In Hindi, it improved the alignment between the computer's word pieces and the actual linguistic roots by nearly fifteen percent. In Marathi, the improvement was even more pronounced, reaching over thirty-four percent in terms of how well the computer could recover the original meaning of a word from its fragments. This structural clarity translated directly into better performance on real-world tasks. When the researchers used the new tokenizer to train a machine translation system to translate between Hindi and Marathi, the quality of the translation improved noticeably. The system produced fewer errors and captured the nuances of the language more accurately than systems using the old, purely statistical methods. Furthermore, the new tokenizer proved to be more robust; when the input text contained small typos or minor spelling variations, the new system maintained the integrity of the word's root, whereas the old systems often fell apart, treating the typo as a signal to completely re-segment the word.

The researchers also demonstrated that this approach did not come at the cost of efficiency. While the new method created slightly more pieces for some words to ensure they were linguistically correct, it did not create an excessive number of fragments that would slow down the computer. The system managed to keep the vocabulary size manageable while achieving a much deeper understanding of the language's structure. By prioritizing the natural boundaries of the language over simple frequency counts, the team showed that it is possible to guide artificial intelligence to respect the logic of human speech. This work suggests that for languages with complex structures, the path to better artificial intelligence lies not just in feeding the computer more data, but in teaching it to see the language the way its speakers do: as a coherent system of roots and attachments, rather than a chaotic stream of letters. The findings offer a practical blueprint for improving how machines handle the world's most linguistically diverse languages, ensuring that the technology serves the structure of the language rather than breaking it apart.