💬 NLP

Morphology Aware Reversible Semantic Tokenization and Hierarchical Word Composition for Tamil Language Models

This paper proposes a morphology-aware, reversible semantic tokenization system with hierarchical word composition for Tamil that outperforms existing baselines in translation quality while significantly reducing sequence length and inference costs under a fixed model budget.

Original authors: Anand Murugan

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anand Murugan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to speak a new language. To do this, you have to break the language down into tiny building blocks that the robot can understand. In the world of computer science, this is called "tokenization." Think of it like a chef chopping ingredients: a standard chef might chop everything into tiny, uniform cubes (like "subword" tokens), which is fast and works for almost any recipe. But for a complex dish like a traditional Tamil curry, chopping everything into identical cubes might destroy the delicate layers of flavor. Tamil is a language where a single word can be a whole sentence packed with meaning—telling you who did what, when, and how. If you chop that word apart without understanding its grammar, you lose the recipe.

This paper explores a different way to chop the ingredients. Instead of just cutting words into random pieces, the researcher built a system that understands the "grammar anatomy" of Tamil words before they are fed to the robot. They ask: Can we teach the robot the exact rules of how Tamil words are built (like a stem plus a tense plus a case marker) so it understands the meaning better? And if we do give it all that extra detail, can we still make the robot fast enough to be useful, or does it get bogged down by too much information?

The researcher, Anand Murugan, tackled this by creating a special "Morphology-Aware" system for Tamil. They started by building a massive digital library of Tamil word rules using 12 different "Finite-State Transducers" (think of these as super-precise, rule-following robots that can break a word down into its parts, like a stem, a tense, and a number). They tested two main ways to feed this information to a translation model (a computer program that translates Tamil to English).

The first method, called "morphology-flat," was like laying out every single ingredient on the table separately. If a word meant "by the tree," the system didn't just say "tree"; it said "tree" + "noun" + "plural" + "by." This gave the model a huge amount of detail. The result? It was the best at translating. In a strict test of 3,539 sentences, it achieved a score of 10.63 (BLEU), beating other popular systems by a noticeable margin. It was the most accurate, but it was also "verbose," meaning it had to process a lot more tokens (building blocks) for every single word, which makes the computer work harder.

The second method, called the "word-composer," was a clever compromise. It tried to keep the best of both worlds. It kept the main "stem" of the word (the lemma) visible, but it bundled all the tiny grammar details (like tense and case) into a single "summary" token. Then, just before the model finished its translation, it would peek back at the original detailed notes to make sure it got the specifics right. This approach was much more efficient. It reduced the number of steps the computer had to take by about 59.3% (dropping from an average of 71.48 steps per sentence down to 29.08). While it was slightly less accurate than the "flat" method on very long, formal sentences, it still beat all the other external systems tested and was much faster to run.

The paper explicitly rules out a few ideas that might seem obvious. For instance, they tried putting the "peek back" step earlier in the process, but that didn't work; the model needed to see the whole sentence context first before looking up the grammar details. They also tried splitting the grammar summary into two parts, but that just added extra work without improving the translation. The author is careful to note that while their system works great for small models with limited data, they haven't proven yet if this advantage holds true for massive, super-powerful AI models that have seen the entire internet.

In the end, the study shows that for Tamil, understanding the "anatomy" of words helps the computer translate better than just guessing the chunks. However, you don't have to lay every single bone on the table to get a good result; a smart summary that knows where to look for the details works almost as well, but with much less effort. It's a reminder that sometimes, giving a machine a little bit of human-like grammar knowledge can make it a much better translator, especially when it's learning a complex language like Tamil.

Technical Summary: Morphology-Aware Reversible Semantic Tokenization and Hierarchical Word Composition for Tamil Language Models

1. Problem Statement

Statistical subword tokenizers (e.g., BPE, SentencePiece) efficiently process text but often fail to align with lexical or grammatical structures. This misalignment is particularly detrimental for Tamil, a highly inflected language where a single surface word can encode stems, case, number, tense, agreement, voice, clitics, and linked verbs. Standard tokenizers may split these complex words into arbitrary subword units that obscure semantic relationships. Furthermore, while morphology-aware tokenization offers explicit linguistic signals, it often increases sequence length (token count), leading to higher computational costs in attention mechanisms.

This work addresses two core questions:

Can Tamil morphology be exposed to a neural model via a deterministic, semantically explicit, auditable, and exactly reversible representation?
Can a model exploit this verbose representation without incurring the full computational cost of flat attention over every morphological factor?

2. Methodology

The proposed system is structured into three distinct layers:

A. Morphological System (FST-Based)

The foundation is an extension of the open-source ThamizhiMorph analyzer and generator.

Architecture: Built using 12 Finite-State Transducers (FSTs) covering nouns, verbs (single and chained), adjectives, adverbs, pronouns, and particles.
Lexical Coverage: The system utilizes 139,899 lemma entries derived from the University of Madras Tamil Lexicon, Tamil Wiktionary, and Vuizur Wiktionary. It defines 23 noun classes/subclasses and 24 verb classes/subclasses.
Function: It analyzes words into lemmas and grammatical features (e.g., <POS_NOUN>, <CASE_ACC>, <TENSE_PAST>) and can generate words from these analyses. It handles stem changes (e.g., maram → marattai) and compound verbs.
Validation: The system undergoes rigorous testing including forward/inverse probes and paradigm generation, reporting zero failures in 574,379 regenerated probes.

B. Reversible Semantic Tokenization

A semantic tokenizer maps the FST analysis into a token stream.

Exact Reversibility: The tokenizer ensures decode(encode(x)) = x for any valid UTF-8 input. It records not only lemmas and features but also specific markers for spelling variations, sandhi (sound-linking), and fallback mechanisms.
Fallback Hierarchy: If a word is not recognized by the FST, the system falls back to: (1) reviewed entity analysis, (2) constrained semantic paths, (3) fixed Tamil grapheme units, and finally (4) UTF-8 bytes. This prevents unknown words from collapsing into a generic <unk> token.
Vocabulary: The fixed vocabulary contains ~141k tokens, including lexical tokens, grammatical labels, grapheme fallbacks, and byte tokens.

C. Hierarchical Word Composition

To mitigate the verbosity of the flat morphology stream, the paper introduces a Word-Composer adapter.

Flat Morphology (morphology-flat): Passes every lemma and grammatical factor as a separate token to the model, exposing full structure but increasing sequence length.
Word-Composer:
- Lemma Preservation: Keeps lemma vectors as direct sentence-level states.
- Grammar Summarization: Replaces multiple grammatical factors with a single summary vector ( $z_i$ ) generated via local attention within the word.
- Late Retrieval: After the encoder processes the sentence, the contextualized summary queries the original fine-grained grammatical factors of that specific word. This allows the decoder to access detailed grammar based on sentence context without expanding the global sequence length.
- Fallback Handling: Unanalyzed words follow a separate path where spelling summaries participate in global attention, while original grapheme/byte vectors remain available for decoder cross-attention.

3. Experimental Design

Model: A 18.97-million-parameter encoder-decoder Transformer (width 192, 6 heads, 3 decoder layers).
Data: 69,591 Tamil–English training pairs (mix of authentic BPCC pairs and AI-generated pairs from WikiText-103).
Baselines: The proposed systems are compared against three external tokenizers: Sarvam-1, AI4Bharat IndicBERTv2, and BrahmicTokenizer-131K.
Controls: All systems share the same optimizer, rotary positions (RoPE), numeric-copy policy, and generation settings.
Evaluation: A protected test set of 3,539 rows (IN22-Gen, IN22-Conv, FLORES+ devtest) evaluated using BLEU, chrF++, and COMETKiwi.

4. Key Results

Translation Quality

Morphology-Flat: Achieved the best scores among all systems: 10.63 BLEU, 35.26 chrF++, and 0.6276 COMETKiwi. This represents a relative improvement of 7.2% in BLEU over the strongest external baseline (AI4Bharat).
Word-Composer: Scored 10.30 BLEU, 34.88 chrF++, and 0.6241 COMETKiwi. While slightly lower than the flat morphology approach, it still outperformed all external tokenizers (e.g., +3.8% BLEU over AI4Bharat).
Domain Analysis: The quality gap between the composer and flat morphology was concentrated in longer, formal FLORES+ sentences. On IN22 partitions, the two systems performed as paired ties.

Efficiency and Representation

Sequence Length: The Word-Composer reduced the mean global source states from 71.48 (flat) to 29.08, a 59.3% reduction.
Computational Cost: Estimated FLOPs suggest the composer requires 9–21% fewer operations depending on decoder caching, primarily due to reduced global attention costs.
Preservation: Both morphology-aware systems achieved 100% recall on required numeric slots and outperformed external baselines in entity token preservation.

Architectural Findings

Late Retrieval: Retrieving grammatical details after sentence encoding significantly improved quality. Moving retrieval earlier (between encoder layers) degraded performance.
Summary Design: Attempts to use two grammar summaries (hand-written or learned) did not justify the added sequence length; the single summary with late retrieval proved optimal.
Reversibility: The tokenizer successfully preserved exact text reconstruction, including handling of rare words via grapheme/byte fallbacks without collapsing them to unknown tokens.

5. Significance and Claims

The paper makes modest, specific claims regarding its contributions:

Explicit Morphology in Low-Resource Settings: The results demonstrate that explicit Tamil morphology provides useful, reusable signals that improve translation quality under a constrained budget (small model, limited data). This challenges the assumption that general, data-hungry methods always dominate, particularly in data-limited regimes.
Efficiency vs. Structure Trade-off: The Word-Composer proves that it is possible to retain the benefits of explicit morphological structure while significantly reducing the computational burden of global attention. It achieves a "signal-preserving" compression that outperforms standard statistical tokenizers.
System Properties: Beyond benchmark scores, the work emphasizes system properties such as exact reversibility, auditable grammar, and controlled fallbacks. These features allow for inspection and debugging of model behavior, which is often opaque in standard subword tokenization.
Limitations of Scale: The author explicitly states that they do not claim linguistic tokenization always wins. They acknowledge that a much larger model with massive data might learn these regularities implicitly. The advantage of explicit structure is currently demonstrated within the specific "low-resource, small-model regime" tested.

In conclusion, the paper argues that for Tamil, a combination of deterministic morphological analysis and hierarchical attention design offers a superior balance of translation quality and computational efficiency compared to standard statistical tokenizers, provided the model architecture is designed to handle the structured input effectively.