💬 NLP

What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?

This paper demonstrates that jointly optimizing tokenization with language modeling fundamentally alters token structures, revealing that tokenizer-free approaches like SSLMs learn morphologically aligned and contextually efficient tokens that reduce perplexity and achieve competitive downstream performance across diverse languages, unlike H-Nets which prioritize byte-level efficiency.

Original authors: Saketh Reddy Vemula, Parameswari Krishnamurthy

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saketh Reddy Vemula, Parameswari Krishnamurthy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, computers do not read text the way humans do. They cannot see a sentence as a flowing stream of meaning; they see a sequence of discrete units. To make sense of language, these systems must first break words down into smaller pieces, a process called tokenization. Imagine a librarian who must organize a library not by whole books, but by individual chapters or even paragraphs, depending on how the books are written. If the librarian cuts the books in the wrong places, the story becomes hard to follow. For decades, researchers have used fixed rules to decide where to make these cuts, often relying on simple statistics about how frequently certain letter combinations appear together. These rules work well enough, but they are static; once set, they never change, even if the computer is learning a new language or a complex dialect. This raises a fundamental question: what happens if we stop using these fixed rules and instead let the computer learn how to cut the words while it is learning the language itself?

A team of researchers at IIIT Hyderabad set out to answer this by building language models that learn to tokenize as they learn. They tested these "tokenizer-free" systems across eighteen different languages, ranging from English and Swedish to Tamil, Hindi, and Hebrew. These languages were chosen to represent a wide variety of how words are built, from simple, short words to complex, long chains of sounds that change meaning with every added piece. The researchers compared their new, learning systems against the traditional, fixed methods. They wanted to see what kind of word pieces the computer would invent on its own, whether those pieces made sense linguistically, and if they helped the computer understand the language better.

The results revealed that when a computer is allowed to learn its own way of breaking down words, it develops two very different strategies depending on the architecture used. One approach, known as a subword segmental language model, learned to cut words in a way that closely matched the natural building blocks of the language. In languages like Hindi, where words are often formed by adding specific endings to a root, this system quickly identified the root and the endings, separating them cleanly. In Hebrew, a language where meaning is woven into the internal structure of a word rather than just added to the end, the system learned to recognize those internal patterns. This method produced tokens that were linguistically meaningful, aligning well with how humans understand the grammar and structure of the language.

In contrast, a different approach called H-Nets took a much more pragmatic route. Instead of trying to find the linguistic building blocks, this system focused entirely on efficiency. It learned to create very long chunks of text, sometimes spanning entire phrases, to minimize the number of steps the computer needed to take to process the data. These chunks often had little to do with the actual grammar or meaning of the words. For example, in languages with complex scripts, this system produced tokens that were significantly longer than those created by any other method, prioritizing the speed of processing over the clarity of the word structure. While this made the system computationally efficient, it resulted in a vocabulary that looked nothing like the standard ways humans or traditional algorithms break down words.

The study also showed that the way these systems learn changes depending on the type of language they are processing. In languages where words are built by stringing many small parts together, the learning process was more dynamic and fluctuating. The system would try different ways of cutting the words, sometimes finding the perfect split and then moving away from it, before finally settling on a solution that balanced efficiency with meaning. In simpler languages, the system found a stable way of cutting words very quickly and stuck with it. This suggests that the complexity of the language itself shapes how the computer learns to tokenize it.

When the researchers tested these new tokenization methods in real-world tasks, such as identifying the sentiment of a sentence or finding names of people and places, the results were encouraging. Even though the computer had learned a completely different vocabulary than the one used in standard models, it performed just as well, and in some cases better, on these tasks. The system that learned to align with the natural structure of the language consistently reduced the confusion the computer felt when predicting the next word in a sentence. This indicates that letting the computer learn its own tokenization strategy allows it to discover meaningful patterns that fixed rules might miss.

Ultimately, the research demonstrates that there is no single "best" way to break down words for a computer. The optimal method depends on what the system is trying to achieve. If the goal is to understand the deep structure of a language, letting the model learn its own segmentation leads to tokens that mirror human linguistic intuition. If the goal is raw processing speed, the model might invent entirely different, longer chunks that sacrifice linguistic clarity for efficiency. By showing that these joint learning approaches can produce effective, meaningful vocabularies without human-designed rules, the study opens the door to more adaptable and linguistically aware artificial intelligence systems that can handle the full diversity of human language.

Technical Summary: What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?

Problem Statement

Tokenization is a critical preprocessing step in language modeling (LM) pipelines, yet it is typically treated as a static, fixed component learned independently of the language model itself using statistical priors like Byte-Pair Encoding (BPE), Unigram Language Models (ULM), or WordPiece. This "fixed-tokenizer" approach assumes that frequently co-occurring characters should form tokens, but it remains unclear whether such heuristics yield optimal performance across languages with diverse morphological complexities, typologies, and scripts. Furthermore, the standard pipeline (tokenizer $\to$ LM $\to$ detokenizer) prevents the tokenizer from adapting to the specific linguistic patterns the model learns during training. While "tokenizer-free" architectures have emerged to address this, the specific nature of the tokens they learn and their comparative effectiveness against fixed approaches remain under-analyzed.

Methodology

The authors conduct a comprehensive analysis comparing tokenizer-free approaches against fixed-tokenizer methods across 18 typologically and script-diverse languages (including Agglutinative, Fusional, Analytic, and Introflexive types).

1. Approaches Evaluated

Tokenizer-Free (Joint Optimization):
- SSLM (Subword Segmental Language Model): Uses LSTMs or Transformers to marginalize over all possible segmentations, jointly optimizing for language modeling and segmentation.
- H-Nets (Hierarchical Networks): An end-to-end architecture performing data-dependent boundary prediction alongside byte-level language modeling.
Fixed-Tokenizers (Baseline & Variants):
- Standard algorithms: BPE, ULM, WordPiece.
- Modified/Novel algorithms: SaGe, Morfessor, BPE-dropout, PathPiece, MorphBPE, MYTE, PickyBPE, SuperBPE, BoundlessBPE.

2. Experimental Setup

Data: Non-parallel monolingual corpora (WMT News Crawl and NLLB) scaled by Byte-Premiums (BP) to ensure comparable information density across languages.
Pretraining: SSLMs and H-Nets were trained on 250,000 sentences. H-Nets utilized $\sim$ 3M parameters; SSLMs utilized $\sim$ 2M parameters.
Downstream Evaluation: To ensure fair comparison, the authors used tokenizer-free approaches as pretokenization to generate a corpus, upon which standard BERT models (12M parameters) were trained with fixed tokenizers (e.g., SSLM-BPE, SSLM-ULM). These were fine-tuned on Sentiment Analysis, POS Tagging, Named Entity Recognition (NER), and Dependency Parsing for English, Hindi, and Telugu.

3. Evaluation Metrics

Intrinsic: Morphological alignment (MorphScore), contextual exponence (number of distinct neighbors), effective vocabulary size, fertility, and mean token length.
Extrinsic: Validation perplexity during pretraining and downstream task performance (Accuracy, F1, LAS).

Key Contributions and Results

1. Learning Dynamics and Morphological Alignment

The study reveals that joint optimization fundamentally alters token structure, but the behavior varies by language typology:

SSLMs recover tokens that are morphologically aligned and contextually efficient. They exhibit four distinct learning phases: rapid initial evolution, fluctuation (pronounced in agglutinative languages), convergence, and saturation.
H-Nets prioritize byte-level computational efficiency, often producing longer tokens with very low overlap with standard subword vocabularies. They trade morphological alignment for sequence efficiency, resulting in low MorphScore F1 (<0.1) across all languages.
Typological Differences: Agglutinative languages (e.g., Tamil, Hungarian) show more dynamic segmentation patterns and fluctuations during learning compared to analytic or fusional languages. For instance, SSLMs in Tamil initially achieve perfect alignment but later converge to less accurate splits, suggesting a trade-off where the model prioritizes contextual utility over strict morphological boundaries.

2. Intrinsic Properties of Learned Tokens

Contextual Exponence: Tokenizer-free approaches (SSLM and H-Nets) consistently yield lower contextual exponence (fewer distinct neighbors) than fixed tokenizers. This suggests they learn units with more predictable grammatical roles, improving semantic quality.
Vocabulary Size & Length:
- Latin-script languages show larger effective vocabulary sizes (14k–15k tokens) with lower contextual exponence.
- Non-Latin scripts (e.g., Tamil, Hindi) show smaller vocabularies but higher contextual exponence, indicating a preference for flexible, fundamental morphemes.
- H-Nets produce exceptionally long tokens (17–18.5 characters) in non-Latin scripts, likely due to boundary prediction favoring long phrases to minimize byte-level loss.
Vocabulary Overlap: H-Nets exhibit near-zero Jaccard overlap with fixed tokenizers, indicating they learn fundamentally different token sets. SSLMs show low but higher overlap with frequency-based methods (BPE, WordPiece), particularly in agglutinative languages where joint optimization learns smaller atomic morphemes compared to the larger, arbitrary units of fixed tokenizers.

3. Downstream Performance

Perplexity: Models pretrained with SSLM-based pretokenization consistently achieve the lowest validation perplexity across English, Hindi, and Telugu, converging earlier than fixed-tokenizer baselines.
Task Performance: Despite having distinct vocabularies, tokenizer-free approaches (via pretokenization) achieve competitive downstream performance.
- SSLM-ULM achieved the highest F1 for POS tagging in Telugu (90.17%).
- SSLM-WPC achieved the highest F1 for NER in Hindi (91.41%).
- While Morfessor-based pretokenization remained the strongest overall, SSLM-based approaches consistently outperformed standard fixed tokenizers and many modified variants (e.g., SaGe, PickyBPE).

Significance and Claims

The paper claims that jointly optimized tokenizers learn linguistically meaningful token structures that are effective for downstream NLP, even when their vocabularies differ significantly from standard approaches.

Fundamental Shift: Joint optimization moves away from strict statistical priors toward contextual and computational efficiency. SSLMs optimize for morphological alignment and context, while H-Nets optimize for byte-level efficiency.
Effectiveness: The findings demonstrate that tokenizer-free approaches do not require a fixed vocabulary to be effective; when used as a pretokenization strategy, they enable language models to achieve lower perplexity and competitive task performance.
Language Specificity: The optimal tokenization strategy is highly dependent on language typology. Agglutinative languages benefit from dynamic segmentation patterns that fixed tokenizers often fail to capture, while analytic languages converge to stable, mid-range alignments.

The authors conclude that while morphology-aware pretokenization (like Morfessor) remains the strongest overall, SSLM-based pretokenization offers a robust, competitive alternative that fundamentally improves language modeling by learning tokens that are better aligned with the linguistic and contextual needs of the model.