💬 NLP

QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages

The paper introduces QuechuaTok, a benchmark demonstrating that for agglutinative low-resource languages like Southern Quechua, morphological boundary accuracy is a critical evaluation metric that reveals the inadequacy of standard fertility rates, as evidenced by a morphology-aware PRPE tokenizer significantly outperforming standard subword methods in morphological correctness despite higher fertility.

Original authors: Maria Contreras

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maria Contreras

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to speak Quechua, a language spoken by millions in South America. Quechua is what linguists call an agglutinative language. Think of it like a set of Lego bricks that snap together to build a single, long word.

In English, if you want to say "from our walking," you use three separate words. In Quechua, you snap those ideas together into one giant block: purisqanchikmanta. This single block contains the root (walk), the tense (past), the people (us), and the direction (from).

The Problem: The "Brick Cutter"

To teach computers, we usually break words down into smaller pieces called tokens. The standard tools for doing this (like BPE, Unigram, and WordPiece) were mostly designed for European languages like English or Spanish, where words are shorter and don't snap together as tightly.

The paper argues that these standard tools are like clumsy brick cutters. They chop the Quechua Lego blocks in random places just because those cuts look common in the data, ignoring the fact that they are cutting right through the middle of a grammatical meaning.

The Experiment: QuechuaTok

The researchers, led by Maria Contreras, built a test called QuechuaTok to see which "brick cutter" works best. They tested four different strategies on a massive collection of 200,000 Quechua sentences:

BPE, Unigram, and WordPiece: These are the standard, statistical methods. They learn by looking at how often certain letter patterns appear, without really "knowing" the grammar.
PRPE: This is a special, "grammar-aware" method. Instead of just guessing based on frequency, it uses a hand-crafted list of Quechua grammatical rules (like a map of where the Lego bricks actually snap) to cut the words.

The Metrics: How Did They Measure Success?

The researchers used three ways to grade the cutters:

Fertility Rate (The "Compactness" Score): This measures how many pieces a word gets chopped into. A lower number is usually seen as "better" because it means the computer has to process fewer pieces.
- The Trap: The paper argues this is a deceptive score. A tool can get a great score just by memorizing the whole long word as one single piece, rather than actually understanding how to break it down correctly.
OOV Rate (The "Lost Words" Score): How often the computer sees a word it doesn't know. (In this study, all tools did well here).
Morphological Boundary Accuracy (The "Grammar Cut" Score): This is the paper's big new idea. They compared the cuts made by the computers against a "Gold Standard" map created by a human linguist's tool (SQUOIA). Did the computer cut the word exactly where the grammatical meaning changes?

The Results: The Surprise Winner

Here is what happened when they ran the tests:

The "Memorizer" (BPE): The standard BPE tool got the best Fertility Rate (1.636). It looked like the winner because it chopped words into the fewest pieces.
- The Catch: It achieved this by memorizing whole, long words as single chunks. It didn't understand the grammar. Its Grammar Cut Score was terrible: only 6.67%. It was cutting the Lego bricks in the wrong places 93% of the time.
The "Grammar Expert" (PRPE): The PRPE tool had a slightly higher Fertility Rate (1.797), meaning it produced a few more pieces.
- The Win: But its Grammar Cut Score was 83.33%. It knew exactly where the grammatical meanings started and stopped.

The Analogy:
Imagine you are trying to sort a pile of mixed-up sentences.

BPE is like a student who memorizes the whole sentence as one long string of letters. They are very fast (low fertility) but if you ask them to explain the grammar, they fail.
PRPE is like a student who understands the rules of grammar. They might take a tiny bit longer to sort the words because they are carefully separating the subject from the verb, but they actually understand what the sentence means.

The Conclusion

The paper concludes that for languages like Quechua, being "compact" (low fertility) is not the same as being "correct."

If you only look at how few pieces a tokenizer creates, you might pick the worst tool for the job. The researchers argue that we need a new standard: Morphological Boundary Accuracy. We need to measure not just how small the pieces are, but whether those pieces respect the actual grammatical structure of the language.

In short: Don't just count the pieces; check if the cuts make sense.

Technical Summary: QuechuaTok

Problem Statement

Tokenization is a critical preprocessing step in Natural Language Processing (NLP), yet standard evaluation metrics, particularly fertility rate (the average number of tokens per word), fail to capture morphological correctness for agglutinative languages. In languages like Southern Quechua (quz), a single surface word form encodes complex grammatical information (verb roots, aspect, person, number, case, evidentiality) through strict suffixation. Standard subword tokenizers (e.g., BPE, Unigram), designed primarily for European languages and trained on frequency patterns, often fragment these forms arbitrarily or memorize long surface forms as atomic units. This behavior ignores morphological boundaries that carry grammatical meaning, leading to a disconnect between token efficiency and linguistic structure. Despite Quechua being the most widely spoken indigenous language family in the Americas, there is a lack of systematic evaluation of tokenization strategies for this low-resource, agglutinative language.

Methodology

The authors present QuechuaTok, a systematic benchmark evaluating four tokenization strategies on a corpus of 200,193 sentences of Southern Quechua. The corpus was derived from the Llamacha/monolingual-quechua-iic dataset (Wikipedia and OSCAR sources) and the somosnlp-hackathon-2022 parallel corpus, preprocessed with Unicode normalization and noise removal.

Tokenizers Evaluated

BPE (Byte-Pair Encoding): Trained at vocabulary sizes of 4k, 8k, and 16k.
Unigram LM: Trained at vocabulary sizes of 4k, 8k, and 16k.
WordPiece: Trained at 4k.
PRPE (Prefix-Root-Postfix Encoding): A morphology-aware tokenizer using a hand-crafted suffix lexicon of 23 Quechua morphemes covering TAM, person agreement, nominal case, evidentials, and derivational suffixes.

All statistical models utilized SentencePiece with byte_fallback=True and preserved the apostrophe for ejective consonants.

Evaluation Metrics

The study introduces a new primary metric alongside standard ones:

Fertility Rate: Average tokens per word (lower is better).
OOV Rate: Percentage of tokens mapped to <unk> (0% for all models due to byte fallback).
Morphological Boundary Accuracy (MorphAcc): The proportion of tokenizer boundaries that match the "silver standard" boundaries. The silver standard was generated using the SQUOIA finite-state morphological analyzer (Rios, 2016) and manually verified.
$\text{MorphAcc} = \frac{| \text{pred\_bounds} \cap \text{silver\_bounds} |}{| \text{silver\_bounds} |}$

Downstream Evaluation

To assess the impact on language modeling, the authors trained a smoothed bigram language model on the tokenized text and measured perplexity (PPL) on held-out sentences.

Key Results

Quantitative Findings

BPE: Achieved the lowest fertility rate (1.636 at 16k vocab) but maintained a consistently low MorphAcc of 6.67% across all vocabulary sizes. This indicates BPE reduces token count by memorizing frequent surface forms as atomic units rather than learning morphological structure.
Unigram LM: Showed a non-monotonic relationship between vocabulary size and MorphAcc. It achieved 66.67% MorphAcc at 4k vocab but dropped to 26.67% at 8k and 33.33% at 16k. This suggests that as vocabulary size increases, Unigram LM begins to overfit to surface frequency, collapsing multiple morphemes into single tokens.
PRPE: Achieved the highest MorphAcc of 83.33% with a competitive fertility rate of 1.797. Its performance is independent of corpus statistics, relying instead on the quality of the morphological suffix lexicon.
WordPiece: Evaluated at 4k, it showed an OOV rate of 0.05% and did not report MorphAcc in the final table, though qualitative analysis showed similar issues to other statistical methods.

Perplexity Analysis

Unigram 4k achieved the lowest perplexity (1344), likely due to the small vocabulary forcing segmentation that coincidentally aligned with morphemes.
PRPE achieved a perplexity of 1879, higher than Unigram 4k but significantly better than BPE 16k (2553).
BPE 16k had the highest perplexity (2553) despite the lowest fertility, confirming that low fertility in this context reflects surface-form memorization rather than effective linguistic structure learning.

Qualitative Observations

Qualitative analysis of words like rimankichikmi ("you all speak [witnessed]") and wasiykikunapiqa ("in your houses") revealed that statistical tokenizers (BPE/Unigram) frequently collapsed multiple morphemes into single tokens (e.g., riman|kichikmi), whereas PRPE correctly segmented the prefix, root, and suffixes (e.g., wasi|yki|kuna|pi|qa).

Key Contributions

Systematic Benchmark: The first comprehensive evaluation of BPE, Unigram, WordPiece, and PRPE tokenizers on a large Southern Quechua corpus.
New Metric: The introduction and validation of Morphological Boundary Accuracy (MorphAcc) against the SQUOIA finite-state analyzer, establishing it as a necessary metric for agglutinative languages.
Empirical Evidence: Demonstration that fertility rate is an unreliable proxy for morphological quality. The paper shows that a tokenizer can achieve high efficiency (low fertility) while failing to capture grammatical structure (low MorphAcc).

Significance and Claims

The paper argues that for agglutinative, low-resource languages, optimizing for token count (fertility) is insufficient and potentially detrimental. The authors claim that morphological boundary accuracy is a necessary complementary metric to ensure tokenizers respect the linguistic structure of the language.

The study highlights that while statistical tokenizers (BPE, Unigram) can achieve low perplexity with small vocabularies or low fertility with large vocabularies, they often do so by memorizing surface forms rather than learning morphological rules. In contrast, PRPE demonstrates that explicit encoding of morphological structure yields superior boundary accuracy, even if it does not always minimize the token count or perplexity to the absolute lowest degree. The authors conclude that no single tokenizer wins all metrics simultaneously, necessitating a multi-faceted evaluation approach that prioritizes morphological correctness for languages like Quechua.

Future work identified by the authors includes expanding the MorphAcc silver standard to over 500 words and integrating the PRPE tokenizer into a Quechua-specific language model (QuechuaBERT).