💬 NLP

Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung

This paper introduces CKTN, the first multilingual corpus and benchmark for Vietnam's Cham, Khmer, and Tay-Nung languages, demonstrating that standard adaptation methods fail due to script and contact differences and proposing a script-aware pretraining recipe that significantly improves encoder performance and semantic generalization.

Original authors: Anh Trac Duc Dinh, Khang Nhat Hoang Vo, Vinh Cong Doan, Tai Tien Ta, Khoa Duc Anh Lam

Published 2026-07-10✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anh Trac Duc Dinh, Khang Nhat Hoang Vo, Vinh Cong Doan, Tai Tien Ta, Khoa Duc Anh Lam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart robot to understand three very different languages spoken by ethnic minorities in Vietnam: Cham, Khmer, and Tay-Nung. These languages are like three distinct musical instruments: Cham is a string instrument often played with a Latin guitar pick (Latin script) but sometimes with a traditional bow (Brahmic script); Khmer is a complex drum kit with no clear breaks between notes (no spaces between words); and Tay-Nung is a wind instrument that looks like a standard flute (Latin script) but has very few sheet music books available.

The problem? The robot's brain (a standard AI model) was trained mostly on English, Chinese, and Vietnamese. When it tries to read these minority languages, it gets confused. It sees a word and breaks it into tiny, meaningless crumbs, like trying to eat a whole pizza one crumb at a time. This is called fragmentation.

The Big Discovery: "Don't Just Count the Crumbs"

The researchers built a massive new library called CKTN, containing 44,367 documents and over 24 million tiny word-pieces (subword tokens). They used this library to test if the robot could learn these languages.

Here is the twist: The robot's usual "report card" metrics were lying to them.

The Trap: The robot could get a perfect score on a "fill-in-the-blank" test (predicting the next word) and a "find the matching sentence" test (retrieval). It looked like a genius.
The Reality: When asked to understand the meaning of a whole article and sort it into a category (like "News" vs. "Music"), the robot failed miserably. It was just memorizing surface-level patterns, like matching the font style, rather than understanding the story.

The paper argues that low error rates on word-prediction and simple matching tasks do not prove the robot actually understands the language. In fact, relying on those tests can make you think the robot is smarter than it really is.

The Solution: A Customized Training Camp

To fix this, the team didn't just feed the robot more data; they gave it a specialized training regimen called CKTN-ELECTRA. Think of it as three steps to rewire the robot's brain:

Expand the Dictionary (Vocabulary Augmentation):
The robot's original dictionary was too small for these languages. The team added new words specifically for Cham, Khmer, and Tay-Nung. Instead of breaking "Cham" into "Ch" + "a" + "m", the robot now sees "Cham" as one whole, solid block. This reduced the "crumb" problem by 25–32%.
The "Script-Safe" Game (Script-Aware Replacement):
They used a game where the robot tries to guess if a word has been swapped out. Usually, a weak robot might swap a Khmer word into a Cham sentence just because it looks different. The new system acts like a strict referee: "No swapping Khmer script into a Latin sentence!" This forces the robot to learn the meaning of the words, not just their visual shape.
The Slow-Start Schedule (Linear Scheduling):
They didn't throw the robot into the deep end immediately. They started the game slowly, letting the robot learn the basics before making it harder. This prevented the robot from panicking and learning bad shortcuts.

The Results: From Confused to Champion

After this special training, the results were dramatic:

The Old Way: A standard model (RemBERT) without these fixes got a 0.1454 accuracy score on sorting articles. It was basically guessing.
The New Way: The CKTN-ELECTRA model scored 0.9214 accuracy. It became the best performer among all the models tested.

However, the paper is careful to note something interesting: The "Find the Matching Sentence" test (Information Retrieval) didn't change much. The robot was already good at that because it just needed to match surface words. The real win was in understanding the meaning (Classification), which the old tests missed.

What the Paper Says "No" To

The authors are very clear about what doesn't work:

No, just adding more data isn't enough. If you don't fix the dictionary and the training rules, the robot will still fail to understand the meaning, even if it gets better at guessing the next word.
No, simple word-matching isn't a good test. If a model scores high on finding similar words but low on understanding the topic, it's not actually "learning" the language.
No, we haven't solved everything. The data they used comes mostly from government news outlets. It's formal and clean, but it doesn't capture the messy, slang-filled way people actually chat on social media. The paper suggests the robot might struggle with informal text.

How Sure Are They?

The team has measured these results with hard numbers. They didn't just guess; they ran the models, split the data into training and testing sets, and calculated exact scores.

They proved that the new method (CKTN-ELECTRA) is significantly better at classification than the old methods.
They suggest (based on their analysis) that the reason the old models failed was because the robot's "brain" was too fragmented and the training game was too easy to cheat.
They caution that while their method works great for news and formal text, we don't know yet how well it handles the chaotic, mixed-up language of the internet.

In short, the paper shows that to teach a robot a rare language, you can't just throw a dictionary at it. You have to fix how it sees the words, teach it to play fair, and test it on things that actually require understanding, not just pattern matching.

Technical Summary: Echoes Across Vietnam's Highlands, Delta, and Coast

Problem Statement

The paper addresses the critical absence of Vietnam's ethnic minority languages—specifically Cham, Khmer, and Tay-Nung—from Natural Language Processing (NLP) research. While often framed as a simple data scarcity issue, the authors argue that the challenge is more complex due to three intersecting factors:

Script Heterogeneity: The languages utilize distinct writing systems (Brahmic-derived abugida, Khmer script without explicit word boundaries, and Latin-based orthography) and often appear in mixed or non-standardized digital forms.
Lexical Fragmentation: Existing multilingual encoders (e.g., mBERT, XLM-R, RemBERT) allocate insufficient vocabulary capacity to these low-resource languages, forcing words to be split into long sequences of continuation subword pieces. This weakens lexical representations and downstream transfer.
Adaptation Failure Modes: Standard adaptation techniques, particularly ELECTRA-style continued pretraining (Replaced Token Detection or RTD), fail in script-heterogeneous settings. A weak, scratch-initialized generator can produce trivial replacements (e.g., inserting a Khmer-script token into a Latin-script sequence) that a discriminator can detect via superficial script cues rather than genuine semantic understanding. This leads to models that minimize loss or excel at lexical-overlap retrieval while failing at semantic generalization.

Methodology

The authors propose CKTN, a new multilingual corpus and benchmark, alongside a script-aware adaptation recipe to address these specific failure modes.

1. The CKTN Corpus and Benchmark

The authors constructed a corpus of 44,367 documents (over 24 million subword tokens) sourced from Vietnamese government and regional news outlets (e.g., Voice of Vietnam, local newspapers). The dataset covers:

Cham: South-central coast (Austronesian).
Khmer: Mekong Delta (Austroasiatic).
Tay-Nung: Northern highlands (Tai-Kadai).

The benchmark supports three tasks:

Continued Pretraining (CPT): 80/20 train/dev split for language modeling.
Category Classification: 28-way classification across 9–10 categories per language, testing semantic generalization across lexically dissimilar documents.
Summary-Document Retrieval: Pairing summaries with source articles to test recovery via shared surface forms.

2. Script-Aware Adaptation Recipe

The proposed method, CKTN-ELECTRA, modifies the standard ELECTRA pretraining pipeline through three key components:

Vocabulary Augmentation: Before pretraining, the source vocabulary is expanded using a candidate selection process on the CKTN corpus. New tokens are calibrated using scores derived from their decomposition in the original tokenizer to prevent over-selection of noisy pieces. Embeddings for new tokens are initialized via averaging the embeddings of their source-token decompositions, anchoring them in the existing multilingual space.
Difficulty-Calibrated Replacement Sampling: To prevent the discriminator from learning shortcuts based on script mismatches, the replacement sampler enforces four constraints:
1. The replacement must not be identical to the original token.
2. It must be well-formed (no control symbols or punctuation-dominant pieces unless the original was).
3. Script Compatibility: Candidates must match the script of the local context (e.g., Khmer tokens are excluded from Latin-script Cham/Tay-Nung sequences).
4. Embedding Similarity: Candidates must fall within a specific embedding similarity band to avoid trivial rejections or uninformative duplicates.
Linear Generator-Discriminator Scheduling: To stabilize training where the generator is trained from scratch and the discriminator is initialized from a pretrained encoder, the RTD loss weight ( $\lambda$ ) is ramped linearly from 0 to a maximum value over the first few epochs. This allows the generator to learn a plausible target-language distribution before the discriminator loss is fully applied.

Key Results

Intrinsic Evaluation

Tokenization Quality: CKTN-ELECTRA significantly reduced fertility (tokens per word) by 25–32% compared to baselines and lowered the continuation-token ratio from as high as 44.5% (RemBERT) to below 11% across all three languages.
Language Modeling Loss: Continued pretraining substantially reduced MLM loss for all baselines, confirming the corpus is sufficient for token-level adaptation. However, the authors note that low MLM loss does not guarantee high-quality document representations.

Extrinsic Evaluation

Category Classification: CKTN-ELECTRA achieved the strongest performance (0.9214 Accuracy / 0.7103 Macro-F1), substantially outperforming XLM-R (0.8169/0.5617), mBERT (0.8024/0.5054), and notably, standard RemBERT (0.1454/0.0053). The collapse of standard RemBERT despite vocabulary augmentation suggests that without the calibrated RTD objective, the model fails to learn transferable semantic structures.
Information Retrieval: In contrast to classification, retrieval metrics (MRR@10, Recall@10) showed minimal improvement for CKTN-ELECTRA and remained high for baselines. The authors attribute this to the task's reliance on lexical overlap between summaries and articles, which can mask poor semantic generalization.

Ablation Studies

Removing either the script-aware filtering or the linear scheduling caused classification performance to collapse to near-random levels (e.g., Macro-F1 dropping to ~0.0012), confirming that both components are necessary to prevent the discriminator from exploiting superficial cues.

Significance and Claims

The paper claims three primary contributions:

Resource Creation: The introduction of CKTN, the first large-scale corpus and benchmark for Cham, Khmer, and Tay-Nung, enabling research into low-resource transfer under script and contact variation.
Identification of Failure Modes: The authors demonstrate that standard multilingual encoders severely fragment these languages and that common adaptation signals (MLM loss and lexical-overlap retrieval) can be misleading, overestimating representation quality in script-heterogeneous settings.
Methodological Advancement: The proposed script-aware adaptation recipe (vocabulary augmentation + calibrated RTD) successfully mitigates the "shortcut learning" problem, yielding an encoder with substantially less fragmentation and superior classification performance.

The authors conclude that while lexical-overlap retrieval is a convenient metric, it is an unreliable proxy for representation quality in low-resource, script-diverse contexts. True adaptation requires methods that force models to learn semantic confusability rather than script-based shortcuts. The work is limited by its reliance on formal news registers (lacking informal/code-switched text) and the relatively small size of the corpus, particularly for Tay-Nung and the Khmer retrieval subset.