What Language is This? Ask Your Tokenizer

1. Problem Statement

Language Identification (LID) is a critical component of multilingual NLP pipelines, used for corpus curation, training data analysis, and cross-lingual evaluation. While existing systems perform near-perfectly on high-resource languages, they suffer from significant limitations:

• Low-Resource Performance: They fail to generalize when trained on very few labeled samples.
• Fine-Grained Discrimination: They struggle to distinguish between closely related languages or dialects (e.g., Bosnian vs. Croatian, or regional Arabic dialects) due to high statistical overlap.
• Brittleness: They are sensitive to orthographic noise (e.g., missing diacritics) and domain shifts (e.g., social media text vs. formal documents).
• Rigidity: Most current approaches enforce a fixed tokenization scheme across all languages, ignoring the fact that optimal segmentation is often language-specific.

2. Methodology: UniLID

The authors propose UniLID, a simple, data-efficient, and computationally efficient LID method based on the UnigramLM tokenization algorithm.

Core Concept

Instead of treating tokenization as a fixed preprocessing step, UniLID treats segmentation as a language-specific latent variable. It learns a separate unigram distribution for each language over a shared vocabulary.

Technical Formulation

1. Generative Framework:
   • The model assumes a string $s$ is generated by a sequence of latent subword tokens $v$ drawn independently from a language-specific unigram distribution $\phi_\ell$.
   • Unlike standard UnigramLM which learns one global distribution, UniLID learns distinct distributions $\phi_\ell$ for each language $\ell \in \Lambda$, while sharing a common vocabulary $V$.
2. Training (Parameter Estimation):
   • For each language, the model estimates parameters $\phi_\ell$ using the Expectation-Maximization (EM) algorithm on a language-specific corpus.
   • E-Step: Computes expected token counts by marginalizing over all valid segmentations of the input strings (using the forward-backward algorithm).
   • M-Step: Updates the unigram probabilities based on these expected counts.
   • This process allows the model to learn language-specific "optimal" segmentations without retraining the vocabulary itself.
3. Inference:
   • Given an input string $s$, the model calculates the likelihood $p(s|\ell)$ for each language by finding the most probable segmentation under that language's specific unigram model:
     $$\hat{\ell}(s) = \arg\max_{\ell \in \Lambda} p_\ell(s) \approx \arg\max_{\ell \in \Lambda} p_{\phi_\ell}(\tau_{\phi_\ell}(s))$$
   • The final language label is assigned using Bayes' decision rule (assuming a uniform prior).
   • Complexity: Inference is efficient, running in $O(|\Lambda| \cdot N \cdot T_{max})$ time, where $N$ is string length and $T_{max}$ is max token length. The lattice construction is done once, and probabilities are computed for each language.

3. Key Contributions

• Novel Formulation: The first LID method to explicitly model segmentation as a language-dependent phenomenon rather than a fixed global constraint.
• Data Efficiency: The method achieves strong performance with extremely limited data (as few as 5 labeled samples per language).
• Incremental Learning: New languages can be added by training only their specific unigram distribution on the shared vocabulary, without retraining the entire model or the tokenizer.
• Integration: The method can be naturally integrated into existing Large Language Model (LLM) tokenization pipelines.

4. Experimental Results

The authors evaluated UniLID against strong baselines: fastText, GlotLID-M, and CLD3 across several benchmarks (GlotLID-C, UDHR, FLORES-200, DSL-ML 2024, WiLI, Tatoeba).

• Standard Benchmarks: UniLID achieves competitive macro-averaged F1 scores with baselines but maintains a significantly lower False Positive Rate (FPR), which is crucial for preventing data contamination in large-scale crawls.
• Low-Resource Regime:
  • With only 5 samples per language, UniLID achieves >70% accuracy, whereas fastText fails to generalize.
  • It surpasses 90% accuracy with fewer than 50 samples.
• Fine-Grained Dialects:
  • On the DSL-ML 2024 benchmark (distinguishing Slavic and Romance dialects), UniLID improved the macro F1 from 0.53 (fastText) to 0.72.
  • It successfully distinguished dialects where fastText achieved 0.00 F1 on several Slavic languages.
• Robustness:
  • Domain Shift: UniLID doubled the F1 score of fastText when trained on Wikipedia (WiLI) and tested on noisy social media text (Tatoeba).
  • Input Length: It outperforms baselines on short text inputs (<200 characters), a known weakness for many LID systems.
• Vocabulary Flexibility: The method works effectively with both custom-trained tokenizers and pre-trained LLM vocabularies (e.g., Mistral-Nemo).

5. Significance and Impact

• Inclusivity: By drastically improving performance in low-resource and fine-grained settings, UniLID enables the inclusion of underrepresented languages and dialects in large-scale multilingual datasets.
• Efficiency: It offers a computationally lightweight alternative to heavy neural classifiers (like Transformers) while outperforming traditional statistical methods (like n-grams) in difficult regimes.
• Future-Proofing: The modular design allows for the easy addition of new languages, making it highly scalable for the expanding landscape of multilingual NLP.
• Ethical Considerations: The authors note that while better LID aids inclusion, it carries dual-use risks (e.g., content filtering or profiling), urging responsible application.

In conclusion, UniLID demonstrates that treating tokenization as a language-specific generative process, rather than a fixed preprocessing step, yields a more robust, data-efficient, and accurate system for language identification.