OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report

Technical Summary: OpenLID-v3: Improving the Precision of Closely Related Language Identification

Problem Statement
Language identification (LID) is a critical preprocessing step for constructing high-quality multilingual datasets from web data, particularly for Large Language Model (LLM) pre-training. However, existing tools like OpenLID and GlotLID struggle with two primary issues: distinguishing between closely related languages and differentiating valid natural language from noise (e.g., code, broken encoding, or machine-generated content). These failures lead to the contamination of language-specific subsets, a problem exacerbated for low-resource languages. Furthermore, standard evaluation benchmarks often rely on clean, parallel corpora (e.g., FLORES+, UDHR) that fail to capture the complexity and ambiguity of real-world web documents, masking poor performance on similar language varieties.

Methodology
The authors present OpenLID-v3, an extension of the fully open-source OpenLID system (based on the fastText architecture). The development of v3 was driven by a manual inspection of OpenLID-v2's performance on the HPLT 3.0 dataset, which revealed specific failure modes. The methodology involved three core technical interventions:

1. Training Data Expansion and Cleaning: The authors extended training data for specific languages, notably re-introducing Latin and adding Serbian written in the Latin script (which was previously missing, causing misclassification as Bosnian or Croatian). They also incorporated subsets of GlotLID's training data reported as non-noisy and added recent Wikipedia dumps.
2. Class Inventory Restructuring:
   • Merging Variants: To reduce confusion, eight Arabic dialects were merged into a single ara_Arab macrolanguage, and two Persian varieties were merged into fas_Arab. Similarly, mutually intelligible Bambara and Dyula were merged.
   • Noise Labeling: A dedicated zxx_Zxxx ("not-a-language") class was introduced using GlotLID's und_* and zxx_* data to explicitly model non-linguistic noise, addressing the "trash bin phenomenon" where non-language text was forced into existing language classes.
3. Ensemble Approaches: The authors experimented with ensembling OpenLID-v3 and GlotLID using top-1 and top-3 agreement strategies to improve precision, though top-3 agreement was found to degrade results for single-label models.

Evaluation Strategy
The paper argues that mainstream benchmarks are insufficient for evaluating LID on similar languages. Consequently, the evaluation strategy included:

• Standard Benchmarks: FLORES+, UDHR, and FastSpell.
• Custom/Specific Benchmarks: New evaluation datasets were created or utilized for three specific groups of closely related languages:
  • BCMS (Bosnian, Croatian, Montenegrin, Serbian): Using Twitter user data, ParlaSent (parliamentary transcripts), and re-annotated HPLT-LID data.
  • Romance Varieties: Focusing on Ligurian, Occitan, and Francoprovençal using the ITDI dataset.
  • Scandinavian Languages: Evaluating Norwegian Bokmål, Nynorsk, Danish, and Swedish using SLIDE and Nordic DSL datasets.
• Metrics: Emphasis was placed on False Positive Rate (FPR) and precision, alongside recall and F1-score, to better reflect the class imbalance found in real web crawls.

Key Results

• Performance on Benchmarks: OpenLID-v3 performs on par with or slightly better than OpenLID-v2 and GlotLID on standard benchmarks (FLORES+, UDHR). The introduction of a softmax threshold (0.5) generally improved results for OpenLID-v3.
• Closely Related Languages:
  • BCMS: OpenLID-v3 significantly improved the identification of Serbian in Latin script, a major failure point in v2. However, confusion between Bosnian and Serbian remains high due to lexical overlap and shared roots. Ensemble methods (OpenLID-v3 + GlotLID) improved precision but drastically reduced coverage (recall) for low-resource variants.
  • Romance Languages: On the ITDI dataset, ensembling OpenLID-v3 and GlotLID achieved the highest precision and lowest FPR, though at the cost of removing many Venetian samples.
  • Scandinavian: GlotLID achieved the best recall, while OpenLID-v3 showed better precision and FPR. The two Norwegian varieties (Bokmål and Nynorsk) remained the most easily confused pair.
• Error Analysis: Manual analysis of BCMS errors revealed that named entities, lexical overlap (shared roots outweighing orthographic markers like jat), historic forms, and ungrammatical syntax are primary sources of misclassification. Short sequences and web artifacts were identified as particularly difficult for all models.

Significance and Claims
The paper claims that while large-scale multilingual benchmarks often show high aggregate scores, they obscure significant deficiencies in identifying closely related languages. OpenLID-v3 demonstrates that:

1. Precision vs. Coverage Trade-off: Ensemble approaches can significantly improve precision (reducing contamination of language subsets) but substantially reduce coverage for low-resource languages, a critical consideration for dataset curation.
2. Need for Specialized Evaluation: Reliable evaluation of LID models, especially for similar languages, requires benchmarks specific to those language groups rather than general multilingual datasets.
3. Open-Source Viability: By prioritizing permissive licenses, OpenLID-v3 offers a robust alternative to proprietary or restricted-license models, capable of handling noise and specific script variations (like Latin-script Serbian) when properly trained.

The authors conclude that for general Scandinavian identification, OpenLID-v3 is recommended if precision is prioritized, while GlotLID is preferred for recall. For distinguishing between highly similar varieties (e.g., Bokmål vs. Nynorsk), models trained specifically for that separation are still necessary. The work highlights the ongoing need for multilabel training data and benchmarks to handle the ambiguity inherent in web documents.