Technical Summary: CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Problem Statement

Language Identification (LID) is a critical prerequisite for curating multilingual corpora, yet current models perform poorly on noisy, heterogeneous web data, particularly for low-resource and under-served languages. While some literature suggests LID is a "solved" problem, this paper argues that existing evaluations are misleading. Most state-of-the-art (SOTA) models are trained on "clean" data (e.g., religious texts, government documents, Wikipedia) and evaluated on similarly clean benchmarks (e.g., FLORES, UDHR). Consequently, these models fail to generalize to the web domain, where text is often unstructured, contains code-switching, and lacks the formal registers of training data. Furthermore, many popular LID systems lack open training data or fail to support the "long tail" of languages, limiting the development of Large Language Models (LLMs) for languages other than English (LOTE).

Methodology

Dataset Creation (CommonLID)

The authors introduce CommonLID, a community-driven, human-annotated benchmark specifically designed for the web domain.

• Data Sourcing: Text was sampled from Common Crawl (CC-MAIN-2024-22 and CC-MAIN-2025-05) and MADLAD-400. To manage the scale of web data, the authors used three existing LID models (fastText, OpenLID, GlotLID) to pre-select documents, acknowledging this as a necessary compromise to avoid the impracticality of random sampling for under-represented languages.
• Annotation Process: Over 80 native speaker annotators, recruited from the NLP community and grassroots organizations, participated via virtual hackathons. They used a modified Dynabench interface to perform line-level language identification.
• Incentives: To ensure ethical collaboration and high-quality engagement, contributors who annotated at least 100 documents were offered co-authorship on the paper.
• Data Processing: The raw annotations underwent rigorous cleaning, including:
  • Splitting text by newlines to avoid complex sentence splitting.
  • Filtering lines with fewer than 10 characters.
  • Removing English boilerplate (using langid.py as a heuristic) from non-English lines, with specific exemptions for languages like Scottish Gaelic and Nigerian Pidgin to avoid false positives.
  • Manual reconciliation of ambiguous cases (e.g., macro vs. micro-language distinctions, particularly for Arabic).
• Dataset Statistics: The final dataset contains 373,230 lines across 109 language varieties. It is over ten times larger than the recent "Smol" dataset, with 78 languages containing at least 100 lines.

Evaluation Framework

The authors evaluated eight popular LID models (AfroLID, CLD2, fasttext, FUN-LangID, pyFranc, CLD3, GlotLID v4, OpenLID-v2) and four OpenAI LLMs (GPT-4o, GPT-4o-mini, GPT-5, GPT-5-mini) across multiple datasets:

1. CommonLID (Web domain, human-annotated).
2. SmolSent, Bibles, Social Media (Noisy/Colloquial).
3. FLORES+, UDHR-LID (Clean, formal domains).

Metrics: The study utilizes macro-averaged F1 scores and False Positive Rates (FPR). To address the challenge of comparing models with disparate language coverage, the authors report scores in two ways:

• "all": Calculated over the entire dataset (uncovered languages score zero).
• "cov.": Calculated only over the subset of languages the model supports.
• Core Set Analysis: A direct comparison was performed on a "Core" set of 76 languages supported by all models to ensure fair comparison.

Key Results

Performance on CommonLID

CommonLID proves to be a significantly more challenging benchmark than existing datasets.

• Low Scores: Most models achieved F1 scores in the 60s (averaged over supported languages) on CommonLID, a stark contrast to the 80s or 90s often seen on clean datasets like FLORES or UDHR.
• Domain Gap: High scores on datasets like UDHR or Bible corpora are attributed to training data overlap (e.g., GlotLID is heavily trained on Bible data; Franc on UDHR). These high scores do not translate to web data performance.
• Model Comparison:
  • GlotLID demonstrated the strongest overall performance and highest language coverage, achieving the highest F1 on the "Core" set (91.6%) and "High-coverage" sets. However, it suffers from significantly slower inference times compared to lighter models like CLD2.
  • CLD2 and GlotLID sit on the Pareto frontier of the compute-performance trade-off.
  • LLMs: Despite their general NLP capabilities, OpenAI's GPT models (even GPT-5) were outperformed by GlotLID in zero-shot LID tasks, particularly on low-resource and African languages (e.g., GPT-5 was ~30% lower in F1 on African languages compared to GlotLID).

Inter-Annotator Agreement

Due to the volunteer nature of the project, only 12.9% of lines were double-annotated. Among these, 2.3% showed genuine disagreement after filtering for boilerplate errors. Disagreements were primarily driven by:

• The continuum nature of Arabic (dialects vs. macro-language).
• Macro/micro language distinctions (e.g., Chinese vs. Mandarin).
• English boilerplate misidentified as other languages.

Key Contributions

1. CommonLID Benchmark: A large-scale, human-annotated LID dataset covering 109 languages specifically derived from the noisy web domain, addressing the scarcity of open, representative evaluation data for under-served languages.
2. Community-Centric Annotation: A novel recruitment and incentive model that treats annotators as co-authors, fostering ethical collaboration and higher data quality than typical crowd-sourcing.
3. Comprehensive Evaluation: A rigorous benchmarking of eight SOTA LID models and four LLMs, revealing that current models are over-optimistic when evaluated on clean data and struggle significantly with web data.
4. Open Release: The dataset, code, and evaluation scripts are released under an open, permissive license to facilitate reproducible research.

Significance and Claims

The paper claims that LID is far from solved, particularly for the web domain and low-resource languages. The authors argue that existing evaluations systematically overestimate LID accuracy because they rely on clean, domain-specific datasets that do not reflect the heterogeneity of web text.

By introducing CommonLID, the authors provide a more realistic benchmark that highlights the limitations of current models. They assert that improving LID is essential for reducing cross-lingual inequity in NLP, as poor LID performance on web data prevents the creation of high-quality training corpora for languages other than English. The work underscores the need for better, domain-appropriate evaluation datasets and models that can generalize beyond the "clean" domains of religious texts and government documents.