Technical Summary: Auditing Data Quality for Low-resource and Multilingual NLP

Problem Statement

While Wikipedia is widely regarded as a fundamental, high-quality resource for Natural Language Processing (NLP), particularly for pretraining multilingual language models, this assumption of inherent quality is increasingly challenged in low-resource and multilingual contexts. Recent studies suggest that non-English Wikipedia editions often contain unnatural, machine-translated, or bot-generated content that can be incomprehensible to native speakers. Relying on such noisy data can lead to poor model performance and untrustworthy technologies. However, manually auditing the quality of over 340 active language editions (containing an estimated 65 million articles) is infeasible. There is a lack of systematic, data-driven methods to quantify Wikipedia quality across these diverse editions and to understand the downstream impact of filtering out low-quality content.

Methodology

The authors propose a data-driven auditing framework that applies data filtering procedures—typically reserved for noisy, uncurated web-text—to the entirety of non-English Wikipedia. The core hypothesis is that if a Wikipedia edition is robust to these cleaning steps, it is of higher quality; conversely, significant data removal indicates the presence of low-quality content.

The methodology involves three primary stages:

1. Data Filtering Processes:

   • Script Filtering: Using regular expressions to remove characters not belonging to the officially documented script(s) for a specific language. This targets foreign-script contamination, often resulting from machine translation artifacts or improperly formatted links.
   • Exact-Match Deduplication: Removing articles with identical text but different IDs or titles. This targets placeholder articles, templates, and filler text.
   • MinHash Deduplication: Using Locality Sensitive Hashing (LSH) to identify and remove near-duplicate articles (Jaccard similarity > 0.85). This targets bot-generated content and repetitive articles that vary slightly in sentences or hyperlinks.

2. Quality Taxonomy Construction:

   • The authors apply these filters to all non-English Wikipedias.
   • They perform k-means clustering on the percentage of remaining documents and characters to categorize editions into four quality tiers (Tier 1 to Tier 4), ranging from least removed (highest quality) to most removed (lowest quality).

3. Validation and Downstream Evaluation:

   • Correlation Analysis: The proposed quality tiers are compared against alternative metrics, including the DEPTH+ metric (an enhanced editing depth measure), the proportion of bot-generated articles, and standard document-level heuristics (e.g., word length, unigram entropy).
   • Downstream Experiments: The impact of filtering is evaluated in three scenarios using models trained on raw Wikipedia (raw_wiki) versus filtered Wikipedia (+filters):
     • Monolingual Pretraining: Training mini DeBERTa models on 50 languages and evaluating on SIB200 (topic classification).
     • Language Adaptation: Adapting XLM-R to new languages and fine-tuning on SIB200 and BELEBELE (cross-lingual reading comprehension).
     • Multilingual Pretraining: Pretraining base-size multilingual models on the top 100 Wikipedias and evaluating on BELEBELE and XNLI.

Key Contributions

1. Empirical Evidence of Quality Issues: The study provides empirical data showing that approximately 11.68% of total characters and 29.36% of total documents are removed from non-English Wikipedias when applying standard cleaning filters. This reveals systematic issues such as script contamination, repeated templates, and high concentrations of bot-generated content.
2. Quality Taxonomy: The authors establish a four-tier quality ranking for non-English Wikipedias. This taxonomy shows strong correspondence with DEPTH+, bot-article ratios, and heuristic measures, validating the filtering approach as a proxy for quality.
3. Downstream Impact Analysis: The paper demonstrates that models trained on filtered data generally match or outperform those trained on raw data. Notably, the largest performance gains are observed for lower-quality language editions (Tiers 3 and 4), suggesting that removing noise is particularly beneficial for low-resource scenarios.
4. Open-Source Toolkit: The authors release an open-source NLP toolkit (texieve) for cleaning text data across many languages.

Results

• Filtering Efficiency: The filtering process removed a substantial portion of data, with MinHash deduplication accounting for the majority of removals (28.33% of articles). Languages like Cebuano, Swedish, and Vietnamese saw significant data reduction (35–80% of documents removed), largely due to bot-generated content.
• Tier Correlation: The quality tiers correlate negatively with bot-article ratios (ρ = −0.56) and positively with DEPTH+ (ρ = 0.31). Heuristic distributions (e.g., word length, entropy) also align monotonically with the tiers, with Tier 1 exhibiting the most "high-quality" characteristics.
• Model Performance:
  • In monolingual pretraining, most languages showed negligible performance differences between raw_wiki and +filters, confirming that removed data was low quality. However, Cebuano saw a 12% improvement with filtering, while Acehnese suffered a drop, likely due to its already small dataset size being disproportionately reduced.
  • In language adaptation, models adapted with +filters data generally performed better, particularly in zero-shot settings on BELEBELE.
  • In multilingual pretraining, results were mixed but generally showed that filtered data did not harm performance and often provided slight gains for lower-tier languages.

Significance and Claims

The paper claims that the assumption of Wikipedia's inherent "high quality" is not universally valid, particularly for non-English and low-resource editions. The study serves as a first step in establishing quality-aware best practices for utilizing Wikipedia in NLP.

The authors emphasize that their automated, model-agnostic approach demonstrates that simple preprocessing can effectively discern between good and bad quality data. They argue that understanding and accounting for these quality variations is essential for developing more equitable and effective language technologies, especially for low-resource languages where the margin for error is small. The work lays the groundwork for future dataset creation and curation efforts, suggesting that future research should engage with Wikipedia editing communities to better understand the sources of quality variation.

The paper remains modest in its scope, noting limitations such as the restriction to encoder-only models (due to existing decoder models likely being pre-trained on Wikipedia) and the reliance on benchmarks not derived from Wikipedia to avoid confounding factors. It does not propose new model architectures but rather advocates for rigorous data auditing as a prerequisite for effective NLP development.