Multilingual and Cross-Lingual Citation Needed Detection on Wikipedia for Lower-Resource Languages

Technical Summary: Multilingual and Cross-Lingual Citation Needed Detection on Wikipedia for Lower-Resource Languages

Problem Definition
Automated fact-checking (AFC) relies on claim check-worthiness detection (CWD) to identify statements requiring verification. On Wikipedia, this manifests as Citation Needed Detection (CND), a binary classification task where editors flag claims lacking authoritative inline citations. While existing research has addressed CND in high-resource languages, lower-resource language communities face a disproportionate burden due to fewer active editors and a lack of automated tools. Furthermore, recent AFC pipelines increasingly rely on Large Language Models (LLMs), which are often inaccessible to low-resource organizations due to proprietary restrictions and high inference costs. This paper addresses the gap in multilingual CND resources and the feasibility of using smaller, cost-effective models for this task.

Methodology
The authors introduce MCN (Multilingual Citation Needed), a large-scale corpus derived from high-quality "Featured Articles" on Wikipedia. The dataset spans 18 linguistically diverse languages across three resource levels (high, medium, and low), covering Indo-European, Turkic, Austronesian, and Austroasiatic families. Claims are labeled based on the presence or absence of inline citations, augmented with contextual sentences and topic metadata.

The study evaluates Small Language Models (SLMs) (specifically Llama-3-8B, Qwen3-8B, and Aya-8b) as alternatives to LLMs. The methodology focuses on three experimental axes:

1. Supervised Fine-Tuning (SFT) Variants: The authors compare three adaptation strategies for decoder-based SLMs using QLoRA:
   • Full-Token Loss (FTL): Standard autoregressive training over all tokens.
   • Target-Only Loss (TOL): Masking input tokens to focus learning on the assistant (label) tokens.
   • Encoder-Style (ES): Adapting the decoder model as a discriminative classifier by feeding the last-token hidden state into a feed-forward classifier, optimizing with cross-entropy loss.
2. Multilingual Benchmarking: SLMs are benchmarked against encoder-based Pre-trained Language Models (PLMs like mBERT and XLM-R) and commercial LLMs (GPT-5, Gemini 2.5) using zero-shot and few-shot prompting.
3. Cross-Lingual Transfer (X-CND): The authors evaluate the transferability of models trained exclusively on English data to medium- and low-resource target languages under zero-shot and few-shot settings. This includes testing on both synthetic parallel data (translated English test sets) and non-parallel real-world data.

Key Contributions

• MCN Corpus: The release of a large-scale CND dataset covering 18 languages, significantly expanding the linguistic and resource-level coverage of prior datasets (e.g., CitationNeeded, CheckThat).
• SLM vs. LLM Benchmark: A comprehensive evaluation demonstrating that fine-tuned SLMs substantially outperform prompted LLMs across all resource levels.
• First Cross-Lingual CND Study: The first systematic evaluation of cross-lingual CND, showing that English-trained SLMs can effectively transfer to lower-resource languages.
• SFT Strategy Analysis: Identification of Encoder-Style (ES) fine-tuning as the most effective strategy for adapting decoder-based SLMs to the discriminative CND task.

Results

• SFT Variants: Encoder-style (ES) fine-tuning consistently outperforms generative variants (FTL and TOL) across most languages and models, yielding accuracy gains of up to 8 percentage points over generative methods.
• Model Performance: Fine-tuned SLMs and PLMs significantly outperform prompted LLMs. For instance, fine-tuned SLMs achieve 66–70% accuracy on low-resource languages, whereas the best-performing LLM (Gemini 2.5) achieves only ~59%.
• Cross-Lingual Transfer:
  • On parallel data, fine-tuned SLMs exhibit strong zero-shot generalization (up to 87% accuracy).
  • On non-parallel real-world data, accuracy drops by approximately 15–20 percentage points, highlighting the challenge of distribution shifts.
  • Crucially, ES-fine-tuned SLMs trained solely on English data outperform prompted LLMs on target languages, even with zero or minimal target-language supervision.
• Few-Shot Adaptation: While few-shot tuning improves performance, it requires careful hyperparameter tuning (specifically lower learning rates) to be effective for QLoRA-fine-tuned SLMs. mBERT benefits most from few-shot tuning, but ES-tuned SLMs still outperform LLMs with minimal adaptation.
• Generalizability: Models trained on Featured Articles generalize well to randomly sampled Wikipedia articles, with performance remaining comparable after correcting for annotation biases in non-featured content.

Significance and Implications
The paper argues that compact, task-specific models are preferable to general-purpose LLMs for CND, particularly for lower-resource Wikipedia communities. The findings suggest that:

1. Accessibility: Organizations with limited resources can deploy effective CND tools using small, open-source models fine-tuned on high-resource languages, bypassing the need for expensive LLM APIs or large target-language datasets.
2. Efficiency: Encoder-style fine-tuning provides a reliable, cost-effective pathway to adapt decoder-based models for discriminative tasks.
3. Deployment Viability: Cross-lingual transfer of English-trained specialized models offers a viable strategy for supporting low-resource communities, provided that the limitations of real-world distribution shifts are acknowledged.

The authors note limitations, including a concentration on Indo-European languages, reliance on QLoRA-based fine-tuning, and the domain-specific nature of Wikipedia's citation policies, which may not directly transfer to other fact-checking domains like political debates.