Multilingual Reference Need Assessment System for Wikipedia

1. Problem Statement

Wikipedia relies on the policy of verifiability, requiring claims to be backed by reliable sources. However, the sheer volume of edits (approx. six per second) makes manual identification of uncited claims difficult, leading to over 550,000 articles on English Wikipedia alone containing "Citation needed" tags.

• Gap: Existing automated solutions are primarily limited to the English language edition. There is a lack of production-level, multilingual systems capable of assessing reference needs across diverse language families and scripts.
• Constraints: Any solution must operate under strict infrastructure constraints:
  • Latency: Inference must be under 500ms to handle real-time edit volumes.
  • Hardware: The production environment relies solely on CPUs, not GPUs, due to open-source GPU adoption challenges.
  • Scalability: The system must support the top 10 Wikipedia language editions by active user count.

2. Methodology

A. Definition and Data Collection

The authors define Reference Need (RN) as the proportion of uncited claims that actually require a citation.

• Ground Truth: They utilize Featured Articles (manually curated, high-quality articles) as the gold standard.
  • Positive Class: Sentences in featured articles that have citations.
  • Negative Class: Sentences in featured articles without citations (assumed to be self-evident or common knowledge).
• Languages:
  • Training: English, French, German, Spanish, and Russian (languages with sufficient Featured Article data).
  • Evaluation: The top 10 languages by active users (including the training set plus Japanese, Portuguese, Chinese, Italian, and Persian).
• Data Processing:
  • Extracted from the Wikimedia Data Lake (May 2024 snapshot).
  • Parsed from wikitext to plaintext while preserving reference tag locations (<ref>) for labeling, then removing them for model input.
  • Features: Target sentence, preceding sentence, subsequent sentence, section title, language code, and paragraph context.
  • Filtering: Removed duplicates, sentences <6 words, and non-content sections (e.g., "References").

B. Model Architecture and Training

• Base Models: Fine-tuned Masked Language Models (MLMs) rather than Large Language Models (LLMs) initially, due to resource constraints.
  • Candidates: bert-base-multilingual-cased, distilbert-base-multilingual-cased, and xlm-roberta-base.
• Input Strategy: Experiments determined that the combination of Language Code + Section Title + Sentence Triplet (Target, Previous, Next) yielded the best performance (0.111 AUC improvement over single-sentence input).
• Training Configuration:
  • Model: distilbert-base-multilingual-cased.
  • Sequence Length: 128 tokens.
  • Hyperparameters: Learning rate 1e-5, batch size 16, 3 epochs.
  • Hardware: AMD Radeon Pro WX 9100 GPU (16GB).

C. System Deployment

• Platform: Deployed via KServe on Kubernetes as an asynchronous microservice.
• Resources: Configured with 4 CPUs and 6GB RAM.
• Optimization: Applied post-training optimizations including Quantization (PyTorch, Intel NC, ONNX) and Pruning to reduce latency while maintaining accuracy.

D. Comparative Analysis

The study compares:

1. Small Language Models (SLMs): Fine-tuned DistilBERT variants.
2. Large Language Models (LLMs): Zero-shot inference using models like Llama-3-70B and Mistral-7B via an external API (Together.AI).
3. Baseline: The existing "Citation-Needed" model (English-only).

3. Key Results

Performance Metrics (AUC-ROC & Latency)

• Best SLM: The fine-tuned distilbert-128 model achieved an AUC-ROC of 0.765 with a mean latency of 0.014s (ONNX optimized) or 0.045s (PyTorch).
• LLM Performance: Zero-shot LLMs (e.g., Llama-3-70B) achieved significantly lower AUC-ROC scores (ranging from 0.557 to 0.619) and much higher latencies (0.11s to 5.2s).
• Baseline Comparison: The proposed system outperformed the English-only "Citation-Needed" baseline (AUC 0.650) even on the English subset.

Multilingual Generalization

• The model, trained on 5 languages, successfully generalized to 10 languages.
• High Performance: Portuguese, Persian, and Italian showed performance comparable to training languages.
• Lower Performance: Chinese and Japanese showed slightly lower metrics, likely due to script differences and fewer training examples in the cross-lingual transfer.

Optimization Impact

• Optimization techniques (Quantization, ONNX) reduced inference latency by >2x compared to the baseline model without significant loss in accuracy.
• Decision: The team selected the PyTorch quantized model for production to avoid the engineering overhead of integrating ONNX runtimes, despite ONNX being slightly faster.

4. Key Contributions

1. Multilingual System: Introduction of the first open-source, production-ready system for assessing reference needs across 10 diverse Wikipedia language editions.
2. SLM vs. LLM Trade-off Analysis: A rigorous empirical study demonstrating that fine-tuned Small Language Models (SLMs) offer a superior balance of accuracy and latency for this specific task compared to zero-shot Large Language Models, especially under CPU-only constraints.
3. Production Insights: Detailed documentation of the trade-offs between model accuracy, computational efficiency, and infrastructure constraints (CPU vs. GPU) in a real-world deployment.
4. Open Release: Release of training/inference code and datasets to facilitate further research into information credibility on the Web.

5. Significance and Limitations

• Significance: This work bridges the gap between academic NLP research and industrial deployment. It proves that resource-constrained environments can effectively utilize optimized SLMs to maintain content quality at scale, supporting the Wikimedia Foundation's AI strategy.
• Limitations:
  • Bias: The model relies on DistilBERT, which may inherit inherent biases.
  • Data Scarcity: Performance may degrade in low-resource languages not represented in the top 10.
  • Cultural Imposition: The cross-lingual transfer relies heavily on Western citation styles, which may not align with policies in all language communities.
  • Scalability: Processing time scales linearly with article length; future work may explore chunk-based strategies.

Conclusion

The paper successfully demonstrates that a carefully optimized, fine-tuned DistilBERT model is the most viable solution for multilingual reference need assessment in a production environment. It achieves high accuracy with sub-50ms latency on CPUs, outperforming both previous baselines and zero-shot LLMs, thereby enabling scalable support for Wikipedia editors in maintaining verifiability.