Cross-Dataset Generalization in Urdu Fake News Detection: An Empirical Study with XLM-RoBERTa and a Length Confound Analysis

Problem Statement

Despite Urdu being spoken by over 231 million people, Fake News Detection (FND) in the language remains under-resourced compared to English. While prior research has established strong in-domain performance on individual Urdu datasets, there is a critical gap in understanding whether models trained on one corpus can generalize to another. Real-world deployment requires models to handle heterogeneous sources with varying styles and lengths, yet no systematic cross-dataset generalization study has been conducted for Urdu FND. This paper addresses this gap by investigating the transferability of models between two major, publicly available Urdu datasets: Ax-to-Grind and Notri-Fact.

Methodology

The study employs a rigorous empirical design involving two large-scale, balanced, native Urdu datasets:

1. Dataset A (Ax-to-Grind): 10,083 articles across 15 domains.
2. Dataset B (Notri-Fact): 13,388 articles covering Pakistani news topics.

Experimental Setup:

• Models: The primary model is XLM-RoBERTa-base, fine-tuned for binary sequence classification. Classical baselines using TF-IDF vectorization with Logistic Regression (LR) and Support Vector Machines (SVM) are also evaluated.
• Conditions: Five experimental conditions are tested:
  1. In-domain training and testing on Dataset A.
  2. In-domain training and testing on Dataset B.
  3. Zero-shot cross-domain transfer: A → B (trained on Ax-to-Grind, tested on Notri-Fact).
  4. Zero-shot cross-domain transfer: B → A (trained on Notri-Fact, tested on Ax-to-Grind).
  5. Length Ablation: Training and testing on Dataset A with all articles truncated to a maximum of 50 words to isolate the effect of document length.
• Evaluation: The primary metric is the macro-averaged F1-score, supplemented by accuracy, precision, and recall.

Key Findings and Results

The experiments reveal a striking asymmetry in cross-dataset generalization:

1. In-Domain Performance: XLM-RoBERTa achieves strong performance within its training distribution (F1 = 0.929 on Dataset A; F1 = 0.954 on Dataset B). Classical TF-IDF baselines perform competitively but consistently lower than the transformer.
2. Asymmetric Collapse (A → B): When the model trained on Ax-to-Grind is tested on Notri-Fact, performance collapses catastrophically to an F1 of 0.005. The model predicts "fake" for 99.7% of the test articles, effectively ignoring the actual balanced distribution of the test set.
3. Partial Success (B → A): Conversely, the model trained on Notri-Fact achieves a meaningful F1 of 0.771 when tested on Ax-to-Grind, retaining significant discriminative ability despite the domain shift.
4. The Length Confound: The paper identifies the root cause of the A → B failure as a systematic length confound in the Ax-to-Grind dataset:
   • In Ax-to-Grind, fake articles average 117 words, while real articles average only 35 words (a 3.4× asymmetry).
   • In Notri-Fact, both classes average 160–170 words, showing no length-based separation.
   • The model learns to exploit the "longer = fake" shortcut during training on Dataset A. When applied to Dataset B, where all articles are long regardless of veracity, the heuristic fails completely, causing the model to default to predicting "fake" for everything.
5. Ablation Evidence: Truncating Dataset A articles to 50 words (removing the length signal) resulted in only a marginal F1 drop (0.929 to 0.922). This confirms that while the length confound inflates in-domain performance, the model does learn some semantic content; however, the absence of this shortcut in Dataset B is what causes the transfer failure.

Key Contributions

• First Cross-Dataset Study: This is the first systematic investigation of cross-dataset generalization for Urdu FND, revealing that high in-domain accuracy does not guarantee robustness.
• Identification of a Length Confound: The paper provides empirical evidence of a severe length bias in the Ax-to-Grind dataset, demonstrating how XLM-RoBERTa (and even classical models) exploits this as a shortcut rather than learning semantic veracity.
• Diagnostic Methodology: The authors propose a reusable diagnostic framework for low-resource NLP settings, combining:
  • Bidirectional transfer asymmetry inspection.
  • Analysis of predicted label distributions (detecting class collapse).
  • Class-conditional length distribution analysis.
• Baseline Contextualization: The study establishes that classical models (TF-IDF + LR/SVM) also partially exploit the length confound, though less severely than the transformer.

Significance and Claims

The paper argues that the current standard for evaluating Urdu FND—relying solely on single-dataset, in-domain splits—is insufficient and potentially misleading. The findings suggest that:

• In-domain metrics can be inflated by dataset artifacts (confounds) rather than genuine model capability.
• Cross-dataset evaluation is mandatory to detect shortcut learning and ensure models are learning semantic features rather than spurious correlations like document length.
• Future dataset construction must include per-class length statistics as mandatory metadata and employ length-stratified sampling to prevent such confounds.

The authors conclude that without addressing these dataset-level artifacts, models trained on current benchmarks may fail catastrophically in real-world scenarios where news sources vary in length and style. The proposed diagnostic methodology offers a practical tool for the community to identify similar confounds in other low-resource multilingual settings.