MultiCW: A Large-Scale Balanced Benchmark Dataset for Training Robust Check-Worthiness Detection Models

1. Problem Statement

The verification of information by media professionals (fact-checkers, journalists) is hindered by the sheer volume of textual data and its linguistic diversity. While Large Language Models (LLMs) are reshaping information verification, automated Check-Worthiness (CW) detection—identifying which claims require fact-checking—remains limited by existing resources.

Current datasets suffer from three critical flaws:

• Language Bias: Most are restricted to English.
• Domain/Style Narrowness: They often focus on specific topics (e.g., COVID-19) or specific formats (e.g., headlines or Twitter posts), lacking generalizability to diverse writing styles (noisy social media vs. structured news).
• Class Imbalance: Existing datasets often have skewed distributions (e.g., 75% non-check-worthy), making it difficult to train robust models that generalize across languages and domains.

2. Methodology

A. Dataset Construction: MultiCW

The authors introduce MultiCW, a large-scale, balanced benchmark designed to address the gaps in existing resources.

• Scale & Scope: Contains 123,722 samples for in-distribution training/testing and 27,761 samples for out-of-distribution (OOD) evaluation.
• Coverage: Spans 16 languages, 7 topical domains (Health, Politics, Environment, Science, Sport, Entertainment, History), and 2 writing styles:
  • Noisy: Informal, slang-heavy, social media style.
  • Structured: Formal, well-organized, news/encyclopedic style.
• Data Sources: Aggregated and harmonized from CLEF-2022/2023, MultiClaim, Ru22Fact, and LESA.
• Balancing Strategy:
  • Class Balancing: Enforced an equal number of check-worthy (CW) and non-check-worthy (non-CW) samples per language.
  • Style Balancing: Separated data into structured and noisy subsets, ensuring balance within each.
  • Augmentation: Used machine translation (DeepTranslate) and Wikipedia sampling (using GLiNER for entity extraction) to fill gaps in low-resource languages and underrepresented classes.
• Definition of CW: A binary definition based on four criteria: Significance (public policy/health impact), Controversy (disputed), Impact (influences opinion/behavior), and Source Reliability (public figures/institutions).

B. Experimental Setup

The authors established baselines by evaluating two categories of models:

1. Fine-Tuned Transformers: Three multilingual models were fine-tuned on the MultiCW training set (70% of data) for 5 epochs:
   • XLM-R (base): Pre-trained on 100 languages.
   • mDeBERTa-v3 (base): Optimized with disentangled attention.
   • LESA: A specialized architecture combining linguistic encapsulation and semantic amalgamation.
2. Zero-Shot LLMs: 15 commercial and open-source LLMs (e.g., GPT-4o, Claude 3.5, Llama 3.1/3.2, Nemotron-4) were evaluated without fine-tuning.
   • Prompting: Tested 6 prompt variants, including Chain-of-Thought (CoT) and Guided Answer (GA) strategies, to determine the best reasoning scaffold.

C. Evaluation Protocol

Models were evaluated on the test set (15% of data) and a separate OOD set (4 additional languages) using:

• Metrics: Accuracy, Macro-precision, and Macro-recall.
• Subsets: Overall, Noisy, and Structured.

3. Key Contributions

1. MultiCW Dataset: The first large-scale, balanced multilingual benchmark for CW detection, covering 16 languages and explicitly balancing for writing styles and class distribution.
2. Out-of-Distribution Benchmark: A dedicated OOD evaluation set (27k samples) to rigorously test model generalization to unseen languages and domains.
3. Comprehensive Baselines: Extensive benchmarking of fine-tuned Transformers against a diverse array of 15 LLMs under zero-shot conditions, providing a clear performance ceiling for current technology.
4. Analysis of Prompting: Systematic comparison of prompting strategies (CoT vs. GA) to understand how reasoning scaffolds affect LLM performance in fact-checking tasks.

4. Results

A. Fine-Tuned vs. Zero-Shot Performance

• Fine-Tuned Transformers: Consistently outperformed zero-shot LLMs.
  • XLM-R and mDeBERTa achieved ~92% accuracy overall on the in-distribution test set.
  • LESA achieved 79% accuracy, showing strong performance on structured text but struggling significantly with noisy text (dropping to 70%).
• Zero-Shot LLMs: The best-performing LLM (Nemotron-4 340B with CoT prompting) reached 79% accuracy, followed by Claude 3.5 Haiku (75%).
  • Most LLMs hovered between 65–75% accuracy.
  • Smaller models (e.g., Llama 3.2 1B/3B) performed poorly (~50%).
  • Key Finding: Task-specific fine-tuning remains crucial; even smaller fine-tuned models outperform massive zero-shot LLMs.

B. Robustness and Generalization

• Writing Styles: All models performed better on structured text (>90% for top models) than noisy text (~87%). Noisy text (slang, errors) remains the primary challenge.
• Out-of-Distribution (OOD): Fine-tuned models maintained strong performance on unseen languages (dropping only 5–9 points), confirming the robustness of multilingual pretraining.
  • XLM-R: 81.3% OOD accuracy.
  • mDeBERTa: 86.6% OOD accuracy.
• Language Variance: Performance varied by language. Western European languages (German, French, Portuguese) saw the highest accuracy (96–98%), while Arabic and Bulgarian showed lower performance (80–85% for fine-tuned models).

C. Prompting Impact

• Chain-of-Thought (CoT): Generally improved performance for larger models (e.g., Claude 3.5 Haiku jumped from 59% to 75% with CoT). However, effects were inconsistent; some models (e.g., GPT-4.1) performed worse with CoT.
• Guided Answer (GA): Simple binary prompts were often less effective than structured reasoning steps for complex claims.

5. Significance and Future Directions

• Advancing Fact-Checking: MultiCW provides the necessary infrastructure to move beyond English-centric, single-domain fact-checking tools, enabling the development of systems that can handle the heterogeneous nature of real-world misinformation.
• Model Selection: The study demonstrates that while LLMs are powerful, fine-tuning on domain-specific, balanced data is currently superior for high-precision check-worthiness detection.
• Future Work: The authors suggest exploring hybrid systems (combining fine-tuned precision with LLM adaptability), investigating few-shot prompting, and addressing remaining limitations such as code-switching, dialectal variations, and multimodal inputs (text + images).

In conclusion, MultiCW establishes a new standard for evaluating check-worthiness detection, proving that while LLMs are closing the gap, specialized fine-tuned models remain the state-of-the-art for robust, multilingual fact-checking support.