AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages

1. Problem Statement

The paper addresses the critical challenge of information poverty and the proliferation of misinformation in low-resource African languages. Current fact-checking systems and Large Language Models (LLMs) are predominantly engineered for English, leading to significant performance gaps when applied to African contexts.

• Specific Challenges:
  • Lack of Resources: There is a scarcity of datasets covering the full pipeline of fact-checking (Information Retrieval, Evidence Extraction, and Claim Verification) for African languages.
  • Cultural Nuance: Misinformation often targets culturally specific topics (health, local news, traditions) where generic models fail due to a lack of cultural grounding.
  • Cross-lingual Limitations: Existing embedding models struggle to retrieve relevant documents when the query and the document are in different languages (cross-lingual retrieval) or even within the same language but across diverse domains.
  • Verification Reliability: LLMs often hallucinate or rely on internal knowledge rather than provided evidence when verifying claims in African languages.

2. Methodology: The AfrIFact Dataset

The authors introduce AfrIFact, a comprehensive benchmark dataset designed to evaluate and improve fact-checking capabilities across 10 African languages (Amharic, Hausa, Igbo, Oromo, Shona, Swahili, Twi, Wolof, Yorùbá, and isiZulu) and English.

• Dataset Composition:

  • Scale: Contains over 18,000 human-generated claims.
  • Domains: Covers two high-impact domains:
    1. Health: Focused on common diseases, outbreaks, and public health (derived from WHO documents).
    2. Culture & News: Focused on cultural knowledge, political leaders, history, and regional events (derived from Wikipedia and XL-Sum news articles).
  • Structure: Each data point includes a claim, a source document, extracted evidence spans (sentence-level), and a veracity label (SUPPORTED, REFUTED, or NOT_ENOUGH_INFORMATION).

• Data Construction Pipeline:

  • Health Domain: Leveraged the AfriDocMT corpus (parallel English-African translations of WHO articles). Claims and evidence were generated in English using GPT-4 and human annotators, then manually translated and aligned into target languages.
  • Culture-News Domain: Curated natively from Wikipedia and news sources. Annotators generated claims and evidence directly in the target languages to ensure cultural authenticity.
  • Quality Control: Used SSA-COMET for translation quality verification and employed a multi-stage annotation process involving native speakers, language coordinators, and meta-annotators to ensure high inter-annotator agreement (Kappa scores > 0.74).

3. Experimental Setup & Methodology

The study evaluates three distinct research questions (RQs) using the AfrIFact dataset:

• RQ1: Information Retrieval (IR): Can embedding models retrieve relevant documents from large, multilingual corpora?

  • Setup: Evaluated on both monolingual and multilingual corpora (mixing Health and Culture-News).
  • Metrics: nDCG@10 and Recall@100.
  • Models: Multilingual embedding models (e.g., multilingual-e5-large, AfriE5-Large-instruct, Qwen3-Embedding variants).

• RQ2: Evidence Extraction (EE): Can models identify specific sentences within a document that support or refute a claim?

  • Setup: Retrieval of top-k sentences from a single document.
  • Metrics: nDCG@3 and Recall@3.

• RQ3: Fact-Checking (FC): Can LLMs correctly classify claims based on evidence?

  • Setup: Zero-shot, Few-shot (3-shot), and Fine-tuning (LoRA, QLoRA, Full) scenarios.
  • Models: Various LLMs including Gemma, Qwen, Command R, AfriqueLLM, and GPT-5.2.
  • Conditions: Evaluated with and without provided evidence spans.

4. Key Results

A. Information Retrieval & Evidence Extraction

• Cross-Lingual Gap: Even the best embedding models show a massive performance drop (50–70%) when retrieving relevant documents in a different language than the query.
• Domain Sensitivity: Models perform significantly better on Culture-News domains than on Health domains. This is attributed to the scarcity of high-quality, structured health data in African languages compared to the abundance of Wikipedia/cultural content.
• Model Size vs. Language Coverage: Increasing model parameter size (e.g., from 0.6B to 8B) yielded diminishing returns. Models with better language coverage (like AfriE5-Large-instruct) outperformed larger models with poor African language support.
• Noise Sensitivity: Embedding models are highly sensitive to corpus size; adding non-relevant documents from other languages or domains significantly degrades retrieval performance.

B. Fact-Checking Capabilities

• Zero-Shot Failure: Most open-source LLMs perform near-randomly (~33% accuracy) in zero-shot settings for African languages, indicating a lack of inherent fact-verification capability.
• Impact of Few-Shot Prompting: Providing 3 balanced examples significantly boosts performance. For instance, AfriqueQwen-14B improved by 43% (from 34% to 77% accuracy) with few-shot prompting.
• Evidence Utilization:
  • In zero-shot settings, models often ignore provided evidence and rely on internal knowledge (sometimes leading to hallucinations).
  • Fine-tuning is crucial: Parameter-efficient fine-tuning (specifically QLoRA 8-bit) enabled models to effectively learn to use provided evidence.
  • Performance Gains: QLoRA fine-tuning improved fact-checking accuracy by up to 26% in the Health domain.
• Domain Disparity: Models struggle more with Health claims than Culture-News claims, likely due to the complexity and lack of training data in the medical domain.

5. Key Contributions

1. AfrIFact Dataset: The first large-scale, multilingual benchmark for fact-checking in African languages, covering the full pipeline (IR, EE, FC) across Health and Culture domains.
2. Comprehensive Evaluation: A systematic analysis revealing that current SOTA embedding models and LLMs are ill-equipped for African languages, particularly in cross-lingual retrieval and zero-shot verification.
3. Methodological Insights:
   • Demonstrated that language coverage is more critical than parameter size for retrieval tasks.
   • Showed that few-shot prompting and parameter-efficient fine-tuning are essential to bridge the gap for African language fact-checking.
   • Highlighted the specific challenges of the Health domain compared to cultural domains.

6. Significance and Future Work

• Combating Misinformation: The dataset provides a necessary tool for developing systems to detect and debunk misinformation in regions where it poses severe risks to public health and social stability.
• Research Direction: The findings call for the development of culturally grounded, language-specific embedding models and the expansion of high-quality training data for low-resource languages.
• Limitations: The study notes that the training set is smaller than the test set (due to resource constraints) and that the current evaluation treats IR, EE, and FC as separate steps. Future work should focus on end-to-end pipelines and expanding coverage to additional domains and languages.

In conclusion, AfrIFact establishes a new standard for evaluating fact-checking in African languages, exposing significant gaps in current AI capabilities and providing a roadmap for improving information reliability in low-resource, culturally diverse contexts.