Technical Summary: Afri-MCQA

Problem Statement

Despite Africa housing over one-third of the world's languages and a rapidly growing population, the continent remains significantly underrepresented in Artificial Intelligence research, particularly in Multimodal Large Language Models (MLLMs). Existing benchmarks for low-resource languages are predominantly text-only, relying on translations of Western-centric datasets. Furthermore, current multimodal evaluations often lack cultural grounding, focusing on global or Western contexts ("WEIRD" coverage) and ignoring the fact that many African languages are primarily oral. Consequently, there is a critical gap in understanding how well MLLMs can reason about African cultural knowledge when presented with visual, textual, and spoken inputs in native languages.

Methodology

To address these gaps, the authors introduce Afri-MCQA, the first large-scale multilingual cultural Visual Question Answering (VQA) benchmark designed specifically for African languages.

Dataset Construction

• Scope: The dataset covers 15 African languages across 12 countries (e.g., Ghana, Nigeria, Ethiopia, Kenya, South Africa), representing an estimated 392.6 million speakers.
• Data Volume: It consists of approximately 7,500 Q&A pairs.
• Modalities: Each data point includes a culturally relevant image and parallel Q&A pairs in both text and speech modalities. The speech modality includes recordings in the native language and African-accented English.
• Annotation Process: Data was entirely created by native speakers residing in the respective countries. The process involved:
  1. Selection: Annotators were hired via Upwork based on fluency, prior experience, and residence.
  2. Phased Collection: A pilot phase screened annotators, followed by a main annotation phase where each language contributed ~500 image-grounded samples.
  3. Quality Assurance: Language coordinators (co-authors of the paper) reviewed all submissions for linguistic accuracy, cultural appropriateness, and audio quality.
• Content: Questions are categorized into 10 cultural domains (e.g., Vehicles, Food, Traditions, Public Figures) and designed to require reasoning (multi-hop, counting, cultural inference) rather than simple identification.

Experimental Setup

The authors evaluated multiple MLLMs (including open-weight models like Qwen 2.5-Omni, Gemma-3n, and the closed-source Gemini-2.5 Pro) across four research questions:

1. Cultural Understanding: Performance on visually-grounded cultural QA.
2. Modality Impact: Comparing text vs. speech inputs.
3. Language Impact: Comparing native language vs. English queries.
4. Task Format: Comparing Multiple-Choice (MC-VQA) vs. Open-ended VQA.

To distinguish between linguistic incompetence and cultural knowledge gaps, the study included control experiments:

• Text Controls: AfriXNLI (NLI) and AfriMMLU (General Knowledge).
• Audio Controls: Language Identification (LID) and Automatic Speech Recognition (ASR) tasks.

Key Results

1. Poor Performance on African Cultural Contexts

Even the best-performing model, Gemini-2.5 Pro, achieved only 78.68% accuracy on English-based Multiple-Choice VQA and 38% on Open-ended VQA. Open-weight models performed significantly worse, with many scoring near zero accuracy on Open-ended VQA when queried in native languages.

2. The Speech Modality Bottleneck

Performance degraded drastically when switching from text to speech, particularly for open-weight models.

• LID and ASR Failures: Models like Qwen exhibited near-random performance in Language Identification (2–4% accuracy) and extremely high Word Error Rates (WER >100%) for African languages.
• Downstream Impact: These foundational speech processing failures directly caused the near-zero accuracy observed in Audio-based Open-ended VQA. Models could not transcribe the input correctly, let alone reason about the cultural content.

3. Linguistic vs. Cultural Gaps

Control experiments revealed that linguistic understanding gaps (measured by AfriXNLI and AfriMMLU) were substantially larger than cultural knowledge gaps (measured by Afri-MCQA).

• For open-weight models, the performance drop from English to native languages was often 20–40% on general linguistic tasks, compared to a smaller but still significant drop on cultural tasks.
• This suggests that while models lack specific African cultural knowledge, their primary failure mode in native languages is a fundamental inability to understand the language itself, especially in spoken form.

4. Task Format Sensitivity

Models performed significantly better on Multiple-Choice VQA than on Open-ended VQA. The gap was most pronounced in native language speech queries, where open-weight models often failed to generate coherent answers even when they could select the correct option in a multiple-choice setting.

Significance and Claims

The paper claims that Afri-MCQA provides a necessary foundation for evaluating and improving multimodal AI for Africa. The authors emphasize three main takeaways:

1. Speech-First Approaches are Critical: Since many African languages are primarily oral, current models' inability to process speech (LID/ASR) is a critical bottleneck that must be addressed before cultural reasoning can be effectively evaluated.
2. Culturally-Grounded Pretraining is Insufficient: The gap between general knowledge benchmarks (AfriMMLU) and cultural benchmarks (Afri-MCQA) indicates that language data alone is insufficient; models require explicit exposure to African cultural content.
3. Cross-Lingual Transfer Limitations: Models may possess cultural facts in English but fail to access or reason about them when queried in native languages, highlighting a disconnect in cross-lingual cultural transfer.

The authors release Afri-MCQA under an academic license or CC BY-NC 4.0 to support the development of more inclusive, culturally aware multimodal systems. They acknowledge limitations regarding the dataset's size relative to the continent's linguistic diversity and the potential for human bias in curation, positioning the dataset as an evaluation benchmark rather than a pretraining resource.