Evaluating Health Misinformation in Low-Resource Languages: Integrating Small Language Models with a Culturally-Sensitive Responsible NLP Framework (Bangla as a Case Study)

Technical Summary: Evaluating Health Misinformation in Low-Resource Languages

Problem Statement
The proliferation of generative AI and Large Language Models (LLMs) has accelerated the creation and dissemination of health misinformation, posing significant threats to global digital environments. While AI serves as both a combatant against and a vector for misinformation, current detection systems exhibit critical limitations in non-English and low-resource language contexts. Specifically, Culturally and Linguistically Diverse (CALD) communities face heightened risks due to structural barriers, a lack of linguistically appropriate health resources, and the inability of existing models to capture cultural nuances and local traditions.

A primary technical bottleneck is the scarcity of reliable, expert-annotated datasets for low-resource languages (e.g., Bangla, Arabic, Vietnamese). Furthermore, LLMs often underperform in specialized domains like healthcare due to insufficient domain-specific knowledge and the prohibitive costs of resource-intensive fine-tuning. Consequently, there is a need for cost-effective, culturally sensitive detection mechanisms that can operate effectively in low-resource settings without relying solely on Western-centric data or metrics.

Methodology
This research employs a multi-stage methodology to evaluate Small Language Models (SLMs) for health misinformation detection in Bangla, followed by the design of a culturally sensitive evaluation framework.

1. Dataset Construction and Translation:

   • The study utilized the MHMisinfo: Video-based Mental Health Misinformation dataset (originally English), comprising 739 videos and associated comments.
   • After cleaning (filtering out short transcriptions), 286 videos were retained.
   • A hybrid human-in-the-loop translation process was implemented using Azure translation services, followed by rigorous validation by two native Bangla-speaking experts (one a medical domain expert) to ensure linguistic fidelity, cultural appropriateness, and medical accuracy.
   • Human experts annotated the translated dataset to extract health-related claims and assign ground-truth labels regarding their accuracy (misinformation vs. factual).

2. Experimental Evaluation of SLMs:

   • The study evaluated a suite of Small Language Models (SLMs) on the Bangla-translated dataset: Phi-4 (14B), Qwen3 variants (4B, 8B, 14B), Llama-3.1-8B-Instruct, Gemma-3-12B-IT, and Ministral-8B-2512.
   • The evaluation pipeline involved using SLMs for claim extraction and misinformation classification via prompt engineering.
   • Performance was measured against human-annotated ground truth using micro-averaged and macro-averaged metrics (Precision, Recall, F1-score) to assess both overall performance and consistency across individual videos.

3. Development of a Responsible NLP Framework:

   • Recognizing that standard technical metrics (Precision/Recall) fail to capture the nuances of health misinformation in low-resource languages, the authors proposed a multi-dimensional framework grounded in Responsible Natural Language Processing (RNLP).
   • The framework evaluates misinformation across six dimensions:
     1. Content Accuracy (CA): Scientific grounding and factual consistency.
     2. Misinformation Detection (M): Misleading framing, omission of key details, and emotional appeals.
     3. Harm and Risk Assessment (HR): Potential for unsafe behaviors and severity of health impact.
     4. Cultural and Linguistic Sensitivity (CLS): Appropriateness of beliefs, practices, and linguistic fidelity.
     5. Communication Quality (CQ): Clarity, balance of perspectives, and visual integrity.
     6. Performance Accuracy (PA): Model reliability in claim extraction and cross-lingual consistency.
   • To synthesize these dimensions, the authors applied the Entropy-TOPSIS method (Technique for Order Preference by Similarity to Ideal Solution). This approach uses information entropy to derive objective weights for each dimension based on their discriminative power, followed by TOPSIS to rank videos against ideal (accurate) and anti-ideal (misleading) solutions.
   • A "cascading penalty rule" was integrated: if a model fails to extract data (PA=0), the framework automatically flags the content for expert review, acting as a socio-technical safeguard against automation failure.

Key Contributions

1. First Expert-Validated Bangla Dataset: Creation of the first expert-validated Bangla health misinformation dataset by translating and annotating the MHMisinfo dataset, reviewed by native-speaking medical and linguistic experts.
2. SLM Benchmarking: Comprehensive evaluation of various SLMs on Bangla health misinformation detection, comparing their performance against human ground truth and analyzing cross-lingual performance degradation.
3. Responsible NLP Framework: Design and testing of a novel, multi-dimensional framework that moves beyond technical metrics to incorporate cultural sensitivity, harm potential, and communication quality, specifically tailored for non-English contexts.
4. Prototype Application: Development of a proof-of-concept mobile application, MALAK (Multilingual AI Agent for Learning, Action, and Knowledge), which operationalizes the framework to analyze health claims against trusted medical sources and provide risk classifications.

Results

• Model Performance: Among the evaluated models, Phi-4 demonstrated the superior performance, achieving the best balance between precision and recall with a micro-F1 score of 63.77% on the Bangla dataset. The Qwen3 models (8B and 4B) showed high precision but lower recall, indicating a conservative extraction behavior. Models like Llama-3.1, Gemma-3, and Ministral exhibited significantly lower performance, particularly in recall and macro-F1 scores.
• Metric Discrepancies: A significant gap was observed between micro- and macro-averaged F1 scores (e.g., Phi-4 micro-F1: 63.77 vs. macro-F1: 15.49). This indicates that while models perform well on aggregate data, they are highly inconsistent across individual videos, often failing to detect minority classes (misinformation instances).
• Framework Application: When applied to a subset of six videos, the Entropy-TOPSIS framework successfully identified high-risk content that standard metrics might miss. For instance, it flagged videos where the model failed to extract claims (PA=0) as "High Risk," triggering expert intervention. Conversely, it highlighted a vulnerability where formal, academic-sounding language (even in conspiracy theories) could mask high harm potential by achieving high performance accuracy scores, suggesting a need for asymmetric penalty rules in future iterations.

Significance and Claims
The paper posits that while SLMs like Phi-4 offer a promising, cost-effective foundation for health misinformation detection in low-resource languages, they are not yet mature enough to operate autonomously in complex cultural contexts. The study emphasizes that technical accuracy alone is insufficient for CALD communities; detection systems must integrate cultural sensitivity and harm assessment to be effective.

The proposed framework offers a foundational approach for developing future models that prioritize both accuracy and cultural responsiveness. By combining objective weighting (Entropy) with multi-criteria decision making (TOPSIS), the authors provide a method to holistically evaluate misinformation that accounts for linguistic nuances and potential societal harm. The work concludes that future efforts must focus on fine-tuning SLMs for specific health domains, creating more expert-annotated benchmarks for low-resource languages, and refining evaluation frameworks to better address the unique vulnerabilities of CALD populations.