💬 NLP

How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking

This paper introduces BanglaMedVQA, the first clinically validated dataset and benchmark for Bangla Medical Visual Question Answering, revealing that current state-of-the-art foundation models, including Gemini and GPT-4.1 mini, perform significantly poorly on specialized diagnostic tasks due to challenges inherent in low-resource languages and complex medical reasoning.

Original authors: Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Fahim, Md Farhad Alam Bhuiyan

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Fahim, Md Farhad Alam Bhuiyan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of very smart, well-read robots (called AI models) that are great at looking at pictures and answering questions about them. You've taught them to speak English fluently, and they can even describe a picture of a broken bone or a tumor quite well.

But what happens if you ask them the same questions in Bangla, a language spoken by nearly 300 million people? That is exactly what this paper investigates.

Here is the story of their findings, broken down into simple parts:

1. The Missing Puzzle Piece

For a long time, scientists have been testing these AI robots on medical pictures in English. They have big, high-quality test sets for English. But for Bangla? There was almost nothing. There was one old dataset, but it was like a puzzle with missing pieces and blurry pictures—nobody knew if the answers were actually correct or if a doctor had even checked them.

The authors of this paper decided to build a brand new, high-quality test set called BanglaMedVQA.

The Ingredients: They took thousands of medical images (like X-rays and CT scans) from two famous, trusted English databases.
The Recipe: They used advanced AI to translate the medical questions and answers into Bangla.
The Quality Control: This is the most important part. They didn't just trust the computer. They hired two real, certified doctors to check every single question and answer. If the doctors said, "No, that's wrong," they fixed it. The result was a dataset with a 97% accuracy rate, verified by humans.

2. The Big Test: How Smart Are the Robots?

The authors put the top AI robots (both the expensive, closed-source ones like Google's Gemini and OpenAI's GPT, and the free, open-source ones) through this new Bangla test.

The Results were surprising and a bit scary:

The "General" Questions: When asked simple things like, "What kind of picture is this?" (Is it an X-ray or an MRI?) or "What body part is this?" (Is it a lung or a brain?), the robots did okay. They got about 40% of the answers right.
The "Specialist" Questions: When the questions got harder—like, "What specific disease is this?" or "Exactly where in the lung is the tumor?"—the robots crashed.
- Even the best robots (Gemini and GPT) performed worse than random guessing on these hard questions. It was as if they were just throwing darts at a board and hoping to hit the bullseye.
- The free, open-source robots did even worse, often getting less than 10% right.

3. The Language Barrier

The researchers also tested the robots in English using the same questions.

The Gap: The robots were significantly better at English than Bangla. It's like a student who is an A+ in English class but fails the same test when it's translated into a language they barely understand.
The Takeaway: The robots haven't been trained enough on medical knowledge in Bangla. They are "low-resource" learners in this language.

4. Can We Help Them Think Better?

The researchers tried a trick called "Chain-of-Thought." Instead of asking the robot for the answer immediately, they told it: "Let's think step-by-step before you answer."

Did it work? Yes, but only a little bit. It helped the robots reason a bit more, especially the free ones, but it didn't fix the core problem. They still struggled to pinpoint exactly where a disease was or what specific condition it was.
The Bottleneck: The paper suggests the main problem isn't just the language; it's that the robots don't truly "see" the medical details in the picture well enough to explain them in Bangla.

5. The Bottom Line

This paper is a wake-up call.

Current Status: We have built a high-quality, doctor-verified test set for Bangla medical AI.
The Reality: Even the most advanced AI models today are not ready to be trusted with complex medical diagnoses in Bangla. They are good at describing the picture generally, but they fail at the critical job of diagnosing the patient.
The Future: We need to build better models and train them specifically on medical data in Bangla before we can ever use them in a real hospital.

In short: The robots are like medical students who have memorized a textbook in English but are currently failing their final exam when asked to diagnose a patient in Bangla. We have created the exam paper (the dataset) to prove this, and the scores show we have a long way to go.

Technical Summary: BanglaMedVQA – Benchmarking Large Vision-Language Models on Bangla Medical Visual Question Answering

Problem Statement
Despite the rapid advancement of Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) in general and medical domains, a critical gap exists for low-resource languages. While English Medical Visual Question Answering (MedVQA) has established benchmarks, Bangla—the seventh most spoken language globally with approximately 278 million speakers—lacks a reliable, clinically validated MedVQA dataset. Existing resources, such as the only publicly available dataset (Remon 2025), suffer from annotation errors, a lack of expert validation, and missing source metadata, rendering them unsuitable for rigorous benchmarking. Consequently, the performance of foundation models in Bangla medical reasoning remains unquantified and likely compromised by the absence of high-quality training and evaluation data.

Methodology
To address this, the authors introduce BanglaMedVQA, a new dataset and benchmarking framework. The methodology involves three primary phases:

Data Curation:
- Source Data: The dataset was constructed by leveraging two established biomedical datasets: MedICaT (Subramanian et al. 2020) for detailed condition and positional metadata embedded in captions, and ChestX-ray14 (Wang et al. 2017) for structured metadata and bounding box annotations.
- Generation Pipeline: A two-stage pipeline was employed. First, metadata (modality, organ, findings, position) was extracted or generated. Then, Gemini-2.5-flash was used with few-shot prompting to generate English Question-Answer (QA) pairs. These were subsequently translated into Bangla using Gemini-2.0-flash.
- Expert Validation: A subset of 500 QA pairs underwent rigorous verification by two certified physicians. The validation focused on multilingual quality (technical term accuracy) and positional consistency (alignment of text with visual regions). This process yielded a 97% acceptance rate and a Cohen's $\kappa$ of 0.89, confirming the dataset's reliability.
- Final Dataset: The resulting dataset contains 2,000 unique image–question–answer pairs, balanced between Chest X-Ray and MedICaT sources, covering five categories: modality, organ, abnormality, condition, and position.
Benchmarking Setup:
- Models Evaluated: The study systematically evaluated eight LVLMs, including closed-source models (Gemini 2.5 Flash, GPT-4.1 mini), open-source general-purpose models (Llama-3.2V, Gemma-3, Qwen2.5-VL, LLaVA-1.5), and open-source medical-specific models (Med-LLaVA, Med-Gemma).
- Prompting Strategies: Experiments were conducted under Zero-Shot and Chain-of-Thought (CoT) prompting conditions. CoT prompting involved appending "Let's think step by step" to encourage intermediate reasoning.
- Cross-Lingual Comparison: To isolate language effects, the same models were tested on the original English QA pairs alongside the Bangla translations.
Evaluation Metrics:
- Accuracy (Acc): Exact match proportion.
- BERTScore (BScore): Semantic similarity using contextual embeddings.
- LAVE (LLM-Assisted VQA Evaluation): A metric designed to provide more reliable performance assessment in VQA tasks.

Key Results
The evaluation reveals significant challenges for current models in the Bangla medical domain:

Overall Performance: Even top-performing closed-source models struggle. Gemini 2.5 Flash achieved the highest overall LAVE scores (55.44% in Bangla Zero-Shot), while GPT-4.1 mini performed significantly worse (38.60%). Open-source models generally lagged, with Gemma-3 12B leading the open-source category (46.54% LAVE).
Specialized vs. General Questions: A critical disparity exists between general questions (e.g., identifying modality or organ) and specialized diagnostic questions (e.g., identifying specific conditions or precise anatomical positions).
- On specialized tasks, performance drops precipitously. For instance, under CoT prompting in Bangla, Gemini 2.5 achieved only 23.25% on "Condition" and 24.50% on "Position."
- GPT-4.1 mini performed near random chance on these specialized categories (13% and 19.50%, respectively).
Language Disparity: Performance in Bangla is consistently lower than in English across all models. Closed-source models demonstrated better cross-lingual transfer, but open-source and medical-specific models showed severe degradation in Bangla, often dropping below 10% accuracy.
Impact of CoT: Chain-of-Thought prompting provided consistent, albeit modest, improvements, particularly for open-source models (e.g., Llama-3.2V accuracy increased from 17.50% to 30.17% in Bangla). However, it did not bridge the gap to clinical reliability for specialized tasks.
Medical-Specific Models: Surprisingly, medical-domain VLMs (Med-LLaVA, Med-Gemma) underperformed relative to general-purpose models in the Bangla context, likely due to limited multilingual adaptation.

Significance and Contributions
The paper makes three primary contributions:

Dataset Introduction: It presents BanglaMedVQA, the first clinically validated MedVQA dataset for Bangla, addressing the scarcity of high-quality resources for low-resource languages in medical AI.
Comprehensive Benchmarking: It provides the first systematic evaluation of both open-source and closed-source LVLMs on Bangla medical tasks, establishing zero-shot and CoT baselines.
Critical Findings: The study highlights that current foundation models, even the most advanced, lack the fine-grained reasoning capabilities required for specialized medical diagnosis in low-resource languages. The results underscore that while models can handle general visual queries, they fail significantly on diagnostic reasoning, often performing close to random guessing on condition and position queries.

The authors conclude that while foundation models show promise in general domains, their application in Bangla medical contexts is currently limited by insufficient visual understanding and linguistic grounding. The paper emphasizes the urgent need for top-notch evaluation methods and robust datasets to drive future progress in this underserved area, cautioning that current models are not yet ready for clinical decision-making in Bangla.