Bias in respiratory diagnoses by Large Language Models (LLMs) in Low Middle Income Countries (LMICs)

This study demonstrates that Large Language Models exhibit a High-Income Country bias in respiratory diagnoses, offering fewer relevant diagnostic suggestions for Low- and Middle-Income Country settings compared to local clinicians, even when prompted with specific location information.

The Gist

Imagine you have a super-smart, all-knowing digital librarian named "The AI." This librarian has read almost every book ever written, but there's a catch: 90% of the books in its library were written in New York, London, and Paris.

Now, imagine a doctor in a small village in Ghana, India, or Brazil asks this librarian: "I have a patient with a fever, a cough, and a strange rash. What could be wrong?"

The AI, drawing from its massive library of Western books, suggests the most common things it knows: maybe a flu, maybe pneumonia, maybe a rare genetic condition common in Europe.

But the local doctor knows something the AI doesn't. In their village, that same fever and rash might be a very common tropical disease that the AI has never even heard of because it wasn't in the "Western books."

This is exactly what the study above found.

The Story of the Study

The researchers wanted to see if these AI "doctors" (called Large Language Models or LLMs) were biased. They set up a little experiment:

1. The Test: They created 5 tricky medical stories (vignettes) about patients with breathing problems. These stories were designed so that the answer depended heavily on where the patient lived.
2. The Human Team: They asked real doctors from the UK, Ghana, India, Jordan, and Brazil to guess the top 4 diagnoses for each story.
3. The AI Team: They asked four famous AIs (ChatGPT, Claude, Google Gemini, and Microsoft Copilot) the same questions.
4. The Trick: They tried two things with the AI:
   • First, they accessed the AI from a "digital disguise" (a VPN) so the AI thought it was physically located in Ghana or India.
   • Second, they explicitly told the AI in the prompt: "This patient is in Ghana."

The Results: The "Western Lens" Problem

Here is the punchline: The AI failed to think like a local doctor.

• In the UK: The AI's guesses matched what the UK doctors thought about 50% of the time.
• In LMICs (Low- and Middle-Income Countries): The AI's guesses only matched what the local doctors thought about 32% of the time.

Even when the AI was told, "You are in Ghana," it still gave answers that felt more like they belonged in a London hospital. It was like asking a chef who only knows how to make French pastries to cook a local street food dish; they might try to use the same ingredients (flour and sugar) even though the local recipe needs spices and root vegetables.

The local doctors in Ghana, India, and Brazil considered a much wider, more diverse range of diseases because they knew the local "ecosystem" of sickness. The AI, however, kept sticking to the "High-Income Country" menu.

Why Does This Happen?

Think of the AI like a student who only studied for an exam using textbooks from one specific country.

• If the exam asks about a situation in that country, the student gets an A.
• If the exam asks about a situation in a different country with different rules, different weather, and different diseases, the student gets confused and gives an answer that sounds smart but is actually wrong for that specific place.

The AI was trained on data from the internet, and the internet is dominated by content from wealthy nations. So, the AI has a "High-Income Country Bias." It assumes the world works like the US or UK unless forced otherwise, and even then, it struggles to truly adapt.

The Big Warning

The authors of this paper are sounding an alarm bell. They are saying:

"Don't let these AI tools diagnose patients in developing countries yet."

If a doctor in a rural clinic relies on an AI that only knows about Western diseases, it could miss a critical, life-threatening local illness. It's like using a map of New York City to navigate the streets of Mumbai; you might recognize the word "street," but you'll get lost immediately.

The Takeaway

The paper concludes that before we let AI doctors into our clinics, especially in poorer countries, we need to:

1. Test them locally: Don't just test them in the US or UK. Test them in Ghana, Brazil, and India.
2. Fix the bias: The companies building these AIs need to feed them more diverse data so they understand the whole world, not just the wealthy parts.
3. Be careful: Until then, these tools should be used with extreme caution, if at all, in places where the "local menu" of diseases is very different from the AI's training.

In short: AI is a powerful tool, but right now, it's wearing "Western glasses." If we want it to help everyone, we need to give it a pair of glasses that fit the whole world.

Technical Summary

1. Problem Statement

The rapid integration of Generative AI (GenAI) and Large Language Models (LLMs) into healthcare raises significant concerns regarding geographic and epidemiological bias. Most foundational LLMs are trained on data corpora predominantly sourced from High-Income Countries (HICs), particularly North America and Europe.

The core problem addressed is whether these models, when applied to clinical scenarios in Low- and Middle-Income Countries (LMICs), exhibit a "High-Income Country bias." Specifically, the authors investigate if LLMs fail to account for local disease epidemiology, leading to diagnostic suggestions that are less relevant or accurate for patients in LMIC settings compared to local clinicians.

2. Methodology

The study employed a comparative experimental design involving clinical vignettes, human experts, and multiple LLMs across five countries.

• Study Design:
  • Clinical Vignettes: Five short, intentionally ambiguous respiratory clinical cases were created. These were designed to be sensitive to local epidemiology (i.e., the most likely diagnosis would vary by region).
  • Human Cohort: 21 practicing doctors (minimum 2 years experience in respiratory, emergency, or general practice) from five countries participated: UK (HIC control), Ghana, India, Jordan, and Brazil (LMICs). Each doctor independently provided the top 4 most likely diagnoses for each vignette without using AI.
  • AI Cohort: Four state-of-the-art LLMs were tested: ChatGPT-4o, Claude 3.5 Sonnet, Google Gemini-2.0-flash-001, and Microsoft Copilot (GPT-4).
• Experimental Conditions:
  1. VPN Access: Researchers accessed the LLMs via Virtual Private Networks (VPNs) originating from the specific LMIC to simulate local user IP addresses.
  2. Prompt Engineering: In a second experiment, the LLMs were explicitly prompted with the clinical vignette plus the specific country name where the case was located.
• Data Analysis:
  • Diagnoses from both doctors and LLMs were coded using ICD-11.
  • Two metrics were calculated:
    • Coverage: The proportion of diagnoses offered by doctors that were also suggested by the LLM.
    • Agreement: The proportion of LLM-suggested diagnoses that were also considered by the doctors.
  • Statistical analysis included Chi-squared tests to compare overlap proportions between HIC (UK) and LMIC settings.

3. Key Contributions

• First Empirical Evidence of LMIC Diagnostic Bias: This is the first study to specifically quantify the diagnostic bias of LLMs when applied to respiratory medicine in LMICs compared to HICs.
• Evaluation of "Contextual" Mitigation: The study tested whether simply informing the model of the location (via prompt) or accessing it from a local IP (via VPN) is sufficient to correct the bias.
• Benchmarking Framework: The authors propose a methodology for evaluating AI alignment with local epidemiology, suggesting that current "foundational" models are not yet ready for direct clinical deployment in diverse global settings without rigorous local validation.

4. Key Results

• Diagnostic Volume and Variety:
  • Human doctors generated 106 unique diagnoses across all cases.
  • LLMs (accessed via VPN) generated only 53 unique diagnoses.
  • Doctors in LMICs proposed a significantly wider range of diagnoses than LLMs, reflecting a broader consideration of local endemic diseases.
• Coverage Discrepancy (The "Bias"):
  • In the UK: LLMs covered 50% (95% CI 32.6–67.4%) of the diagnoses suggested by UK doctors.
  • In LMICs: LLMs covered only 32.0% (95% CI 23.1–42.3%) of the diagnoses suggested by LMIC doctors.
  • Statistical Significance: The difference in overlap was statistically significant (Chi-squared p = 0.028), indicating LLMs performed significantly worse in LMIC settings.
• Ineffectiveness of Mitigation Strategies:
  • The bias persisted even when the LLM was explicitly told the location of the patient in the prompt.
  • Accessing via VPN did not alter the model's internal knowledge base or training data bias.
• Agreement: No clear patterns emerged regarding the "agreement" metric (LLM suggestions matching doctor thoughts), suggesting the models are not just missing diagnoses but are potentially hallucinating or prioritizing HIC-centric pathologies.

5. Significance and Implications

• Clinical Safety: The findings suggest that deploying current LLMs as diagnostic aids in LMICs could lead to missed diagnoses of locally prevalent conditions (e.g., tropical diseases, specific infectious etiologies) in favor of HIC-centric pathologies.
• Training Data Limitations: The results confirm that the training corpora of major LLMs are heavily skewed toward HIC medical literature and epidemiology, creating a systemic blind spot for global health.
• Policy Recommendations:
  • Do not use AI diagnostic tools in LMICs without prior clinical evaluation in those specific regions.
  • Mandatory Local Validation: Developers must assess models for LMIC bias prior to release.
  • New Benchmarks: The authors call for the development of continuous, live benchmarking studies to evaluate AI models against diverse global epidemiological data, rather than relying on static, HIC-heavy datasets.

Conclusion: The study concludes that current LLMs exhibit a significant High-Income Country bias in respiratory diagnosis. This bias is robust against simple prompt engineering or IP masking, indicating a fundamental limitation in the training data of foundational models. Consequently, these tools should be used with extreme caution, if at all, in LMIC clinical settings until they are rigorously aligned with local epidemiological realities.