💬 NLP

Prompting language influences diagnostic reasoning and accuracy of large language models

This study demonstrates that prompting language significantly impacts the diagnostic reasoning and accuracy of most large language models in clinical settings, with English generally outperforming French across four of five evaluated models, highlighting a critical barrier to equitable global deployment.

Original authors: Adrien Bazoge, Josselin Corvellec, Sofiane Djillali Sid-Ahmed, Pierre-Antoine Gourraud

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Adrien Bazoge, Josselin Corvellec, Sofiane Djillali Sid-Ahmed, Pierre-Antoine Gourraud

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of five brilliant medical students. They are incredibly smart, have read millions of medical books, and can solve complex puzzles. However, there's a catch: they all went to school in different countries, and their "native language" for thinking is English.

This paper is like a report card testing how well these students perform when asked to solve medical cases in English versus French. The researchers wanted to see if the language used to ask the question changes the quality of the answer.

Here is the breakdown of what they found, using simple analogies:

1. The Setup: The "Medical Puzzle" Test

The researchers created 180 medical puzzles (called vignettes). These weren't just multiple-choice questions; they were short stories about patients with symptoms, similar to what a doctor sees in a real clinic.

The Test: They gave these puzzles to five different AI models (the "students"): o3, DeepSeek-R1, GPT-4-Turbo, Llama-3.1, and BioMistral.
The Twist: They asked the same puzzles twice to each student: once in English and once in French.
The Graders: Two real doctors acted as the teachers. They didn't just check if the final answer was right; they graded the entire thought process (the reasoning) on a scale of 0 to 18. They looked at things like:
- Did the student notice all the clues?
- Was the logic sound?
- Did they consider other possibilities?
- Was the final diagnosis correct?

2. The Results: The "English Advantage"

For four out of the five students, the results were clear: They performed significantly better when speaking English.

The Gap: Even though these models can speak French fluently, their "brain" worked better in English. It's like a musician who can play a song in French, but their fingers move faster and their rhythm is more perfect when playing in English.
The Score: On average, the English answers were better by a noticeable margin (between 0.37 and 0.91 points on the 18-point scale).
Where the Gap Happened: The difference wasn't just about the final answer. The students made more logical mistakes, missed more clues, and had weaker reasoning when forced to think in French.
- Analogy: Imagine a detective solving a mystery. In English, they follow the trail of clues perfectly. In French, they might get distracted, miss a clue, or jump to a conclusion that doesn't quite fit the evidence, even if they eventually guess the right name.

3. The Exception: The "Super-Student"

One model, o3, was the only one that showed no difference between English and French.

Why? The paper suggests this model has a special "superpower" called advanced reasoning capabilities. It seems that as AI gets better at thinking through problems step-by-step (like doing complex math in its head), the language barrier starts to disappear. It's like a student who has mastered the logic of the puzzle so well that it doesn't matter which language the instructions are written in.

4. Why This Matters (According to the Paper)

The paper argues that language isn't just a wrapper for the AI; it's part of how the AI thinks.

The "Training Diet" Problem: Most of these AIs were trained on huge piles of English data (like a student who only read English textbooks). Even if they can translate French, their "muscle memory" for medical reasoning is built on English patterns.
The Risk: If a doctor in a French-speaking hospital uses these tools, the AI might give a slightly less accurate diagnosis or a messier explanation than it would for an English-speaking doctor.
The Conclusion: The paper concludes that for AI to be truly fair and useful everywhere, we can't just rely on English. We need to ensure these models are just as sharp in French (and other languages) as they are in English.

Summary

Think of these AI models as multilingual chefs. They can cook a great meal (diagnosis) if you give them the recipe in English. If you give them the same recipe in French, four of the five chefs will still cook a good meal, but it might be slightly less seasoned or the presentation might be a bit off. Only one chef (o3) cooks the exact same perfect meal regardless of the language of the recipe.

The paper warns us that until we fix this "language bias," relying on these tools in non-English settings might mean getting a slightly lower-quality result.

Technical Summary: Prompting Language Influences Diagnostic Reasoning and Accuracy of Large Language Models

Problem Statement
While Large Language Models (LLMs) show promise in clinical decision support, their reliability in non-English languages remains uncertain. Most evaluations rely on English-language data and standardized multiple-choice benchmarks, which fail to capture the complexity of real-world clinical reasoning. Given that medical practice is deeply intertwined with language, culture, and local epidemiology, there is a critical gap in understanding whether LLMs trained predominantly on English corpora can perform diagnostic reasoning with equal fidelity in other languages, such as French. This disparity poses risks to equitable healthcare deployment and patient safety, particularly in multilingual settings.

Methodology
The authors conducted a bilingual comparative evaluation of diagnostic reasoning and final diagnosis accuracy across five LLMs: o3, DeepSeek-R1, GPT-4-Turbo, Llama-3.1-405B-Instruct, and BioMistral-7B.

Dataset: The study utilized 180 clinical vignettes covering 16 medical specialties (excluding purely technical or surgical fields). Vignettes were derived from synthetic creation, reference materials, and de-identified real-world encounters. Each vignette was constructed in French and independently translated into English by two physicians, ensuring semantic equivalence and clinical accuracy.
Prompting: Models were queried once per vignette in both languages using a standardized prompt instructing the model to act as a physician, provide clinical reasoning, and output the final diagnosis in JSON format.
Evaluation Framework: Two physicians independently assessed all model outputs using an 18-point scale comprising six criteria:
1. Internal Validity (0–5): Extraction and interpretation of vignette data.
2. External Validity (0–3): Accuracy of medical knowledge beyond the vignette.
3. Hypotheses and Differential Diagnosis (0–1): Generation of relevant differentials.
4. Logical Structure (0–4): Coherence and organization of reasoning.
5. Expression (0–2): Syntactic and semantic correctness.
6. Final Diagnosis Accuracy (0–3): Alignment with the reference diagnosis.
Statistical Analysis: Performance differences between English and French were analyzed using linear mixed models (LMM) with language as a fixed effect and vignette/rater as random intercepts. One-sided tests were used to evaluate the hypothesis that English performance exceeds French performance, with Bonferroni correction for multiple comparisons. Inter-rater reliability was assessed using Intraclass Correlation Coefficients (ICC) and Cohen's kappa.

Key Results

Overall Performance: Four of the five models (DeepSeek-R1, GPT-4-Turbo, Llama-405B, BioMistral-7B) performed significantly better in English than in French. The mean difference in overall scores ranged from 0.37 to 0.91 points on the 18-point scale (adjusted p < 0.05).
- Llama-405B showed the largest gap (0.91 points).
- BioMistral-7B, despite being medical-specialized, scored substantially lower overall (median ~8–9/18) and exhibited a significant language gap (0.78 points).
- o3 was the only model showing no overall language effect (mean difference 0.08, p = 1.000), achieving near-ceiling performance (median 18/18) in both languages.
Reasoning Dimensions: The language gap was not uniform across reasoning types. Significant disparities favoring English were observed in:
- Differential Diagnosis: Llama-405B, GPT-4-Turbo, and DeepSeek-R1 generated more consistent differentials in English.
- Logical Structure: GPT-4-Turbo, Llama-405B, and BioMistral-7B demonstrated more coherent reasoning organization in English.
- Internal Validity: Llama-405B and DeepSeek-R1 showed better extraction and interpretation of clinical data in English.
- Expression: BioMistral-7B exhibited more frequent syntactic errors in French.
Reasoning Types: The language gap was most pronounced for hypothetico-deductive and algorithmic reasoning, which require complex multi-step inference. No language effect was observed for probabilistic reasoning across any model.
Correlation: For BioMistral-7B, the correlation between reasoning quality and diagnosis accuracy was significantly stronger in English ( $\rho=0.75$ ) than in French ( $\rho=0.60$ ), suggesting that reasoning and accuracy are more tightly coupled in the English context for this model.

Significance and Claims
The paper claims that prompting language remains a critical determinant of LLM performance in clinical diagnostic reasoning. The findings demonstrate that the performance gap is not merely a matter of factual recall but extends to the quality, coherence, and internal validity of the clinical reasoning process itself.

The authors argue that:

Equity Implications: If models produce lower-quality reasoning and less accurate diagnoses in a patient's native language (non-English), this could exacerbate existing health disparities. This is particularly concerning as LLMs are increasingly used directly by patients for health information.
Beyond Exams: Unlike previous studies limited to multiple-choice medical board exams, this study confirms that language gaps persist in open-ended diagnostic reasoning tasks that more closely mimic real clinical practice.
Model Architecture Matters: The lack of a language gap in o3 suggests that advances in reasoning capabilities (e.g., test-time compute scaling, extended chain-of-thought) may partially mitigate language-related disparities, even when training data is predominantly English.
Resource Disparity: Since French is a relatively high-resource language with substantial representation in training corpora, the authors posit that these disparities are likely to be amplified for lower-resource languages.

The study concludes that ensuring equitable performance across languages must be a priority for model development and regulatory evaluation to prevent LLMs from missing the opportunity to reduce, rather than widen, inequities in access to trustworthy health information. To support future research, the authors have released the full bilingual dataset and evaluation code.

1. The Setup: The "Medical Puzzle" Test

2. The Results: The "English Advantage"

3. The Exception: The "Super-Student"

4. Why This Matters (According to the Paper)

Summary

More like this