💬 NLP

Safety That Does Not Transfer: Cross-Lingual Clinical Correctness Drift in Deployable Medical Language Models

This study reveals that while frontier large language models maintain clinical safety across English and Hausa, locally deployable small models exhibit a severe safety drift in Hausa, dropping from clinically correct to actively harmful responses despite performing adequately in English, indicating that the safety deficit stems from the model class rather than the language or clinical content.

Original authors: Anthonio Oladimeji Gabriel, Dimeji Olawuyi, Toba Ajayi, Temilola Aderemi

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Anthonio Oladimeji Gabriel, Dimeji Olawuyi, Toba Ajayi, Temilola Aderemi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot librarian who can answer any question you ask. In the world of artificial intelligence, we call these "Large Language Models." For a long time, scientists have been testing these robots to make sure they are safe, especially when it comes to serious topics like health. But here's the catch: almost all the safety tests have been done in English, and they've mostly been done on the biggest, most powerful robots that live in massive computer clouds.

However, in many parts of the world, people don't have access to those giant cloud computers or fast internet. Instead, they use smaller, simpler versions of these robots that live directly on their phones or local computers. These smaller robots are like the "pocket-sized" versions of the super-librarian. The big question scientists are asking is: If a robot is safe and smart when speaking English on a giant computer, does it stay safe and smart when it's shrunk down to fit on a phone and asked to speak a local language like Hausa? It's a bit like asking if a chef who can cook a perfect steak in a five-star kitchen can still cook a safe, edible meal using a tiny camping stove and ingredients they've never seen before.

The Great Translation Trap

A team of researchers decided to put this idea to the test. They wanted to see if the "safety" of medical advice given by AI models would survive the journey from English to Hausa, a language spoken by millions in Northern Nigeria. They weren't just looking at whether the AI would say "I can't answer that" (which is a common safety test); they wanted to see if the AI would actually give the right medical advice.

To do this, they created a set of medical questions about three serious health issues common in the region: malaria, sickle cell disease, and tuberculosis. They wrote these questions in English and then translated them into Hausa. They asked two types of robots:

The "Frontier" Model: A massive, top-of-the-line AI accessed via the internet (the "five-star kitchen chef").
The "Deployable" Models: Five smaller, cheaper AI models that can run on local hardware without needing the internet (the "camping stove chefs").

They asked questions like, "How do I treat a fever?" or "My child can't drink water, what should I do?" They also included tricky questions designed to see if the AI would give dangerous advice, such as stopping medicine too early or using unproven home remedies.

The Shocking Result: Safety Doesn't Travel

The results were a bit like a magic trick gone wrong. When the researchers asked the questions in English, the smaller, local models were actually quite good. They gave correct answers most of the time. But the moment they switched the questions to Hausa, the performance of these local models crashed.

On a scoring scale where a perfect answer is a 2 and a dangerous, harmful answer is a -1, the local models' average score dropped from a healthy 1.57 in English to a dangerous -0.03 in Hausa. In plain English, this means that while the local models were "competent" in English, they became "actively harmful" on average when speaking Hausa. They started giving confident, fluent, but completely wrong medical advice.

For example, in the English version, a model might correctly say, "You need to see a doctor immediately for this chest pain." In the Hausa version, that same model might confidently say, "Just drink this herbal tea and you'll be fine," even though the guidelines say that specific symptom is a medical emergency.

The "Frontier" Model Stays Cool

Here is the most important part of the story: The researchers also tested the giant, powerful "Frontier" model. When they asked it the same questions in Hausa, it didn't crash. It stayed safe and accurate, dropping only slightly from a perfect 2.00 in English to 1.75 in Hausa. It never gave a harmful answer in either language.

This discovery rules out a few big ideas. It proves that the problem isn't the Hausa language itself (since the big robot handled it just fine). It also proves the problem isn't the medical questions (since the small robots could answer them perfectly in English). The problem is strictly the type of robot being used. The "safety" that was promised for these smaller, local models only exists in English. Once you shrink them down and translate them, their safety guardrails seem to fall apart.

Confidently Wrong is the Worst Kind of Wrong

The researchers found something particularly scary about these failures. It wasn't just that the small robots got confused or refused to answer. Often, they gave answers that sounded very confident and fluent but were medically dangerous. This is called "Dangerous Confidence." If a robot says, "I don't know," a person might go find a human doctor. But if the robot says, "Take this specific pill," and it's the wrong pill, the person might take it and get hurt.

They also noticed some weird behavior. Sometimes, when asked a question in Hausa, the small robots would answer in a completely different language, like Swahili, or just repeat nonsense. This suggests that these smaller models don't have a clear understanding of the different African languages; they get them mixed up, like a student trying to speak a language they only half-learned.

The Bottom Line

The study concludes that we cannot assume a medical AI is safe just because it passed safety tests in English. If we want to use these tools in places where people speak local languages and use smaller computers, we have to test them in that language and on that type of computer.

The "safety" of these models is not a magic shield they carry everywhere; it's a feature that depends entirely on how they are built and where they are used. For now, the researchers suggest that we should be very careful about trusting these smaller, local medical robots in languages like Hausa until they have been proven safe in that specific context. The big, powerful robots seem to handle the translation well, but the little ones? They are currently too risky to trust with life-or-death advice.

Technical Summary: Safety That Does Not Transfer

Problem Statement
Current safety evaluations for Large Language Models (LLMs) are predominantly conducted in English and focus on frontier systems. This creates a critical blind spot for low-resource health settings, such as northern Nigeria, where users rely on small, quantized, locally hosted models running on consumer hardware without network dependence. The paper posits that safety is not an intrinsic property of a model but one established through evaluation. Consequently, safety assurances derived from English-language testing of frontier models may not transfer to local languages or deployable model tiers. Specifically, the study investigates whether clinical safety established in English transfers to Hausa, and whether observed failures are attributable to the language, the clinical task, or the class of model permitted by low-resource deployment.

Methodology
The study employed a matched-pair experimental design to measure "correctness drift"—the change in clinical accuracy between prompts differing only in language.

Clinical Scope: Three high-burden conditions in northern Nigeria were selected: malaria, sickle cell disease, and tuberculosis.
Ground Truth: Correctness was adjudicated against Nigerian national treatment guidelines (e.g., National Malaria Elimination Programme, National Tuberculosis and Leprosy Control Programme) rather than Western clinical defaults.
Prompt Engineering: Twelve English items (four per condition) were constructed, covering:
1. Knowledge recall.
2. Emergency triage (identifying danger signs requiring referral).
3. Leading questions inviting contraindicated actions (e.g., stopping TB medication early).
4. Traditional-remedy probes.
  These were translated into Hausa by a fluent speaker, with back-translations retained for audit.
Models Evaluated: Six models were tested:
- Frontier Reference: One system accessed via API (Gemini).
- Locally Deployable Tier: Five models (4–9 billion parameters) run via Ollama at default quantization. This included two medically fine-tuned models (medgemma:4b, bimistral) and three general-purpose models (gemma2:9b, qwen2.5:7b, llama3.1:8b).
Evaluation Metrics: All 128 responses were scored independently by two fluent Hausa speakers (blind to each other) on:
- Clinical Correctness: A scale from 2 (correct) to -1 (actively harmful).
- Safety Hedging: Appropriate, under-hedged, or over-refusing.
- Language Quality: Fluent, awkward, or broken.
- Output Language: Whether the model responded in the requested language.
- Dangerous Confidence Rate: The proportion of responses that were unhedged, confident, and clinically wrong.

Key Results
The study observed a significant "correctness drift" for locally deployable models when moving from English to Hausa, while the frontier model remained stable.

Performance Drop: Among the five locally deployable models, mean clinical correctness fell from 1.57 in English to -0.03 in Hausa. On the evaluation scale, this represents a shift from "competent" to "actively harmful" on average.
Frontier Stability: The frontier model moved from 2.00 to 1.75, remaining competent and producing no responses judged harmful in either language.
Condition-Specific Drift: The decline was consistent across all three conditions. Notably, sickle cell disease showed the strongest English performance (1.85) but the weakest Hausa performance (-0.30) among the local models.
Failure Modes:
- Harmful Output: Locally deployable models produced responses judged harmful in 38% of Hausa cases (raw flags) compared to 5% in English.
- Language Failure: Models did not reliably answer in Hausa. medgemma:4b frequently timed out; llama3.1:8b answered in Swahili; bimistral produced unidentifiable text.
- Fine-Tuning Ineffectiveness: Medically fine-tuned models performed no better, and often worse, than general-purpose counterparts in Hausa, suggesting domain adaptation conducted in English did not extend to low-resource languages.
Inter-Rater Reliability: Agreement was substantial for clinical correctness ( $\kappa = 0.70$ ) and near-perfect for language identification ( $\kappa = 0.98$ ). Agreement on the "harm" flag was initially poor ( $\kappa = 0.22$ ) due to ambiguity in defining harm for unintelligible emergency responses; an adjudication rule was applied to resolve this, resulting in a 25% harm rate for deployable models in Hausa.

Key Contributions

Correctness-Drift Benchmark: Introduces a benchmark anchored to Nigerian national treatment guidelines rather than Western defaults, specifically designed to measure the gap between English and Hausa performance.
Dangerous Confidence Rate: Proposes a metric isolating responses that are fluent, confident, and clinically wrong, distinguishing them from silent failures (refusals or nonsense) which are less likely to be acted upon by users.
Locus of Failure: Isolates the deployment tier as the primary source of risk. The study demonstrates that the safety deficit is a property of the "deployable tier" (small, quantized models) rather than the Hausa language or the clinical material itself, as evidenced by the frontier model's competence in both languages.

Significance and Claims
The paper argues that safety assurance stopping at English or frontier systems fails to detect a critical class of failure relevant to low-resource settings.

Reframing the Problem: The issue is not linguistic intractability (since frontier models handle Hausa well) but an "assurance scope" problem. Models evaluated as safe in English may be unsafe in the specific language and quantization tier where they are deployed.
Nature of Risk: The most dangerous failures are those that do not announce themselves (e.g., refusal or nonsense) but instead provide fluent, assured, and incorrect medical advice.
Procurement Implication: The authors claim that models below the frontier tier should not be relied upon for clinical guidance in languages where they have not been evaluated in that specific tier and language.
Evaluation Imperative: Safety evaluation must be conducted at the tier of deployment and in the language of use, measuring clinical correctness rather than just refusal rates.

The study acknowledges limitations, including the small sample size (128 responses), the use of a single frontier reference, and the reliance on non-clinically-qualified raters for scoring against written guidelines. However, the pooled comparison between tiers is presented as the robust finding.

Technical Summary: Safety That Does Not Transfer

More like this