💬 NLP

When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations

This study systematically evaluates the sensitivity of general-purpose and medical-specific Large Language Models to prompt variations using the MedMCQA benchmark, revealing that even minor lexical or syntactic perturbations can significantly degrade their reliability and induce harmful clinical outputs, thereby highlighting critical safety risks for their deployment in healthcare.

Original authors: Mahdi Alkaeed

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mahdi Alkaeed

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Fickle Chef" of Medicine

Imagine you have a world-class chef (the Large Language Model, or LLM) who is supposed to cook the perfect meal for a patient based on a recipe card (the prompt). This chef is incredibly smart and knows thousands of recipes.

However, this paper discovers a scary flaw: This chef is extremely sensitive to how the recipe is written.

If you change a single word, rearrange the order of the ingredients, or even just spell a word slightly differently, the chef might suddenly decide to serve a completely different dish—or worse, a poisonous one. The paper argues that in healthcare, where a "dish" is a medical diagnosis or advice, this kind of unpredictability is dangerous.

The Experiment: Testing the Chef's Stability

The researchers wanted to see if these medical AI chefs could handle small changes without messing up. They used a massive library of medical questions (called MedMCQA) and tested two types of "chefs":

General Chefs: Smart AI models not specifically trained for medicine (like GPT-3.5 or Llama3).
Specialist Chefs: AI models trained specifically on medical books and journals (like BioBERT or ClinicalBERT).

They tested the chefs under three conditions:

The Original Recipe: The standard, clean question.
The "Synonym Swap" (Lexical Perturbation): Changing words to their synonyms (e.g., changing "shortness of breath" to "dyspnea") or fixing typos.
The "Reordered Kitchen" (Syntactic Perturbation): Changing the structure of the sentence (e.g., switching from "The nurse gave the drug" to "The drug was given by the nurse").

What They Found: The "Fragile" Truth

The results showed that no chef is truly safe yet. Here is what happened:

1. The "Synonym Swap" was mostly okay (but not perfect)
When the researchers just swapped words for similar ones (like changing "five mg" to "5 mg"), most chefs stayed calm. They still gave the right answer. It's like if you asked for "a glass of water" instead of "a tumbler of water," the chef still brings water.

However, even here, the chefs sometimes got confused about how to present the answer (like forgetting to write it in a specific format), even if the medical advice was correct.

2. The "Reordered Kitchen" caused chaos
When the researchers changed the sentence structure or the order of the options, the chefs started to fail.

The Analogy: Imagine a chef who is great at following a list, but if you put the last item on the list first, they forget the first item entirely.
The Result: In some cases, simply rephrasing the question caused the AI to give a wrong diagnosis. For example, a patient with symptoms of COPD might be diagnosed with Pulmonary Edema just because the sentence structure was tweaked.

3. The "Specialist Chefs" weren't immune
You might think a chef trained only on medical books would be safer. The paper found that specialist chefs (BioBERT, etc.) were actually just as fragile, and sometimes more sensitive to structural changes, than the general chefs. They didn't have a "superpower" against these tricks.

The "Adversarial" Danger: The Sneaky Note

The paper also looked at "adversarial" prompts—these are like someone slipping a sneaky note into the recipe card to trick the chef.

The Analogy: Imagine someone whispering, "Ignore the first step, the patient is actually allergic to this," even though the patient isn't.
The Result: These small, tricky changes could make the AI recommend the wrong dosage of medicine or miss a critical symptom entirely. The paper calls this "clinically dangerous."

The Verdict: Why This Matters

The paper concludes that current medical AI is not "intrinsically safe."

The Problem: If a doctor asks the AI the same question in two different ways, the AI might give two different answers. One might be right, and the other might be wrong.
The Risk: In a hospital, you can't have a system that changes its mind just because you rephrased a sentence. If the AI hallucinates a drug interaction or misses a diagnosis because of a typo, patients could get hurt.

Summary in a Nutshell

Think of these AI models as brilliant but nervous students. They know the material perfectly when the test is written clearly. But if the teacher changes the font, swaps a few words, or rearranges the paragraphs, the student might panic and get the answer wrong.

The paper says: "We cannot trust these students to grade medical exams yet because they are too easily confused by how the question is asked." Before we let them help doctors, we need to teach them to be more stable and less sensitive to the way we talk to them.

Technical Summary: When Large Language Models Fail in Healthcare

Problem Statement

Large Language Models (LLMs) are increasingly deployed in healthcare for critical tasks such as clinical question answering, diagnosis support, and report summarization. However, these models exhibit significant sensitivity to subtle prompt perturbations, both lexical (word-level) and syntactic (structure-level). In safety-critical clinical environments, even minor variations in phrasing—such as synonym substitution, reordering of symptoms, or slight syntactic shifts—can lead to inconsistent outputs, diagnostic errors, or harmful advice. The core problem addressed is the lack of intrinsic robustness in both general-purpose (e.g., GPT-3.5, Llama3) and medical-specific LLMs (e.g., ClinicalBERT, BioLlama3, BioBERT) when faced with benign or adversarial input variations. This unpredictability poses unacceptable risks for clinical trust, as models may alter diagnoses or recommend incorrect dosages based solely on reworded inputs.

Methodology

The study employs a systematic sensitivity analysis framework using the MedMCQA benchmark, a large-scale dataset containing over 194,000 medical multiple-choice questions. The evaluation covers five models: GPT-3.5, Llama3, ClinicalBERT, BioLlama3, and BioBERT.

Perturbation Strategy

The authors categorize perturbations into two types:

Lexical Perturbations: Involving token-level substitutions, such as replacing medical terms with clinically accurate synonyms (e.g., "shortness of breath" to "dyspnea") or introducing minor orthographic errors. These are generated using BioSyn and validated via BioBERT and SciBERT embeddings to ensure medical equivalence.
Syntactic Perturbations: Involving structural changes, such as reordering clauses, changing active/passive voice, or altering sentence focus (e.g., clefting), while preserving the underlying clinical meaning.

Evaluation Framework

The pipeline involves:

Generating perturbed prompts ( $P'$ ) from original prompts ( $P$ ).
Validating semantic equivalence using Universal Sentence Encoder (USE) and BioSyn to ensure the perturbed input retains the original clinical intent.
Selecting top- $K$ syntactically aligned prompts based on cosine similarity scores.
Running these prompts through target LLMs and comparing outputs against ground truth.

Metrics

Performance is quantified using:

BERTScore: Measures syntactic and semantic similarity between the model's response and ground truth.
Universal Sentence Encoder (USE) Score: Evaluates sentence-level embedding similarity to assess preservation of clinical meaning.
Response Rate (RR): The proportion of valid responses generated.
Follow-up Instruction Rate (FIR): The extent to which the model adheres to specific formatting or structural constraints (e.g., JSON output) within valid responses.

Stability is categorized by score thresholds: 0.9–1.0 (Very Stable), 0.7–0.9 (Moderately Stable), 0.5–0.7 (Mild Sensitivity), and <0.5 (High Sensitivity).

Key Contributions

Formal Definition and Categorization: The paper formally defines and categorizes prompt perturbations into lexical and syntactic variations, systematically evaluating their impact on LLM performance in healthcare tasks.
Comparative Analysis: It provides a comprehensive comparison between general-purpose models and domain-specific medical models, revealing that neither category is intrinsically immune to sensitivity.
Identification of Vulnerabilities: The study identifies critical failure modes, including "instruction drift" (where models fail formatting constraints despite correct medical logic) and "structural misinterpretation" (where syntactic changes lead to false diagnoses).
Benchmarking Evidence: The authors consolidate evidence on LLM robustness across healthcare domains, detailing observed effects of specific perturbation types on model consistency and accuracy.

Results

General Sensitivity: All evaluated models, including medical-specific ones, showed reduced performance under perturbation. Accuracy dropped from approximately 98–99% in unperturbed settings to 94–95% under syntactic changes.
Lexical vs. Syntactic Impact: Models demonstrated relative resilience to simple lexical substitutions (e.g., synonyms) but showed significant fragility under syntactic reordering and misleading contextual cues.
Instruction Drift: Lexical variations often caused models to prioritize natural language flow over strict formatting instructions (e.g., JSON structure), leading to a drop in FIR scores even when the medical diagnosis was correct.
Structural Collapse: Syntactic perturbations occasionally caused total systemic failure, where the model misweighted clinical signs (e.g., confusing "increased sputum" with "Pulmonary Edema" instead of "COPD"), resulting in false diagnoses (RR failure).
Model Performance: BioLlama3 demonstrated the highest robustness among the tested models, while BioBERT and ClinicalBERT exhibited higher sensitivity, particularly to syntactic variations. General-purpose models like GPT-3.5 showed competitive but not superior stability compared to specialized models.

Significance and Claims

The paper asserts that medical LLMs are not intrinsically safe. The findings challenge the assumption that domain-specific fine-tuning guarantees reliability in clinical settings. The authors emphasize that:

Unpredictability is Unacceptable: In high-stakes healthcare, models that change diagnoses due to reworded inputs or hallucinate medications under slight rephrasing cannot be reliably trusted by clinicians.
Need for Robustness: Current benchmarks focusing solely on accuracy on clean data are insufficient. There is an urgent need for systematic robustness evaluation that accounts for lexical and syntactic variations.
Future Directions: The authors call for standardized metrics, controlled perturbation testing (similar to frameworks like MedFuzz and RAmBLA), and the development of models that are either provably resilient or transparently limited in their deployment to ensure patient safety.

The study concludes that while LLMs hold significant potential for healthcare, their current fragility under prompt perturbations necessitates careful prompt design and further research into adversarial robustness before widespread clinical adoption can be considered safe.