Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs: Identifier vs Context Effects

Technical Summary: Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs

Problem Statement
The integration of Large Language Models (LLMs) into clinical workflows faces a critical reliability challenge: the potential for models to alter clinically correct diagnoses based on non-decisive cultural information. While prior research has examined biases related to race and gender, the specific interaction between cultural identifiers (e.g., ethnicity, religion) and contextual cues (e.g., cultural practices, settings) in medical reasoning remains under-investigated. Existing benchmarks often lack the granularity to distinguish whether errors stem from identity markers or narrative context. This study addresses the gap in understanding whether cultural cues in patient narratives—such as references to Ramadan fasting or Indigenous community gatherings—spuriously shift model predictions away from gold-standard diagnoses, even when the clinical facts remain unchanged.

Methodology
The authors introduce a counterfactual evaluation framework designed to isolate the effects of cultural cues on medical question-answering (QA) accuracy.

• Dataset Construction: Starting with 150 items from the MedQA dataset, the authors generated 1,650 variants. Each original question was augmented with non-medically decisive text targeting three cultural groups: Indigenous Canadian, Middle-Eastern Muslim, and Southeast Asian.
• Augmentation Types: The study systematically varied the input using three perturbation types:
  1. Identifier (Id): Explicit cultural self-identification (e.g., "a 35-year-old Muslim man").
  2. Context (Ctx): Culturally related situational details (e.g., "symptoms began during evening prayer").
  3. Combined (Id + Ctx): Both identifier and context inserted.
  • A Neutral control was also created by inserting length-matched, non-cultural text to rule out prompt length as a confounding variable.
• Validation: A clinician verified that the gold-standard answer remained invariant across all augmented variants, ensuring that any accuracy drop was due to spurious sensitivity to cultural cues rather than clinical ambiguity.
• Models Evaluated: The study tested five models: GPT-5.2, Llama-3.1-8B, DeepSeek-R1, and MedGemma (4B and 27B).
• Evaluation Protocols: Models were tested under two prompting regimes:
  1. Option-only: The model outputs only the selected option.
  2. Brief-explanation: The model provides a 2–3 sentence rationale followed by the option.
• Metrics: Performance was measured using condition accuracy, flip rates (changes in option selection), and harmful flip rates (correct original answers becoming incorrect). An LLM-as-judge framework, validated by human annotators (Cohen's κ = 0.76), was used to detect culture-referential reasoning in explanations.

Key Contributions

1. Clinician-Verified Counterfactual Benchmark: A novel dataset that isolates cultural identifiers from context while preserving gold diagnoses, allowing for the precise measurement of cultural bias.
2. Granular Analysis of Bias Sources: A multi-model evaluation distinguishing the specific impacts of identity tokens versus narrative context on diagnostic accuracy.
3. Reasoning-to-Error Framework: An analysis linking culture-referential rationales directly to diagnostic failures, demonstrating that models often rely on unsupported cultural priors rather than clinical evidence.

Results

• Systematic Accuracy Degradation: Across all models, cultural cues significantly reduced diagnostic accuracy (Cochran's Q, p < 10⁻¹⁴). The most substantial performance drops occurred when identifiers and context co-occurred, with accuracy reductions of 3–7 percentage points under option-only prompting.
• Identifier Dominance: Contrary to the assumption that context drives bias, the study found that identifier tokens alone often triggered error rates comparable to the combined (Id + Ctx) condition. Context-only edits generally produced smaller, non-systematic changes.
• Prompting Regime Effects: While brief-explanation prompting improved robustness in single-cue conditions (Id or Ctx), it failed to prevent accuracy declines when both cues were present.
• Reasoning Failures: More than half of the culturally grounded explanations generated by the models ended in incorrect answers. The analysis revealed that models frequently substituted high-salience clinical features with unsupported generalizations about diet, lifestyle, or disease prevalence based on cultural cues.
• Model Variability: DeepSeek-R1 and MedGemma-4B exhibited the highest sensitivity to cultural cues, particularly in Indigenous contexts. MedGemma-27B demonstrated relatively higher stability, though still showed sensitivity.

Significance and Claims
The paper claims that current medical LLMs exhibit spurious sensitivity to cultural cues, leading to diagnostic failures that compromise patient safety and equity. The authors argue that these models rely on cultural stereotypes rather than clinically grounded personalization when processing patient narratives.

The study concludes that for medical LLMs to be deployed reliably in high-stakes clinical settings, rigorous auditing and mitigation of culturally induced diagnostic errors are prerequisites. The authors emphasize that model development must account for the cultural backgrounds of diverse populations to ensure that knowledge alignment does not inadvertently introduce bias. The released prompts and augmentation tools are intended to support future evaluation and the development of more equitable healthcare AI systems.