Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA

1. Problem Statement

The safe deployment of Large Language Models (LLMs) in clinical settings relies not only on high accuracy but also on robust uncertainty calibration. In clinical workflows, models must defer to human clinicians when their confidence is low. However, current research suggests that social identity descriptors (e.g., race, gender) can bias LLM outputs.

This paper addresses a critical gap: How do clinically non-diagnostic social identity markers—specifically sexual orientation and religious affiliation—affect both the accuracy and the calibration (reliability of confidence scores) of medical LLMs? The authors hypothesize that these markers may not only shift predictions but also degrade the model's ability to correctly estimate its own uncertainty, posing a risk to equitable and safe patient care.

2. Methodology

Data and Counterfactual Construction

• Dataset: The study uses 2,364 questions from the MedQA-USMLE dataset (US Medical Licensing Examination), filtered for adult patient vignettes (≥18 years) and excluding pre-existing mentions of psychiatry, sexual orientation, or religion.
• Counterfactual Variants: The authors generated counterfactual versions of each vignette by inserting a single templated sentence specifying:
  • Sexual Orientation: "The patient identifies as heterosexual" or "The patient identifies as homosexual."
  • Religion: "The patient is Catholic," "Muslim," or "atheist."
  • Intersectional: Combinations of orientation and religion (e.g., "homosexual" + "Catholic").
• Control: "Heterosexual" insertions served as a near-neutral baseline.

Models Evaluated

Nine LLMs were tested, spanning general-purpose and biomedical variants, ranging from 3B to 70B parameters, including:

• Open-Weight: Llama-3.2-3B, Qwen3-4B/30B, Bio-Medical-Llama-3-8B, Llama-3.1-8B/70B, OpenBioLLM-70B.
• Closed-Source: GPT-4.1-mini, GPT-5.1.

Metrics and Uncertainty Estimation

• Accuracy: Measured via multiple-choice selection (MCQA) and open-ended generation.
• Uncertainty Quantification: The study utilizes Semantic Entropy, a method that quantifies predictive uncertainty by measuring the entropy of semantically distinct outputs across multiple generations (K=10), rather than just surface token probabilities.
• Calibration Metrics:
  • Brier Score: Measures the mean squared error between predicted probabilities and binary correctness (lower is better).
  • Confidence: Defined as $1 - \text{normalized entropy}$.
  • ECE (Expected Calibration Error) & AUROC: Used as supplementary metrics to assess calibration and discrimination.

Experimental Design

1. Multiple-Choice QA: Evaluated accuracy and calibration shifts across all identity conditions.
2. Open-Ended Case Study: To rule out artifacts specific to multiple-choice formats, the authors converted questions to open-ended generation. They used an LLM-as-a-judge (validated by a clinician) to assess correctness and semantic clustering to calculate entropy.
3. Qualitative Analysis: Examined specific failure cases to understand the nature of the errors.

3. Key Contributions

1. Discovery of a "Calibration Crisis": The paper demonstrates that identity markers do not merely shift accuracy; they systematically degrade uncertainty calibration. Models become less reliable in their confidence signals when sensitive identity cues are present.
2. Intersectional Vulnerability: The study reveals that the combination of sexual orientation and religious markers produces non-additive, idiosyncratic harms. Intersectional identities (e.g., homosexual + Catholic) often cause performance and calibration drops that exceed the sum of individual identity effects.
3. Beyond Multiple-Choice: By validating findings in an open-ended generation setting, the authors prove that these distortions are not artifacts of the multiple-choice format but reflect fundamental reasoning flaws in the models.
4. Clinical Relevance: The paper highlights that even "frontier" models (e.g., GPT-5.1) exhibit significant calibration loss, suggesting that current state-of-the-art models are not yet safe for confidence-based clinical deferral workflows involving diverse patient populations.

4. Key Results

Accuracy Shifts

• Homosexual Markers: Consistently triggered significant accuracy drops across most models. For example, OpenBioLLM-70B saw a −7.21% accuracy drop with "homosexual" insertions compared to the baseline.
• Heterosexual Markers: Generally acted as a neutral baseline with negligible shifts.
• Religious Markers: Showed varied but generally negative trends, with "Muslim" and "Catholic" insertions causing occasional significant drops.
• Intersectionality: Combining "homosexual" with religious descriptors exacerbated the decline. For Bio-Medical-Llama-3-8B, the drop for "+homo+Cat" (−5.58%) was significantly larger than the drops for "+homo" (−2.37%) or "+Cat" (−1.10%) alone.

Calibration and Confidence Degradation

• Brier Score Increases: Identity insertions led to consistent increases in Brier scores (indicating worse calibration). Under the "+homo" condition, Llama-3.1-70B and OpenBioLLM-70B saw relative Brier score increases of +29.3% and +32.0%, respectively.
• Confidence Mismatch: Models often decreased their confidence when identity markers were inserted. However, this increased uncertainty was insufficient to maintain calibration (as evidenced by rising Brier scores). This creates a dangerous scenario where models might not defer to clinicians even when they are wrong, or they might defer excessively, eroding system utility.
• Frontier Models: Even GPT-5.1, the most robust model, showed significant calibration loss under "+homo" conditions, though it was less sensitive to intersectional cues than other models.

Open-Ended Generation

• The degradation persisted in open-ended settings. GPT-5.1 showed a −3.81% accuracy drop and a 5.3% increase in Brier score under "+homo" conditions.
• Qualitative Failure Modes: The study identified "stereotype-driven reasoning." For instance, when a patient was described as "homosexual," a model shifted from a clinically supported diagnosis (e.g., hepatorenal syndrome) to a stereotype-driven explanation (e.g., HIV-associated nephropathy) despite the clinical vignette lacking evidence for the latter, all while maintaining high confidence.

5. Significance and Implications

• Risk to Equitable Care: The findings suggest that clinical AI systems may provide less accurate and less reliable advice for LGBTQ+ and religious minority patients, potentially leading to inequitable triage and unsafe care.
• Failure of Confidence-Based Deferral: Since clinical safety often relies on models deferring to humans when confidence is low, the distortion of confidence signals means these safety mechanisms may fail for specific demographic groups.
• Limitations of Attribute Removal: The authors argue that simply stripping identity attributes from inputs is not a robust solution. Identity can be inferred from proxy signals, is sometimes clinically relevant, and models must be robust to benign input variations.
• Future Directions: The paper calls for counterfactual calibration as a standard robustness check for clinical readiness, calibration-aware training, and the development of deferral policies explicitly validated across identity groups.

In conclusion, the paper provides empirical evidence that social identity markers act as a "poison pill" for LLM reliability in medicine, creating a specific vulnerability where models become both less accurate and less trustworthy in their uncertainty estimates for marginalized groups.