💬 NLP

Evaluating the Presence of Sex Bias in Clinical Reasoning by Large Language Models

This study reveals that contemporary large language models exhibit significant, model-specific sex biases in clinical reasoning when processing non-informative patient vignettes, underscoring the critical need for conservative configuration, data auditing, and human oversight to ensure safe healthcare integration.

Original authors: Isabel Tsintsiper, Sheng Wong, Beth Albert, Shaun P Brennecke, Gabriel Davis Jones

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Isabel Tsintsiper, Sheng Wong, Beth Albert, Shaun P Brennecke, Gabriel Davis Jones

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have four different "digital doctors" (AI chatbots) that are incredibly smart at reading medical textbooks and writing reports. You want to see if they are fair when diagnosing patients. To test this, the researchers created 50 fake patient stories. These stories were carefully written so that the patient's sex (male or female) didn't actually matter for figuring out what was wrong. It was like a puzzle where the solution was the same regardless of whether the piece was red or blue.

Here is what happened when they asked these AI doctors to solve the puzzles:

1. The "Guessing Game" Bias

When the researchers asked the AI, "What is the patient's sex?" without giving any clues, the AI didn't say, "I don't know." Instead, they started guessing based on stereotypes, like a person guessing someone's job based on their name.

The "Female" Guessers: Three of the AIs (ChatGPT, Claude, and DeepSeek) had a strong habit of guessing the patient was female. For example, if the story was about a heart issue or a urinary problem, they still often guessed "female."
The "Male" Guesser: One AI (Gemini) did the opposite, guessing male most of the time.
The Specialty Stereotypes: The bias got even stranger depending on the medical field.
- If the story was about psychiatry, rheumatology, or blood disorders, all the AIs guessed "female" 100% of the time.
- If the story was about heart or urinary issues, all the AIs guessed "male" 100% of the time.
- It's as if the AIs have a mental map where "sadness" is always a woman and "heart trouble" is always a man, even when the story says nothing about that.

2. The "Temperature" Knob

The researchers tried turning a "temperature" dial on the AIs. Think of this like a spice level knob:

Low temperature (0.2): The AI is very strict, logical, and repetitive.
High temperature (1.0): The AI is more creative, random, and wild.

They hoped that making the AI more "creative" (turning up the heat) might break the stereotypes. It didn't. While the high-temperature AI gave slightly different lists of guesses, the underlying bias (guessing the wrong sex based on the specialty) stayed exactly the same. The "spice" didn't fix the recipe; it just made the same biased dish taste a little different.

3. The "I Don't Know" Option

In a second test, the researchers gave the AIs a third option: "Abstain" (or "I don't know").

ChatGPT immediately took the safe route and said "I don't know" every single time.
The others said "I don't know" most of the time, but not always.

The Catch: Even when the AIs stopped saying the patient's sex, they still thought about it. When asked to list possible diseases for a "male" version of the story versus a "female" version, the lists of diseases changed. The AI was still treating the two patients differently behind the scenes, even if it didn't explicitly label them.

4. The Bottom Line

The paper concludes that these AI models are like students who have memorized old, biased textbooks. They haven't learned to think neutrally; they are just repeating the patterns they saw in their training data.

The Bias is Built-In: The bias isn't a glitch; it's part of how the models are built.
Settings Don't Fix It: You can't just tweak a setting (like temperature) to make them fair.
The Danger: Because these models change their medical advice based on whether they think the patient is a man or a woman (even when the symptoms are identical), the authors say these general AI tools should not be used to make real medical decisions right now.

The Takeaway: If you ask these digital doctors for help, they might be smart, but they are also carrying heavy baggage of old-fashioned stereotypes. Until they are taught to ignore those stereotypes, humans need to keep a close eye on their work.

Technical Summary: Evaluating the Presence of Sex Bias in Clinical Reasoning by Large Language Models

Problem Statement

Large Language Models (LLMs) are increasingly integrated into healthcare workflows for documentation, education, and clinical decision support. However, these models are trained on vast text corpora that inherently encode historical social and biological biases, including sex disparities in diagnosis and treatment. While prior research has examined AI performance against human benchmarks and public perception of AI, there is a critical gap in empirical evidence regarding whether contemporary LLMs reproduce biological sex bias patterns when interpreting clinical information. Specifically, it remains unclear if these models default to sexed assumptions in the absence of explicit patient data and whether such biases alter downstream diagnostic reasoning.

Methodology

The authors conducted a controlled, three-experiment study using 50 clinician-authored clinical vignettes spanning 44 medical specialties. These vignettes were specifically designed to be "sex-neutral," meaning the initial diagnostic pathway should not be influenced by patient sex. The study utilized four general-purpose LLMs: OpenAI ChatGPT (gpt-4o-mini), Anthropic Claude 3.7 Sonnet, Google Gemini 2.0 Flash, and DeepSeek-chat.

The experimental design involved three distinct tasks:

Binary Sex Assignment: Models were presented with neutral vignettes (omitting patient sex) and instructed to assign a binary sex (male/female).
Abstention Capability: The same neutral vignettes were presented, but the prompt allowed a third option: "abstain." This tested whether models could refrain from unsolicited demographic inference.
Differential Diagnosis Reasoning: Models were asked to generate a top-five ranked list of differential diagnoses for vignette pairs that differed only by the patient's sex (male vs. female variants).

Experimental Conditions:

Temperatures: Each model was queried at three temperature settings (0.2, 0.5, and 1.0) to assess the influence of sampling variability.
Repetition: Each condition was repeated 10 times to ensure robustness.
Analysis:
- For Experiments 1 and 2, proportions of sex assignments and abstentions were analyzed using exact binomial tests.
- For Experiment 3, diagnostic lists were compared using four metrics: Jaccard similarity (overlap), item-level agreement (rank alignment), cumulative match characteristic (divergence point), and Kendall's tau-b (ordering similarity).
- Statistical significance was assessed using Chi-square tests, ANOVA, and Wilcoxon signed-rank tests.

Key Results

1. Sex Assignment Skew

All models demonstrated significant, model-specific biases in assigning sex to neutral clinical vignettes ( $\chi^2_{(3)}=363.6, p<0.001$ ).

ChatGPT: Showed the strongest female skew, assigning female sex in 70% of cases at temperature 0.5.
DeepSeek & Claude: Exhibited moderate female bias (61% and 59% respectively).
Gemini: Showed a consistent male skew, assigning female sex in only 36% of cases.
Temperature Effect: Temperature alone had no significant main effect on sex assignment ( $p=0.95$ ), though a significant interaction existed between model and temperature.

2. Specialty-Specific Patterns

Bias was not uniform across medical domains; it mirrored known clinical stereotypes:

Female-Labelled Specialties: Psychiatry, rheumatology, and haematology were labelled female in 100% of cases across all models. Endocrinology was predominantly female (60–100%).
Male-Labelled Specialties: Cardiology and urology were labelled male in 100% of cases. Pulmonology was predominantly male (95–100% for most models).

3. Impact of Abstention

When permitted to abstain:

ChatGPT: Abstained in 100% of cases across all temperatures, effectively eliminating explicit sex labelling but rendering the model non-informative for this task.
Other Models: Claude (84%), Gemini (80%), and DeepSeek (58%) abstained frequently at temperature 0.5.
Downstream Impact: Crucially, even when models abstained from explicit labelling, sex-contingent differences in the generated differential diagnoses persisted.

4. Diagnostic Reasoning Divergence

When generating differential diagnoses for male vs. female variants of the same vignette:

Divergence Rates: At temperature 0.5, 58–78% of comparisons resulted in divergent lists (differing in content or order). ChatGPT showed the highest divergence (78%), while DeepSeek showed the lowest (58%).
Similarity Metrics: Diagnostic overlap (Jaccard similarity) declined as temperature increased for all models. ChatGPT showed the most marked decline (0.76 at temp 0.2 vs. 0.50 at temp 1.0).
Conclusion: Increasing temperature increased variability but did not eliminate sex-contingent diagnostic differences. The underlying bias appeared intrinsic to the model's learned representations rather than an artifact of sampling.

Significance and Claims

The authors claim this study provides the first direct empirical evidence that sex bias is not merely theoretical but detectable in the outputs of widely accessible, general-purpose LLMs applied to healthcare.

Bias is Intrinsic and Model-Specific: The direction and magnitude of bias are determined primarily by model architecture and training data, not by inference-time temperature settings. Different commercial models encode distinct "sex priors."
Stereotypes Override Clinical Neutrality: The models replicate entrenched linguistic and cultural stereotypes (e.g., associating psychiatry with women and urology with men) rather than performing neutral, evidence-based reasoning. This suggests the models are engaging in "stereotype completion" rather than Bayesian inference over population statistics.
Limitations of Configuration: Simple configuration changes, such as allowing abstention or adjusting temperature, are insufficient to neutralize bias. While abstention reduces explicit labelling, it fails to prevent downstream diagnostic differences.
Clinical Implications: The findings argue against the use of general-purpose LLMs to guide diagnosis or treatment decisions without rigorous oversight. The authors advocate for:
- Conservative, documented model configurations.
- Specialty-level clinical data auditing to identify bias hotspots.
- Continued human oversight when deploying these models.
- Structural changes in model development, including curated training corpora and reinforcement learning objectives that penalize stereotype completion.

The paper concludes that while LLMs can generate fluent clinical text, their current iteration exhibits stable, systematic sex biases that compromise safety and equity, necessitating a shift from general-purpose deployment to purpose-built, audited, and human-supervised systems.