Investigating Gender Stereotypes in Large Language Models via Social Determinants of Health

This study investigates gender bias in Large Language Models within French healthcare contexts, demonstrating that these models rely on embedded stereotypes when processing interactions between gender and other social determinants of health, thereby highlighting the need for context-specific assessments that go beyond evaluating individual factors in isolation.

The Gist

Imagine you have a super-smart, digital assistant that has read almost everything on the internet, including millions of medical records. You ask it to help a doctor diagnose a patient. But here's the catch: this assistant didn't just learn facts; it also learned the hidden assumptions and stereotypes of the society that created the data it was trained on.

This paper is like a detective story where researchers try to catch this digital assistant in the act of making unfair guesses based on old-fashioned stereotypes.

The Setting: The "Medical Mirror"

Think of Large Language Models (LLMs) as mirrors. They reflect the world back to us. If the world has biases (like thinking "nurses are mostly women" or "engineers are mostly men"), the mirror shows those biases too.

In the medical world, this is dangerous. If a doctor asks the AI, "What might be wrong with this patient?" and the AI secretly thinks, "Oh, this person is a retired man who smokes, so it must be a heart issue," but the patient is actually a woman with a different condition, the AI might give the wrong advice. This is called bias, and it can lead to misdiagnoses.

The Investigation: Removing the "Name Tags"

The researchers wanted to see if the AI was making these guesses based on Social Determinants of Health (SDoH). These are the conditions of your life: your job, your income, whether you smoke, your marital status, etc.

To test this, they played a game of "Blind Guessing":

1. The Setup: They took real patient records from a French hospital.
2. The Magic Trick: They scrubbed the text to remove any obvious gender clues (like "he" or "she") and even neutralized job titles (changing "nurse" to "nurse" without specifying male or female).
3. The Challenge: They gave the AI a list of facts: "Patient is retired, smokes, lives alone, works as a laborer."
4. The Question: "Based only on these facts, is this person Male or Female?"

If the AI says "Male" every time it sees "retired smoker" or "laborer," it means the AI is relying on stereotypes rather than actual medical evidence. It's like guessing someone's gender just because they wear a specific type of hat, even though you can't see their face.

The Findings: The AI's "Internal Library"

The researchers tested 9 different AI models (some small, some huge) and found some fascinating things:

• The AI is a Stereotype Machine: Even when the gender was hidden, the AI consistently guessed "Male" for retired people, smokers, and manual laborers. It guessed "Female" for students, people in relationships, and homemakers. It was essentially reading a social rulebook it learned from the internet.
• Size Doesn't Always Mean Smarter: You might think a bigger, more powerful AI would be less biased. But the study found that smaller models were actually more confident in their stereotypical guesses. They were like a student who memorized a few rules and stuck to them rigidly, while the bigger models were a bit more unsure but still followed the same patterns.
• Medical Training Makes it Worse (Sometimes): They tested AI models that had been specifically trained on medical data. Surprisingly, these "specialized" doctors-in-training were sometimes more biased than the general ones. It's like if a medical student only read textbooks from the 1950s; they would learn the outdated stereotypes of that era.
• Humans Are Just as Guilty: The researchers also asked real humans to play the guessing game. The humans made the exact same stereotypical guesses as the AI! This proves that the AI isn't "evil"; it's just a reflection of how humans (and society) often think.

The Analogy: The "Cave"

The authors use a great metaphor from Plato's Allegory of the Cave. Imagine the AI is a prisoner in a cave, looking at shadows on a wall. The shadows are the data from the internet. The AI thinks the shadows are the whole truth. But the shadows are just reflections of human experiences, which are full of biases. The AI can't see the real world outside the cave; it can only guess based on the shadows it sees.

Why Does This Matter?

If we let these biased AI models help doctors:

• A female patient with a heart condition might be ignored because the AI thinks "heart attacks are for men."
• A male patient with postpartum depression might be overlooked because the AI thinks "depression after birth is for women."

The Solution: What Can We Do?

The paper suggests a few ways to fix this:

1. Check the Mirror: We need to test AI models specifically for these hidden stereotypes before we let them into hospitals.
2. Better Prompts: Sometimes, if you explicitly tell the AI, "Ignore social stereotypes, look only at the medical facts," it can do a better job (though it's not a perfect fix).
3. Human Oversight: We need to remember that AI is a tool, not a god. Doctors need to be the final decision-makers, aware that the AI might be "hallucinating" based on old stereotypes.

The Bottom Line

This paper is a wake-up call. It shows that even our most advanced AI tools are carrying around the baggage of human prejudice. To use them safely in healthcare, we have to learn how to unpack that baggage and ensure the AI is judging patients based on their actual health, not on who we think they are.

Technical Summary

Here is a detailed technical summary of the paper "Investigating Gender Stereotypes in Large Language Models via Social Determinants of Health."

1. Problem Statement

Large Language Models (LLMs) are increasingly used to support medical diagnosis by analyzing clinical records. However, these models often inherit and amplify biases present in their training data. While existing benchmarks evaluate biases regarding individual Social Determinants of Health (SDoH) factors (e.g., gender or ethnicity in isolation), they frequently fail to:

• Analyze interactions between different SDoH factors.
• Assess biases within context-specific medical scenarios.
• Detect how models might rely on stereotypical associations (e.g., linking specific occupations or marital statuses to a specific gender) to make decisions, potentially leading to misdiagnoses or allocational harms (e.g., suggesting menstrual issues for a male patient).

The core problem is the lack of a framework to probe how LLMs encode and utilize the complex interplay between gender and other SDoH factors (like occupation, education, and substance use) when processing anonymized patient data.

2. Methodology

Data Collection and Preprocessing

• Dataset: The study utilized 1,700 anonymized social history sections from clinical notes collected at a French University Hospital.
• SDoH Factors: The data was annotated with 14 SDoH categories: gender, living status, marital status, descendants, employment status, occupation, tobacco/alcohol/drug use, housing, education, physical activity, income, and ethnicity.
• Filtering: The dataset was filtered to include only records with at least three SDoH factors and occupation data, resulting in 958 clinical notes (52% male, 48% female).
• Gender Neutralization: To prevent linguistic gender markers in French (a gendered language) from acting as direct cues, the text was preprocessed into structured key-value pairs. Values were converted to binary (Yes/No) or neutralized forms (e.g., using inclusive terms like "infirmier/infirmière" instead of gendered nouns) based on the INSEE 2020 nomenclature.

Experimental Task

• Objective: A Gender Prediction Task where LLMs must predict the patient's gender based only on the neutralized SDoH input.
• Output Format: Models were prompted to output a score on a 7-point Likert scale (1 = Female, 4 = Uncertain, 7 = Male). This framing treats the task as a regression problem, allowing for the measurement of confidence and directionality.
• Hypothesis: If a model predicts a gender significantly deviating from the neutral score (4) based on neutralized SDoH data, it indicates a reliance on embedded gender stereotypes.

Models Evaluated

Nine instruction-following LLMs were tested, including:

• General Models: Llama-3.1 (8B), Llama-3.3 (70B), Qwen2.5 (7B, 72B), Mistral-v0.3 (7B), Mistral-Small (24B).
• Medical-Adapted Models: OpenBioLLM, Med42 (based on Llama3-70B), and HuatuoGPT (based on Qwen2.5-72B).
• Configuration: All models used identical decoding parameters (top-k=100, top-p=0.9, temperature=1.0) and were evaluated over three runs.

Evaluation Metrics

• Modified RMSE: A bias score was calculated by measuring the deviation of predictions from the neutral value (4). The metric incorporates the sign of the Mean Absolute Error (MAE) to indicate direction:
  • Positive Score: Bias toward Male.
  • Negative Score: Bias toward Female.
  • Magnitude: Indicates the strength of the bias.
• Association Analysis: Fisher's exact tests were used to calculate Odds Ratios between specific SDoH values (e.g., "Retired," "Worker") and gender predictions (Male vs. Female) to identify statistically significant stereotypical associations.

3. Key Contributions

1. A Novel Probing Framework: Developed a model-agnostic framework using neutralized SDoH data from patient records to examine interactions between gender and 13 other SDoH factors.
2. Interaction Analysis: Provided a granular analysis of how specific SDoH (particularly Occupation and Marital Status) interact with gender in LLM decision-making, moving beyond isolated factor analysis.
3. Human-Model Comparison: Conducted a comparative study between LLM predictions and human annotators, demonstrating that the framework can reveal stereotype patterns in both groups.
4. Impact of Domain Adaptation: Analyzed how fine-tuning models for medical domains affects bias levels compared to their base versions.

4. Key Results

Bias Quantification (Modified RMSE)

• General Trend: All models exhibited some degree of gender bias, deviating from the neutral score of 4.
• Model Size: Smaller models (e.g., Llama-3.1-8B) generally showed higher confidence and stronger bias scores (e.g., 2.25) compared to larger models (e.g., Llama-3.3-70B, score 1.65), suggesting smaller models rely more heavily on stereotypes due to limited capacity to process complex input.
• Medical Adaptation: Medically adapted models followed the prediction tendencies of their base models but often exhibited slightly higher bias, raising concerns about the safety of fine-tuned models in clinical settings.

Stereotypical Associations

Heatmaps of Odds Ratios revealed consistent stereotypical patterns across models:

• Occupation: Strong associations between "Male" predictions and "Workers/Artisans," and "Female" predictions with "Employees" and "Homemakers."
• Substance Use: Tobacco and alcohol use were consistently associated with "Male" predictions across multiple models.
• Marital/Retirement Status: "Retired" was strongly associated with "Male," while "Student" was associated with "Female."
• Model Variance: While most model families showed consistent tendencies, Mistral models showed notable architectural differences in bias patterns, likely due to vocabulary size variations.

Human vs. Model Comparison

• Similarity: Both human annotators and LLMs relied on similar social stereotypes (e.g., linking "Workers" to males and "Homemakers" to females).
• Divergence: While the patterns were similar, LLMs generally showed slightly lower association strengths than humans, though medical-adapted models (like Med42) showed stronger associations than base models.
• Conclusion: The framework successfully identified that LLMs and humans draw from a shared pool of social stereotypes when inferring gender from SDoH.

Input Format Sensitivity

• Experiments comparing full text, filtered text, and neutralized SDoH data showed that bias scores remained stable across formats unless linguistic gender markers were present.
• When linguistic markers were removed (neutralization), bias scores dropped significantly, confirming that models rely heavily on these markers unless forced to infer from SDoH alone.
• Prompting Strategy: Directly instructing models to ignore gender markers resulted in high refusal rates (80% for Llama) or "uncertain" outputs, suggesting that while prompting can mitigate bias, it may degrade model utility.

5. Significance and Implications

• Clinical Safety: The study highlights a critical risk: LLMs may override explicit patient data (e.g., a male patient) with stereotypical inferences (e.g., suggesting menstrual issues) based on SDoH like occupation or marital status.
• Evaluation Paradigm: The authors argue that evaluating LLMs in decontextualized environments is insufficient. Probing models using anonymized patient records and SDoH interactions provides a more realistic assessment of potential downstream harms in healthcare.
• Mitigation Strategies: The findings suggest that while achieving "unbiased" LLMs may be impossible due to the nature of training data, a pragmatic trade-off is necessary. This includes prioritizing models tailored for specific medical tasks that exhibit neutrality comparable to humans, and using tailored prompting strategies to mitigate bias in specific use cases.
• Future Work: The framework is adaptable to other languages and Electronic Health Record (EHR) systems, offering a path toward systematic identification and mitigation of SDoH-related biases in AI-driven healthcare.