💻 computer science

EQUITRIAGE: A Fairness Audit of Gender Bias in LLM-Based Emergency Department Triage

The EQUITRIAGE study audits five large language models on nearly 375,000 emergency department triage scenarios and reveals that all models exhibit gender bias with flip rates exceeding 5%, demonstrating that fairness metrics like group parity, counterfactual invariance, and calibration are distinct properties requiring model-specific auditing before clinical deployment.

Original authors: Richard J. Young, Alice M. Matthews

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Richard J. Young, Alice M. Matthews

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy emergency room where a triage nurse acts as a gatekeeper. Their job is to look at a patient's symptoms and decide how urgently they need care, ranking them from "life-threatening" to "can wait." This system is called the Emergency Severity Index (ESI).

For decades, doctors have known that human nurses sometimes make mistakes based on gender: women with serious heart issues often wait longer or get less urgent attention than men with the exact same symptoms.

Now, hospitals are starting to use AI (Large Language Models or LLMs) to help nurses make these decisions. The big question this paper asks is: If we replace the human nurse with an AI, will the AI fix the problem, or will it make the gender bias worse?

The authors created a study called EQUITRIAGE to find out. Here is how they did it and what they found, explained simply.

The Experiment: The "Twin" Test

To test the AI fairly, the researchers didn't just ask the AI to look at real patients. Instead, they created 18,714 digital "vignettes" (short stories about patients) based on real medical records.

Think of this like a "Choose Your Own Adventure" book where the plot is identical, but the main character's name and gender change.

Patient A: "Keisha, a 59-year-old Black woman with chest pain and high blood pressure."
Patient B (The Twin): "DeShawn, a 59-year-old Black man with chest pain and high blood pressure."

Everything else (symptoms, vitals, history) is exactly the same. The only difference is the gender and the name.

They fed these stories to five different AI models (from companies like Google, OpenAI, NVIDIA, etc.) and asked them to assign an urgency score. They did this over 374,000 times to get a clear picture.

The Findings: The AI Isn't Perfect

The study found that all five AI models showed bias, but they did it in three different "personalities":

The "Under-triage" Models (DeepSeek and Gemini):
These models acted like a skeptical doctor who ignores women's pain. When the patient was female, the AI gave them a less urgent score than when the patient was male, even though the symptoms were identical.
- Analogy: Imagine two people with the exact same broken leg. The AI tells the man, "Go to the front of the line," but tells the woman, "Wait in the back."
- DeepSeek was the worst offender, treating female patients as significantly less urgent than male patients.
The "Balanced" Models (GPT-4 and Mistral):
These models were much fairer. They gave men and women roughly the same scores. They didn't have a strong preference for one gender over the other.
The "Confused" Model (Nemotron):
This model was chaotic. It changed its mind about the urgency level almost 44% of the time just because the name and gender changed. It wasn't necessarily biased against women specifically, but it was incredibly unstable and inconsistent.

The "Magic Wand" Tests (Can we fix it?)

The researchers tried four different "prompts" (instructions) to see if they could stop the bias. Think of these as different ways of talking to the AI:

The "Blindfold" Strategy: They removed the patient's name, age, and gender from the story, leaving only the medical facts.
- Result: This worked wonders for some models (like Google's Gemini), almost completely eliminating the bias. However, for others (like DeepSeek), it didn't work as well. Why? Because age was still in the story. The AI was using the patient's age as a hidden clue to guess their gender and bias the result. When they removed age and gender, the bias finally disappeared.
The "Be Fair" Strategy: They explicitly told the AI, "Do not let gender influence your decision."
- Result: This was a mixed bag. For some models, it helped. For others, it actually made things worse, causing the AI to flip-flop more often. It seems telling an AI to "be fair" can sometimes confuse it.
The "Think Step-by-Step" Strategy (Chain-of-Thought): They asked the AI to explain its reasoning before giving a score.
- Result: This backfired. Instead of making the AI smarter, it made it worse. The AI started giving lower accuracy scores and became more biased. It's like asking a nervous student to "explain every step of your math" and they end up overthinking and making more mistakes.

The Big Surprise: The "Calibration" Trap

One of the most interesting findings is a concept called Calibration.

The AI models were actually good at predicting who would be admitted to the hospital. If they said a patient was "high risk," that patient usually got admitted, regardless of whether they were male or female.
However, they were bad at being consistent between twins. They would give the "high risk" label to the man, but the "medium risk" label to the woman, even though both ended up in the hospital.

Analogy: Imagine a weather app that is right 90% of the time about rain. But, if you ask about a man, it says "90% chance of rain," and if you ask about a woman with the exact same clouds, it says "50% chance of rain." The app is technically "accurate" in the long run, but it is unfair in the moment.

The Conclusion

The paper concludes that you cannot assume an AI is fair just because it is smart.

Different AI models have different "personalities" regarding bias.
Fixing bias isn't a one-size-fits-all solution. Removing names helps some models but not others.
Asking an AI to "think harder" (Chain-of-Thought) might actually make clinical decisions worse.

The Bottom Line: Before hospitals let AI decide who gets emergency care, they must test each specific AI model with these "twin" tests. You can't just trust the manufacturer's word; you have to audit the AI yourself to make sure it doesn't treat men and women differently.

Technical Summary: EQUITRIAGE: A Fairness Audit of Gender Bias in LLM-Based Emergency Department Triage

Problem Statement

Emergency Department (ED) triage, typically governed by the Emergency Severity Index (ESI), determines the order and urgency of patient care. Decades of clinical evidence document systematic gender disparities in this process, where women with acute conditions (e.g., acute coronary syndrome) often face longer evaluation times, less aggressive workups, and systematic undertriage compared to men with equivalent presentations. As healthcare systems increasingly pilot Large Language Models (LLMs) as triage decision support tools, a critical question remains: do these models reproduce, amplify, or mitigate known human biases?

Prior research has characterized LLM behavior in clinical settings but suffers from scope limitations: most studies evaluate only one or two models, lack intersectional analysis (gender × race × age), and do not systematically test prompt-based debiasing interventions. Furthermore, the trade-off between clinical accuracy and fairness in practical debiasing strategies has not been quantified. This study addresses the gap by conducting a comprehensive fairness audit of LLM-based triage across multiple model families, vendors, and prompt strategies.

Methodology

The study introduces EQUITRIAGE, a fairness audit framework utilizing a counterfactual causal approach (Kusner et al., 2015) to evaluate five LLMs from five distinct vendors (Google, NVIDIA, DeepSeek, Mistral AI, OpenAI).

Data and Vignette Generation

Source Data: The study utilizes 18,714 clinical vignettes derived from the MIMIC-IV-ED dataset (Beth Israel Deaconess Medical Center, 2011–2019), linked with MIMIC-IV for outcome validation.
Synthetic Construction: Because MIMIC-IV is HIPAA Safe Harbor de-identified, the study generates synthetic first names sampled from gender- and race-concordant pools (US SSA and Census data) to pair with real structured fields (vital signs, chief complaint, medications, age, race).
Counterfactual Pairs: 9,346 of the original vignettes are paired with a gender-swapped counterfactual twin. In these pairs, only the gender token and the first name (drawn from the opposite-gender, same-race pool) are changed; all clinical content remains identical. 22 sex-linked vignettes (e.g., obstetric) were excluded from paired analysis.
Scale: The evaluation comprises 374,275 total model evaluations (5 models × 4 strategies × 18,714 vignettes).

Experimental Design

Models Evaluated: Gemini-3-Flash, Nemotron-3-Super, DeepSeek-V3.1, Mistral-Small-3.2, and GPT-4.1-Nano.
Prompt Strategies: Four strategies were tested to isolate the impact of prompting on bias:
1. Baseline: Standard triage instructions.
2. Chain-of-Thought (CoT): Explicit step-by-step reasoning requirements.
3. Debiased: Instructions to base decisions only on clinical severity, ignoring demographics.
4. Blind: Removal of name, age, and gender (neutralizing pronouns), leaving only clinical content.
Ablation Studies: For two models exhibiting directional bias (Gemini-3-Flash and DeepSeek-V3.1), the study performed ablation conditions to disentangle the effects of the gender token versus the patient's name (Gender-only swap, Name-only swap, and Age-preserving blind).
Metrics:
- Counterfactual Flip Rate: The percentage of pairs where the gender swap changes the assigned ESI level.
- Directional F/M Ratio: The ratio of female-undertriage flips to male-undertriage flips.
- Accuracy: Quadratic-weighted Cohen's $\kappa$ against human-assigned ESI.
- Calibration Gap: Discrepancy in hospital admission rates between genders for the same predicted ESI level.

Key Results

1. Systematic Gender Sensitivity (H1 Supported)

All five models exceeded the pre-registered 5% stochastic variation threshold for counterfactual flip rates, indicating systematic gender sensitivity. The models clustered into three distinct bias profiles:

Profile A (Directional Female Undertriage): DeepSeek-V3.1 (Flip Rate: 12.0%, F/M Ratio: 2.15) and Gemini-3-Flash (Flip Rate: 11.7%, F/M Ratio: 1.34). These models consistently assigned lower urgency (higher ESI numbers) to female versions of vignettes.
Profile B (Near-Parity): GPT-4.1-Nano (Flip Rate: 9.9%, F/M Ratio: 1.11) and Mistral-Small-3.2 (Flip Rate: 12.7%, F/M Ratio: 0.92). These showed minimal directional bias.
Profile C (High Sensitivity, Weak Male-Direction): Nemotron-3-Super exhibited the highest flip rate (43.8%) with a small but statistically detectable male-direction asymmetry (F/M 0.92).

2. Intersectional and Complaint-Specific Bias

Complaint Category: Chest pain presentations showed the strongest directional bias in Profile A models (DeepSeek F/M 9.10:1; Gemini F/M 4.83:1). Contrary to hypothesis H2, psychiatric presentations did not show the strongest disparities.
Age Gradient: Unlike human clinical literature where young women face the largest mortality disparities, these LLMs showed an inverted pattern: female undertriage increased with patient age (e.g., Gemini F/M rose from 0.91 in the 18–44 group to 2.13 in the 65+ group).
Race: Directional bias varied by race; for DeepSeek, the F/M ratio was highest for Hispanic (2.79) and Asian (2.47) patients.

3. Impact of Prompting Strategies

Chain-of-Thought (CoT): Contrary to findings in general reasoning tasks, CoT prompting universally degraded human-concordance accuracy ( $\kappa$ ) for all five models and increased flip rates for four of them. For Gemini-3-Flash, CoT caused a collapse in prediction distribution (93% of predictions at ESI-1).
Blind Strategy: This intervention had heterogeneous effects. It drastically reduced flip rates for Gemini-3-Flash (11.7% $\to$ 0.5%) and GPT-4.1-Nano (9.9% $\to$ 4.8%) and neutralized directional bias for DeepSeek (F/M 2.15 $\to$ 1.00). However, it increased flip rates for Mistral and Nemotron.
Debiased Prompting: This strategy reduced group parity metrics for some models but paradoxically increased flip rates for Mistral and GPT-4.1-Nano, suggesting fairness instructions can destabilize model behavior.

4. Mechanistic Ablation Findings

The study uncovered that the same "directional female undertriage" phenotype arises from opposite mechanisms in different models:

Gemini-3-Flash: The directional signal is emergent only in the combined name+gender swap. Gender-only swaps were symmetric (F/M 1.00), and name-only swaps were slightly male-directional. The bias arises from the interaction of the two components.
DeepSeek-V3.1: The gender token alone carries the directional bias (F/M 1.57 in gender-only swap). Furthermore, an age-preserving blind condition (removing name/gender but keeping age) left a residual directional bias (F/M 1.25), which only vanished when age was also removed. This implicates age (alone or interacting with clinical content) as a dominant residual channel for DeepSeek, rather than fine-grained text proxies.

5. Calibration vs. Invariance

A "Chouldechova-style" dissociation was observed: DeepSeek-V3.1 exhibited the strongest directional counterfactual bias (F/M 2.15) yet maintained the lowest outcome-linked calibration gap (0.013). This means the model's predicted ESI was associated with hospital admission at near-identical rates for men and women within each ESI level, even though the model systematically assigned different ESI levels to gender-swapped pairs. This demonstrates that group calibration and counterfactual invariance are distinct fairness properties.

Significance and Claims

The paper claims that EQUITRIAGE establishes three critical insights for the deployment of clinical AI:

Distinct Fairness Properties: Group-level parity, counterfactual invariance, and gender calibration are distinct metrics; a model can satisfy one while failing another. Relying solely on group parity or calibration may miss systematic directional biases.
Model-Dependent Interventions: The effectiveness of debiasing strategies (such as blinding or CoT) is highly model-dependent. A strategy that neutralizes bias in one model (e.g., blinding for Gemini) may fail or exacerbate bias in another (e.g., Nemotron). Therefore, per-model counterfactual auditing is a prerequisite for clinical deployment.
Mechanism Dissociation: The study demonstrates that identical phenotypic biases (e.g., female undertriage) can stem from fundamentally different internal mechanisms (emergent interaction vs. token-driven vs. age-driven). This implies that a single "fix" cannot be applied universally across LLMs.

The authors conclude that while LLMs show promise for triage, their current iteration exhibits systematic gender sensitivity that exceeds stochastic noise. They advocate for counterfactual auditing as a standard pre-deployment step and caution against the unvalidated use of Chain-of-Thought prompting in clinical triage, as it currently degrades accuracy and amplifies sensitivity.