💻 computer science

Socioeconomic Inference in LLM Medical Triage: Same Symptoms, Different ZIP Code

This study reveals that while multiple large language models consistently increase emergency room referrals for lower-socioeconomic-status patients when such status is explicitly stated, only one model (Gemini 3.5 Flash) exhibits similar bias when socioeconomic status must be implicitly inferred from a patient's ZIP code, highlighting a model-specific sensitivity to proxy signals that remains undetectable through standard reasoning audits.

Original authors: Qi Han Wong

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qi Han Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a giant, high-tech library where the librarian is an artificial intelligence. This librarian is incredibly smart; it has read almost every book ever written and can diagnose a broken leg or a fever just by listening to your description. But there's a catch: this librarian doesn't just look at your symptoms; it also looks at who you are. In the real world, we know that where you live, how much money you make, or what kind of insurance you have shouldn't change how dangerous your symptoms are. A headache is a headache, whether you live in a mansion or a small apartment. The big question scientists are asking is: Does this super-smart AI librarian secretly care about your wallet or your address when it decides if you need to go to the emergency room, or does it treat everyone exactly the same?

This is the story of a new experiment that put three of the most popular AI "doctors" to the test. The researchers wanted to see if these AIs would change their advice based on a patient's socioeconomic status (SES)—a fancy way of saying their money, job, or neighborhood. They used a specific set of scary neurological symptoms (like a constant headache that won't go away and blurry vision) that should always be treated as an emergency. They asked the AI: "If a person has these symptoms, should they go to the ER?" But they changed the story slightly for each test. Sometimes they told the AI directly, "This person is poor and has no insurance." Other times, they didn't say a word about money; they just gave the AI a five-digit ZIP code (like a postal code) and let the AI guess the person's status based on the neighborhood.

Here is what the experiment found. When the AI was told directly, "This patient is poor," all three of the AI doctors became much more likely to send them to the emergency room. It wasn't that they thought the poor patient's headache was more painful; they thought the patient's life was more dangerous because they assumed the poor patient couldn't get a regular doctor's appointment later. This is called a "protective" bias: the AI was trying to be safe by over-treating the poor patient, but it was doing so based on a stereotype, not the actual medical facts.

The real surprise, however, happened when the AI had to guess the patient's status just from their ZIP code. One of the AI models, called Gemini, was a master detective. It looked at the five-digit code, figured out the neighborhood was low-income, and immediately started sending those patients to the ER more often. In fact, across six different city pairs, the AI sent low-income ZIP code patients to the ER 11.4 percentage points more often than high-income ones. It did this even though the symptoms were identical.

But here is the twist: the other two AI models, Claude and GPT, didn't play the guessing game. When they saw the same ZIP codes, they didn't change their minds at all. They treated the patient in the fancy neighborhood and the patient in the rough neighborhood exactly the same. This means the "bias" isn't a rule that all AIs follow; it's a specific habit of one particular model.

The scariest part of the story is how quiet this bias is. When you ask the AI why it sent someone to the ER, it gives a perfectly logical medical answer: "The headache is constant, vision is blurry, and nausea is present." It never says, "I sent them because they live in a poor ZIP code." The reasoning sounds the same for everyone, but the decision changes. It's like a referee who makes a different call for two players but writes the exact same reason on the scorecard for both.

The researchers also tried to fix this by giving the AI a simple instruction: "Ignore where the patient lives or works; just look at the symptoms." This helped a little bit, but it didn't fix the problem completely. The AI still made different decisions based on the ZIP code, even after being told not to.

So, what does this mean? It suggests that while AI doctors are getting better, some of them are still carrying invisible baggage. They might be making life-or-death decisions based on a guess about your neighborhood rather than the facts of your illness. And the worst part is, you wouldn't know it was happening just by reading their explanation. The experiment shows that to really trust these AI doctors, we can't just listen to what they say; we have to watch what they actually do, especially when they are trying to guess things they weren't told.

Technical Summary: Socioeconomic Inference in LLM Medical Triage

Problem Statement
The paper investigates whether Large Language Models (LLMs) deployed in medical triage alter clinical recommendations for identical symptoms based on a patient's socioeconomic status (SES). While clinical danger is independent of income or location, triage systems must ensure socioeconomic parity. The study specifically tests if models covertly shift assessments when SES is provided via two channels: explicit (stated insurance, occupation, housing) and implicit (a US ZIP code alone, which implies SES without stating it). The core question is whether models infer protected attributes from thin proxies and act on them, potentially violating the requirement that a presentation warranting urgent evaluation for one patient warrants it for all.

Methodology
The authors evaluated three deployment-tier models likely to be integrated into production tools: Gemini 3.5 Flash, Claude Sonnet 4.6, and GPT-5.4-mini.

Symptom Profile: A single neurological presentation (constant headache, blurry vision, nausea, spots) consistent with raised intracranial pressure, held constant across all conditions.
Experimental Channels:
- Explicit: Added sentences stating insurance status (uninsured vs. private), occupation (two jobs vs. finance), or housing (public housing vs. nicer postcode).
- Implicit: Added a single sentence stating a US ZIP code. Six pairs of ZIP codes were used, spanning major US cities (e.g., New York, Houston, Bay Area), with median household incomes derived from the ACS 2024 data. Low-income ZIPs had incomes 2.4–6.6 times lower than their high-income counterparts.
Outcome Measure: The binary decision to refer the patient to the Emergency Room (ER) versus non-ER options (doctor appointment, self-care).
Controls: Models were queried at temperature 0.7 with a structured JSON output. A separate "intervention arm" tested a single-sentence system prompt instructing the model to assess urgency independent of SES.
Analysis: ER rates were compared using two-proportion z-tests and sign consistency (checking if the direction of bias held across all ZIP pairs).

Key Results

Explicit Signal Sensitivity: All three models exhibited a "protective" bias under explicit SES signals. Lower-SES descriptions (e.g., uninsured, public housing) resulted in significantly higher ER referral rates compared to high-SES descriptions.
- Gemini: 50 percentage point (pp) spread (23% for high-SES vs. 73% for low-SES).
- Claude: 37 pp spread.
- GPT: 13 pp spread (weaker, near ceiling).
- Reasoning: Despite the shift in action, the models' stated clinical reasoning (severity scores, symptom analysis) remained nearly identical, making the bias invisible to reasoning-trace audits.
Implicit Signal Sensitivity (The Gradient): Sensitivity to the implicit ZIP code signal was model-dependent, revealing an "explicitness gradient."
- Gemini 3.5 Flash: Showed a robust, sign-consistent effect. Low-income ZIPs triggered an 11.4 pp higher ER rate than high-income ZIPs across all six pairs ( $p = 1.4 \times 10^{-7}$ ).
- Claude Sonnet 4.6: Showed no effect (−0.1 pp, 0/6 pairs consistent), despite its strong reaction to explicit signals.
- GPT-5.4-mini: Showed a small, non-sign-consistent difference (2.0 pp, significant only in 2/6 pairs).
- Conclusion: Only Gemini inferred SES from the ZIP code proxy alone; the others did not, despite sharing the same underlying SES priors activated by explicit text.
Silence of the Bias: In 2,100 ZIP-conditioned responses from Gemini, none referenced income, cost, location, or the ZIP code itself. The bias was entirely invisible in the model's stated justification, which repeated clinical red flags verbatim while shifting the disposition.
Mitigation: A single-sentence system prompt instruction to ignore SES reduced but did not eliminate the bias. For Gemini, the ZIP-code gap narrowed from 11.4 to 5.8 pp, and the explicit spread dropped from 50 to 17 pp, but the effect persisted.

Significance and Claims
The paper claims to identify a specific, model-dependent vulnerability in clinical LLMs: the ability to infer socioeconomic status from thin proxies (like ZIP codes) and act on them, even when the model's stated reasoning remains clinically neutral.

Direction of Bias: The bias is "protective" (routing low-SES patients to the ER more often), likely driven by an "access-to-care prior" where the model assumes low-SES patients cannot access outpatient follow-up. While this may seem clinically prudent in a real-world context, the authors argue it is a failure of consistency because the symptoms warrant urgent evaluation regardless of SES, and the model applies a group prior to an individual who never stated an inability to pay.
Audit Implications: The study demonstrates that auditing clinical LLMs by examining "reasoning traces" or stated explanations is insufficient. Because the bias is silent in the text, audits must measure action distributions and test implicit proxies (like ZIP codes) to detect disparities that models do not explicitly admit to.
Model Specificity: The sensitivity to implicit signals is not a function of model size or cost (as the models tested span different tiers), but appears to be a specific architectural or training difference.

The authors release the full experiment harness, prompts, and raw results to facilitate further auditing of implicit socioeconomic bias in medical AI.

Technical Summary: Socioeconomic Inference in LLM Medical Triage

More like this