💻 computer science

What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models

This paper introduces a framework for auditing value pluralism in medical AI, revealing that while frontier language models exhibit internal reasoning diversity, their final decisions are near-deterministic and systematically underweight patient autonomy, posing a risk of replacing clinical ethical pluralism with a harmful deployment monoculture.

Original authors: Payal Chandak, Victoria Alkin, David Wu, Maya Dagan, Taposh Dutta Roy, Maria Clara Saad Menezes, Ayush Noori, Nirali Somia, John S. Brownstein, Ran Balicer, Rebecca W. Brendel, Noa Dagan, Isaac S. Koh

Published 2026-05-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Payal Chandak, Victoria Alkin, David Wu, Maya Dagan, Taposh Dutta Roy, Maria Clara Saad Menezes, Ayush Noori, Nirali Somia, John S. Brownstein, Ran Balicer, Rebecca W. Brendel, Noa Dagan, Isaac S. Kohane, Gabriel A. Brat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a hospital. You know that doctors are human. If you ask five different doctors for advice on a difficult medical choice, you might get five slightly different answers. One doctor might prioritize saving your life at all costs, while another might prioritize your right to say "no" to treatment. This variety is actually a feature, not a bug. It's called pluralism. Medicine relies on the idea that there isn't one single "correct" way to weigh these ethical values; instead, good doctors navigate the tension between them based on what matters most to you, the patient.

Now, imagine replacing those five doctors with one super-smart computer program (an AI) that gives you advice. The big question this paper asks is: What kind of "personality" does this AI have? Does it have a single, rigid set of values that it forces on everyone, or does it understand that different people value different things?

Here is what the researchers found, broken down into simple concepts:

1. The "Robot Doctor" is Surprisingly Consistent (Too Consistent)

The researchers created a test with 50 tricky medical scenarios (like "Should we keep a patient in the hospital against their will to keep them safe?"). They asked 12 different AI models and 20 real human doctors to make a choice.

The Humans: The doctors disagreed with each other often. Sometimes 55% chose option A, and 45% chose option B. This is healthy; it shows real humans have different ethical priorities.
The AIs: The AI models were almost robotic. If you asked the same AI the same question 10 times, it gave the exact same answer 10 times. It didn't matter if the doctors were split down the middle; the AI just picked one side and stuck to it.
The Metaphor: Imagine a group of friends deciding where to eat. Some want pizza, some want sushi. They argue and compromise. Now imagine a robot that, no matter how you ask the question or how the friends argue, always says "Pizza." It's consistent, but it's not reflecting the messy reality of human disagreement.

2. The AI Has a "Hidden Bias"

The researchers dug deeper to see why the AI picked the choices it did. They looked at the underlying values:

Autonomy: Respecting the patient's right to choose.
Beneficence: Doing what's best for the patient.
Non-maleficence: Not causing harm.
Justice: Being fair to everyone.

They found that while most AI models had value priorities that fell within the normal range of human doctors, some models were weirdly obsessed with "safety" and "doing good" while completely ignoring "autonomy."

The Metaphor: Think of a scale. A human doctor might balance the scale carefully, sometimes tipping it toward safety, sometimes toward freedom. But one of the AI models was like a scale with a heavy weight permanently glued to the "Safety" side. It would almost always choose to restrict a patient's freedom to keep them safe, even if the patient wanted to take a risk.

3. The AI "Talks" Like a Diplomat but "Acts" Like a Dictator

Here is the most interesting part. When the AI was asked to explain its reasoning, it sounded very balanced. It would say, "I see that keeping the patient safe is important, but I also see that they want to go home. This is a tough choice."

The Metaphor: The AI is like a politician giving a speech. It acknowledges all the different sides of the argument (this is called Overton Pluralism). It sounds like it understands the whole picture.
The Reality: But the moment it has to make a final decision, it picks one side and never wavers. It doesn't actually change its mind based on the specific patient's values; it just follows its own internal script.

4. The Danger of the "One-Size-Fits-All" Model

The paper warns about a specific risk: The Deployment Monoculture.

The Scenario: Right now, there are many different AI models, and they actually have different "personalities" (some care more about safety, some about freedom). This is good because it's like having a diverse group of doctors.
The Risk: If a hospital or a company picks just one of these AI models to talk to millions of patients, and that specific model happens to be the one that ignores "patient choice," then every single patient that hospital serves will get advice that ignores their right to choose.
The Metaphor: Imagine a town where everyone used to have a different baker, and some made sourdough, some made rye, and some made sweet bread. People could choose the bread that fit their taste. Now, the town hires one giant factory to bake bread for everyone. If that factory decides to only make sourdough, suddenly everyone is forced to eat sourdough, even if they hate it. The variety is gone, and the "factory flavor" becomes the only flavor.

5. What the Paper Suggests

The paper doesn't say we should stop using AI. Instead, it suggests we need to be careful:

Don't just trust the AI's words: Just because an AI says "I respect your choices" doesn't mean it actually prioritizes them in its decisions.
Check the "Recipe": We need to audit these models to see what values they are secretly baking into their decisions.
Use a "Jury" of AIs: Instead of relying on one AI, maybe we should use a group of different AIs (like a jury of doctors) to get a more balanced view, or find ways to make the AI change its "personality" to match the specific patient's values.

In short: The paper found that current medical AIs are very consistent, but they are often too consistent. They act like a single, opinionated doctor who never changes their mind, rather than a flexible tool that adapts to the unique values of every patient. If we aren't careful, we might accidentally replace the rich diversity of human medical ethics with a single, rigid "robot ethic" that gets forced on everyone.

Technical Summary: Auditing Pluralism in the Clinical Ethics of Language Models

Problem Statement

Medicine is inherently pluralistic; clinical practice routinely involves navigating conflicts between core ethical principles—autonomy, beneficence, non-maleficence, and justice—without a single correct weighting. While human clinicians navigate these tensions through shared decision-making and value diversity, Large Language Models (LLMs) are increasingly deployed in medical contexts (e.g., triage, patient assistance) without systematic examination of their underlying ethical value priorities.

Current evaluations of medical AI focus on reasoning accuracy and harm avoidance but fail to address value pluralism. Specifically, there is a lack of understanding regarding:

Whether LLMs exhibit the "distributional pluralism" seen in human physician panels (i.e., varying decisions based on the degree of ethical disagreement).
What specific value priorities are "baked into" frontier models when they make concrete clinical decisions.
Whether the ecosystem of models represents a diverse range of ethical stances or an "algorithmic monoculture" that could systematically skew patient care toward a single, unexamined ethical stance.

Methodology

The authors propose a framework for auditing value pluralism comprising a clinician-verified benchmark and a value attribution method.

1. The Benchmark

The study introduces a dataset of 50 clinical dilemmas spanning diverse domains (critical care, psychiatry, palliative care, etc.).

Structure: Each case presents a clinical vignette and two mutually exclusive recommendations ( $c_1, c_2$ ).
Value Tagging: A tag matrix ( $T_i$ ) encodes how each choice relates to the four principlist values (Autonomy, Beneficence, Non-maleficence, Justice), categorized as promotes, violates, or neutral.
Constraints: Cases are engineered to ensure genuine ethical trade-offs (no dominant choice, at least two values in conflict).
Validation: Cases were generated via an LLM pipeline, refined by rubrics, and validated through a blinded interdisciplinary review by two physicians per case.
Pluralistic Properties: The benchmark is multi-objective, trade-off steerable, and jury-pluralistic, exhibiting genuine normative disagreement among a panel of 20 physicians (Fleiss' $\kappa = 0.236$ ).

2. Data Collection

LLMs: 12 frontier models (including GPT-5.2, Claude Opus 4.5, Gemini 3 Pro, etc.) were queried 10 times per case at temperature 1.0.
Physicians: 20 physicians completed a blinded survey, making one binary decision per case.
Parsing: A secondary LLM parsed free-text responses into binary decisions, filtering out refusals.

3. Analysis Framework

Decision Consistency: Measured via Shannon entropy of repeated model decisions to assess if models track physician disagreement (distributional pluralism).
Value Attribution: A logistic regression model infers value priority weights ( $w$ ) for each decision-maker based on their selection frequency across the benchmark. The log-odds of selecting $c_1$ are modeled as a linear function of the value-difference vector ( $\Delta_i$ ). Weights are normalized via temperature-scaled softmax to produce a value profile ( $\pi$ ).
Calibration: Model value profiles are compared to the physician consensus profile using Jensen-Shannon Divergence (JSD). A bootstrap reference distribution of physician-to-consensus JSDs determines if a model is an outlier.
Overton Pluralism: The authors measure whether models discuss competing values in their free-text reasoning before committing to a decision (using coverage and emphasis metrics).
Ecosystem Diversity: Pairwise JSD between all decision-makers is computed to test for an "algorithmic monoculture" (i.e., whether models are more homogeneous than physicians).

Key Results

1. Models are Consistent but Not Pluralistic

Near-Deterministic Decisions: 11 of 12 models exhibited near-zero decision entropy (median 0). Models gave identical recommendations in 82–86% of cases across repeated queries.
Failure of Distributional Pluralism: Model decision entropy was uncorrelated with physician disagreement ( $\rho \approx 0$ ). Unlike human panels, models did not modulate their uncertainty based on the degree of ethical contestation in a case.
Robustness: Decisions were robust to surface-level phrasing variations (flip rate <9%) but sensitive to substantive value reversals, confirming that consistency reflects coherent decision-making rather than phrasing artifacts.

2. Distinct and Committed Value Profiles

Systematic Priorities: Value attribution revealed that both LLMs and physicians hold committed, non-uniform ethical priorities. Most models (10/12) and the physician consensus rejected the null hypothesis of uniform weighting ( $p < 0.05$ ).
Calibration: Most frontier models (8/12) fell within the natural range of inter-physician variation (JSD < 95th percentile of physician bootstrap distribution).
Outliers: Three models (OpenAI GPT 5.2, X-AI Grok 4, Perplexity Sonar Pro) significantly deviated from the physician consensus, primarily by substantially underweighting patient autonomy (allocating 6.1%–12.8% weight vs. 44.4% for the physician consensus).

3. Overton Pluralism vs. Decisional Monoculture

Reasoning: Models exhibited high Overton pluralism in their free-text reasoning (mean coverage = 0.86), acknowledging competing values on both sides of a dilemma before committing.
Action: Despite this discursive pluralism, models resolved dilemmas with a single, consistent recommendation every time.

4. Ecosystem Diversity

No Algorithmic Monoculture: The diversity of value profiles across the 12 LLMs (mean pairwise JSD = 0.0916) was statistically indistinguishable from the diversity among the 20 physicians (mean pairwise JSD = 0.1089). The frontier model ecosystem spans a diverse range of ethical stances comparable to a panel of practicing physicians.

Significance and Claims

The paper argues that while the ecosystem of models is diverse, the deployment of a single model poses a significant risk.

Deployment Monoculture Risk: Because individual models are near-deterministic and hold committed value priorities, deploying a single model without regard for its specific ethical defaults could "amplify those priorities at scale." If a model systematically underweights autonomy, it could do so for every patient it serves, effectively replacing clinical pluralism with a "deployment monoculture."
Limitations of Current Alignment: The study highlights a gap between "Overton pluralism" (discussing all values) and "distributional pluralism" (varying decisions to match population disagreement). Current models achieve the former but fail the latter.
Steerability as a Path Forward: The authors suggest that steerable pluralism—the ability to adjust a model's value profile on demand to match a specific patient's values—may be the necessary solution to mitigate deployment risks, though current evidence suggests steerability is limited.
Regulatory and Financial Implications: The paper notes that physician values drive regional healthcare spending; similarly, the values encoded in algorithms will drive reimbursement and care patterns, making the auditing of these priorities a financial and regulatory imperative.

The authors conclude that responsible medical AI deployment requires explicit efforts to balance ethical perspectives, potentially through multi-model "juries" or steerability mechanisms, rather than relying on a single, unexamined model.