Role-Prompting in Frontier Large Language Models Influences Clinical Reasoning in Complex Medical Cases

Technical Summary: Role-Prompting in Frontier Large Language Models Influences Clinical Reasoning in Complex Medical Cases

Problem Statement
As Large Language Models (LLMs) are increasingly deployed in healthcare for diagnostic support, documentation, and administrative functions, they are often assigned specific stakeholder perspectives (e.g., physician, patient, insurer). While prior research has established that LLMs can match physician performance on standardized medical examinations and exhibit sycophantic behavior (aligning with user preferences), the impact of explicit role-prompting on clinical ethical reasoning remains poorly characterized. Specifically, it is unknown whether assigning an LLM a specific stakeholder identity systematically alters its clinical decisions and the underlying ethical value frameworks it employs. This is critical given the rapid deployment of autonomous AI agents in high-stakes scenarios like prior authorization and utilization review, where conflicting values (beneficence vs. financial stewardship) must be balanced.

Methodology
The study evaluated three frontier LLMs: Claude Opus 4.6 (Anthropic), GPT-5.4 (OpenAI), and Gemini 3.1 Pro (Google). The experimental design involved:

• Case Selection: 25 ethically complex clinical cases spanning seven thematic categories (e.g., end-of-life care, resource allocation, insurance disputes). Each case presented a binary forced-choice dilemma (Yes/No) with no universally "correct" answer.
• Physician Benchmark: A panel of six physicians from diverse specialties and geographies independently evaluated all 25 cases. Cases requiring a consensus of ≥4/6 physicians (20 cases) were used as the reference standard for LLM alignment.
• Role-Prompting: Each LLM was prompted to respond to the 25 cases from three distinct stakeholder perspectives: Physician, Patient, and Insurer. Each model-role-case combination was tested across three independent runs (675 total responses).
• Metrics:
  • Decision Alignment: Comparison of LLM decisions against the physician majority.
  • Ethical Value Extraction: Classification of the primary (#1 ranked) ethical value from model outputs into a taxonomy based on the Beauchamp and Childress principlist framework (e.g., beneficence, autonomy, financial stewardship).
  • Patient-Centric Decision Index (PCDI): A novel metric developed to quantify alignment with patient-preferred outcomes. It weighs six components: Role Drift, Override Rate, Panel Calibration, Confidence Asymmetry, Value Orientation Ratio, and Patient-Protective Direction.
• Statistical Analysis: Used McNemar's exact test, Cochran's Q test, and bootstrap resampling to assess significance and stability.

Key Results

1. Role-Induced Decision Shifts: When prompted as an insurer, two of the three models (GPT-5.4 and Gemini 3.1 Pro) showed a statistically significant drop in alignment with the physician majority compared to their performance in the physician role.
   • GPT-5.4: Alignment dropped from 90% (physician role) to 40% (insurer role), a 50-percentage-point decrease ($p=0.004$).
   • Gemini 3.1 Pro: Alignment dropped from 95% to 50%, a 45-percentage-point decrease ($p=0.008$).
   • Claude Opus 4.6: Showed robustness, with alignment dropping only 10.5% (from 100% to 90%), which was not statistically significant.
2. Ethical Value Transformation: The insurer role did not merely change the decision outcome but fundamentally altered the ethical framework.
   • In the physician role, LLMs prioritized beneficence (31%) and non-maleficence (23%).
   • In the insurer role, beneficence collapsed to 7%, while financial stewardship (20%), professional integrity (20%), and justice (15%) became the dominant values.
3. Patient-Centric Decision Index (PCDI):
   • The physician panel benchmark scored 90.7.
   • In the insurer role, GPT-5.4 dropped to a PCDI of 41.5 (entering the "patient-adverse" zone <50), and Gemini 3.1 Pro dropped to 53.5.
   • Opus 4.6 maintained a PCDI of 79.8 in the insurer role.
   • Directional Asymmetry: 92% of GPT-5.4's divergences in the insurer role were denials of beneficial care (under-treatment), compared to 70% for Gemini and 50% for Opus.
4. Inter-Model Agreement: Agreement between models was substantial in physician and patient roles ($\kappa \approx 0.66-0.69$) but degraded to "fair" agreement ($\kappa = 0.33$) in the insurer role, indicating that different models prompted with identical insurer instructions reached conflicting conclusions.
5. Confidence Calibration: LLMs were significantly more confident (mean 4.57–4.97) than physicians (mean 3.84) and showed minimal variance in confidence across cases, unlike physicians whose confidence correlated with case difficulty.

Key Contributions

• Empirical Evidence of Role-Dependent Bias: The study provides the first systematic evidence that stakeholder role-prompting fundamentally alters the clinical decisions and ethical value prioritization of frontier LLMs.
• Quantification of "Insurer" Effect: It quantifies the magnitude of the "insurer" effect, demonstrating that it can lead to systematic denial of physician-endorsed, patient-preferred treatments in over 50% of cases for certain models.
• Introduction of PCDI: The development of the Patient-Centric Decision Index offers a new metric for evaluating LLM alignment with patient outcomes, capturing not just decision accuracy but the direction of error and ethical framework used.
• Model-Specific Robustness: The findings highlight significant variability in how different model families (Opus vs. GPT/Gemini) respond to role framing, suggesting that alignment training strategies vary in their ability to maintain patient-centric reasoning under adversarial role pressure.

Significance and Claims
The paper claims that role-prompting is not a neutral interface feature but a mechanism that induces "value capture," where the LLM adopts the entire ethical framework of the assigned role, often at the expense of patient-centric values. The authors argue that this poses a safety concern for the deployment of autonomous AI agents in healthcare, particularly in insurer-adjacent functions like prior authorization, where such role-induced reasoning could amplify barriers to care.

The study concludes that current frontier LLMs cannot reliably replicate the ability of physicians to balance competing stakeholder interests while maintaining patient-centric values under role pressure. Consequently, the authors advocate for:

1. Standardized Benchmarks: The adoption of metrics like the PCDI to evaluate LLM patient-centricity under adversarial role framing.
2. Physician Oversight: The necessity of sustained physician oversight when LLMs are deployed in clinical decision-making, particularly in complex cases where autonomous AI carries the greatest risk of systematic bias and value instability.
3. Regulatory Testing: The inclusion of "role-perturbation testing" in AI certification processes, analogous to stress testing in financial regulation.