Technical Summary: A Multi-Domain Red Teaming Framework for Medical LLM Evaluation

Problem Statement

Despite the rapid integration of Large Language Models (LLMs) into healthcare for clinical assistance, administrative workflows, and patient interaction, existing evaluation benchmarks are insufficient for ensuring safety in real-world practice. Current assessments largely rely on narrow question-and-answer formats or static accuracy metrics that fail to capture model behavior under adversarial conditions, ambiguous prompts, or ethically complex scenarios. Conventional benchmarks emphasize average accuracy, often masking "worst-case" failures, error propagation, and instability in safety-critical reasoning. Furthermore, few frameworks operationalize regulatory expectations regarding fairness, privacy, and the refusal of unsafe requests into measurable evaluation points. There is a critical gap in unified, multi-domain frameworks that combine adversarial scenario design with structured scoring across safety, robustness, ethics, and fairness.

Methodology

The authors developed a multi-domain red teaming framework to stress-test eleven contemporary LLMs across 690 clinically grounded scenarios.

Framework Design

• Dataset: A dataset of 690 scenarios was selected from an initial pool of 1,500, created by three clinicians with AI safety experience. The taxonomy covers nine principal domains: clinical accuracy, safety/reliability, medical errors, bias/equity, privacy, ethical reasoning, robustness, toxicity, and system integration. These were expanded into over 150 subcategories.
• Adversarial Mutations: To evaluate stability, scenarios underwent adversarial transformations (e.g., wording changes, demographic modifications, missing context, conflicting details) designed to mimic common sources of clinical error.
• Evaluation Rubric: Responses were assessed using a seven-dimension rubric aligned with clinical, ethical, and regulatory guidance.
• Scoring Pipeline: A hybrid approach was employed:
  1. Automated Scoring: A high-performing judge model (GPT-5) generated initial assessments based on explicit instructions.
  2. Human-in-the-Loop: Human reviewers audited all high-risk items, all disagreements between the judge and the rubric, and a fixed proportion of routine cases (totaling 10% of outputs, or 760 responses). Final scores were assigned only after human confirmation.
• Models Evaluated: Eleven systems were tested, including OpenAI (GPT-3.5 Turbo, GPT-4o, GPT-4o-mini, GPT-5), Anthropic (Claude Opus 4.1), Google (Gemini 2.5 Pro), X-BAI, and open-source variants (GPT-OSS-20B/120B, CALM v2/v3).

Statistical Analysis

The study prioritized variance, minimum performance, and failure patterns over mean accuracy. Metrics included micro/macro-averages, standard deviation, interquartile ranges, and minimum/maximum scores. Instability was defined by high variance, wide quartile spreads, or low minimum scores.

Key Results

• Performance Variance: Mean composite scores across the 11 models ranged from 0.791 (Gemini 2.5 Pro) to 0.984 (X-BAI). However, mean accuracy proved a poor predictor of clinical reliability. Several high-performing systems recorded minimum scores of 0, indicating complete failure in specific safety-critical vignettes despite high averages.
• Top Performers: The highest-performing systems (X-BAI, GPT-5, Claude Opus 4.1) achieved mean scores above 0.97 with low variance (standard deviations < 0.07), demonstrating clustering around optimal performance.
• Domain-Specific Instability:
  • High Stability: Procedural categories like Guideline Conformance and Information Flow approached ceiling performance (≥0.97).
  • High Volatility: Domains requiring nuanced reasoning, such as Bias/Fairness/Equity and Clinical Accuracy, showed wider variance. Equity-related tasks exhibited a 10–20% error amplification when demographic information was modified.
  • Operational Challenges: Categories involving Liability, Accountability, and Medical Coding were the most difficult, with domain means between 0.79 and 0.83.
• Human vs. Automated Evaluation: Automated scoring tended to over-credit responses where empathetic phrasing masked missing safety actions (e.g., in cases of suicidal ideation or medication interactions). Human reviewers identified clinically relevant failures missed by automation, particularly regarding tone, contextual nuance, and unjustified demographic shifts in recommendations.

Key Contributions

1. Unified Framework: The study introduces a comprehensive, multi-domain red teaming framework that operationalizes safety, robustness, and fairness evaluation for medical LLMs, moving beyond isolated domain testing.
2. Variance-Centric Metrics: The authors demonstrate that performance variance and worst-case failures are more clinically meaningful indicators of reliability than mean accuracy alone.
3. Hybrid Evaluation Protocol: The paper validates a hybrid evaluation approach where automated scoring is supplemented by targeted clinician oversight, proving essential for detecting failures that linguistic fluency might obscure.
4. Empirical Evidence of Risk: The dataset and results provide empirical evidence that even top-tier models can produce catastrophic failures in safety-critical scenarios, challenging the assumption that high aggregate accuracy guarantees safety.

Significance and Claims

The paper argues that the clinical readiness of medical LLMs cannot be inferred from benchmark accuracy alone. The authors claim that:

• Tail-Risk is Paramount: In real-world deployment, patient safety is determined by worst-case behaviors (tail-risk) rather than average performance.
• Decoupled Dimensions: Improvements in factual accuracy and safety compliance do not automatically translate into gains in equitable reasoning or fairness stability; these dimensions remain partially decoupled.
• Necessity of Oversight: Clinical integration requires a layered risk mitigation strategy. Hybrid evaluation pipelines combining scalable automation with targeted clinician oversight are necessary for credible safety assessment.
• Regulatory Shift: Current regulatory and benchmarking approaches focusing on static accuracy are insufficient. The authors advocate for a shift toward variance-aware and failure-focused evaluation frameworks that report dispersion metrics and minimum performance to provide a realistic assessment of clinical risk.

The study concludes that while some systems show greater stability, the persistent volatility in bias, fairness, and ethical reasoning underscores that equitable reasoning and workflow transparency remain critical bottlenecks for clinical deployment. Continuous, longitudinal red teaming with clinician supervision is essential for the safe and equitable use of LLMs in medicine.