Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in Dermatology

Technical Summary: Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation

Problem Statement
While multimodal Large Language Models (MLLMs) have demonstrated promising performance on public dermatological benchmarks, critical uncertainty remains regarding their generalizability to real-world clinical workflows. Existing evaluations rely predominantly on curated, single-lesion images (often dermoscopic) captured under standardized conditions and are linked to multiple-choice or category-based classification tasks. In contrast, real-world dermatological consultations involve heterogeneous inputs: clinical photographs captured with variable lighting and image composition (often without dermoscopy), paired with brief, potentially incomplete or misleading consultation notes from referring physicians. This study addresses the "bench-to-bedside" gap, questioning whether current MLLMs can reliably generate open-ended differential diagnoses and perform severity-based triage under these realistic, uncurated conditions.

Methodology
The authors evaluated five MLLMs—four open-weight models (InternVL-Chat v1.5, LLaVA-Med v1.5, SkinGPT-4, MedGemma-4B-Instruct) and one commercial model (GPT-4.1)—in two distinct evaluation environments:

1. Public Benchmarks: Three standard datasets (DermNet, Fitzpatrick17k, SCIN) comprising approximately 46,500 images.
2. Real-World Cohort: A retrospective, multicenter, hospital-based dermatological consultation dataset with 5,811 cases and 46,405 clinical images, collected between 2014 and 2023. This cohort included paired clinical photographs and consultation notes from referring physicians. Notably, this internal cohort was enriched for severe cases (44.0% compared to approximately 19% in public datasets).

The models were assessed based on two clinically relevant tasks:

• Open Differential Diagnosis: Generation of a top-3 list of possible diagnoses.
• Severity-Based Triage: Identification of cases requiring urgent evaluation.

Performance was measured using:

• Automated Metrics: BERTScore for semantic similarity and an "LLM-as-a-judge" pipeline to verify whether the correct diagnosis appeared in the top-3 predictions and to assign severity.
• Human Evaluation: A subset of outputs was reviewed by three board-certified or board-eligible dermatologists to assess alignment with clinical intent, factual accuracy, and the specificity of descriptions of visual features.

The study systematically varied inputs to test robustness: image only versus image plus clinical context, and the presence of a "leading" context (where the referring physician suggests a specific diagnosis) versus a non-leading context.

Key Findings

1. Significant Bench-to-Bedside Gap: Diagnostic performance declined substantially as models transitioned from public benchmarks to the real-world consultation cohort. On public datasets, the best open-weight model (MedGemma-4B) achieved a top-3 accuracy of 26.55%, and GPT-4.1 reached 42.25%. In the real-world setting with images only, these figures dropped to 13.35% and 24.65%, respectively.
2. Context Dependency and Fragility: The inclusion of clinical context (referring notes) improved accuracy across all models (e.g., GPT-4.1 rose to a top-3 accuracy of 38.93%). However, this integration led to significant vulnerability. Models were highly sensitive to misleading or incomplete context; in non-leading contexts, adding notes improved performance for some models but degraded it for others (degradation rates up to 84.2% for SkinGPT-4). Models frequently over-anchored to erroneous physician suggestions rather than correcting them based on visual evidence.
3. Visual Anchoring as the Primary Bottleneck: Review by expert dermatologists revealed that diagnostic errors were not primarily due to deficits in downstream medical reasoning or general knowledge. Instead, the primary differentiating factor between correct and incorrect answers was visual specificity. Correct answers consistently described precise, diagnosis-relevant morphological features, whereas incorrect answers, despite otherwise factual language, provided generic descriptions.
4. Triage Performance: While exact diagnosis remained challenging, models demonstrated moderate sensitivity (exceeding 60% for some models with context) in identifying severe cases, suggesting potential utility as a screening adjunct, although reliability for autonomous deployment remains insufficient.

Implications and Claims
The work argues that current benchmark evaluations significantly overestimate the clinical maturity of dermatological MLLMs. The authors claim that the primary barrier to safe deployment is not the models' reasoning capabilities, but rather:

• Fragile Context Integration: Models lack robustness against the noisy, incomplete, and sometimes misleading nature of real-world clinical notes.
• Insufficient Visual Anchoring: Models fail to extract and prioritize specific dermatological visual features under realistic imaging conditions (variable lighting, multiple lesions, background distractions).

The study concludes that future advances in dermatological MLLMs must prioritize realistic, workflow-based evaluation, robustness against misleading context, and stronger dermatology-specific learning of visual representations, rather than focusing exclusively on improving reasoning capabilities. The authors emphasize that current models are not ready for autonomous diagnostic use in consultative dermatology and that human supervision remains indispensable for triage due to context sensitivity.