Evaluation of Medical Vision Language Models HuluMed and MedGemma, and general purpose chatbots Gemma 3, ChatGPT Plus, and Claude Pro on real previously unseen wound images

Technical Summary: Evaluation of Medical Vision-Language Models on Real Wound Images

Problem Statement
Chronic wound assessment is a clinically demanding task requiring the accurate interpretation of wound morphology, tissue composition, vascular characteristics, and infection risk. While Artificial Intelligence (AI) has been explored for medical image interpretation, traditional approaches relying on convolutional neural networks (CNNs) have largely been limited to narrow tasks such as segmentation and tissue classification, lacking the broader reasoning capabilities necessary for complex clinical management. Although recent Vision-Language Models (VLMs) have emerged to bridge visual interpretation with clinical reasoning, it remains unclear whether medically specialized models (fine-tuned for healthcare) consistently outperform general-purpose multimodal systems in the specific domain of wound care. Furthermore, existing benchmarks often rely on curated datasets rather than real-world, previously unseen clinical images with structured management questions.

Methodology
This study evaluated the performance of six VLMs—four open-source/medically specialized models and two proprietary generalist systems—using a curated dataset of 20 real, previously unseen wound images. The dataset, approved by an Institutional Review Board (IRB) and de-identified, encompassed diverse etiologies including diabetic/neuropathic ulcers, ischemic/arterial insufficiency, venous ulcers, surgical wounds, amputation-related wounds, and lymphedema-associated wounds.

The evaluation framework consisted of a structured twelve-question clinical matrix designed to assess both direct visual interpretation and higher-order clinical reasoning. The questions covered:

1. Etiology classification.
2. Vascular assessment and intervention indications.
3. Morphological analysis (fibrinous tissue, granulation coverage).
4. Pathology triage (infection risk).
5. Diagnostic and procedural planning (imaging needs, debridement setting).
6. Prognostic reasoning and advanced management (dressing selection, compression therapy, negative pressure therapy, skin substitutes).

A Clinical Reference Standard was established by a panel of wound-care experts who independently annotated each case with specific textual criteria. Models were tested under zero-shot conditions. Responses were scored using a strict binary framework: a score of 1 was awarded only if the model's output matched the absolute medical criteria defined by the experts. Responses exhibiting hedging, structural ambiguity, or non-committal language were scored as 0, penalizing models that failed to provide clear, actionable decisions.

Key Results
Across 240 clinician-graded decisions (20 cases × 12 questions), a clear performance hierarchy emerged:

• Proprietary Generalist Systems: ChatGPT achieved the highest overall accuracy with 174/240 correct responses (72.50%), followed by Claude with 149/240 (62.08%).
• Open-Source/Medically Specialized Models: HuluMed led this group with 96/240 (40.00%), followed by Gemma 3 (81/240, 33.75%), MedGemma 4B (62/240, 25.83%), and MedGemma 27B (42/240, 17.50%).

Performance Analysis:

• Visual vs. Reasoning: All models performed relatively better on tasks involving direct visual recognition (e.g., identifying necrosis, granulation tissue, or visible infection signs) compared to tasks requiring the synthesis of visual findings into management plans (e.g., selecting imaging modalities, determining debridement urgency, or recommending specific therapies).
• Specialization vs. Generalization: Contrary to the hypothesis that medical fine-tuning would yield superior results, the frontier general-purpose models significantly outperformed the medically specialized alternatives. Notably, neither MedGemma variant outperformed the non-medical Gemma 3 baseline.
• Failure Modes: Medically specialized models, particularly MedGemma, frequently exhibited excessive uncertainty, linguistic verbosity, and etiological confusion (e.g., misclassifying arterial vs. venous causes). They also generated inappropriate management recommendations, such as suggesting debridement before perfusion restoration in ischemic wounds. Claude's lower score relative to ChatGPT was partly attributed to its tendency to use protective, conditional language which was penalized under the strict binary scoring scheme.

Significance and Claims
The authors claim that this study provides a benchmark reflecting real-world wound-care decision support by utilizing real, unseen images and a clinically structured framework that extends beyond diagnosis to include management planning. The primary finding is that current frontier general-purpose multimodal systems demonstrate substantially stronger wound-analysis performance than medically specialized open-source alternatives. This suggests that broad multimodal reasoning capabilities, derived from massive pretraining scales, currently outweigh the benefits of narrow domain-specific fine-tuning for complex wound management tasks.

The paper concludes that while VLMs show promise as assistive tools for documentation and clinical triage, substantial limitations remain in advanced reasoning, procedural planning, and autonomous reliability. Consequently, the authors assert that these systems should currently function as interactive decision-support aids under direct clinician supervision rather than as autonomous diagnostic platforms. The study highlights the need for improved multimodal reasoning and more rigorous evaluation methodologies before clinical deployment.