NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

1. Problem Statement

Neurological disorders (e.g., multiple sclerosis, stroke, brain tumors) require accurate neuroimaging interpretation for diagnosis and clinical decision-making. While traditional deep learning models (e.g., CNNs, U-Nets) excel at specific tasks like segmentation, they lack the ability to integrate imaging evidence with broader clinical context.

Recent Multimodal Large Language Models (MLLMs) offer a transformative potential for clinical reasoning by processing both images and text. However, their reliability, calibration, and operational trade-offs in the specific domain of neuroimaging remain poorly understood. Existing benchmarks often focus on generic Visual Question Answering (VQA) or lack structured clinical output requirements, failing to address critical deployment issues such as:

• Structured Reporting: The need for standardized, machine-readable outputs (JSON) rather than free text.
• Calibration & Safety: The risk of hallucinations and overconfidence in critical medical decisions.
• Operational Efficiency: The high cost and latency of running frontier models in clinical workflows.
• Specificity: The unique challenges of neuroimaging (subtle lesions, spatial heterogeneity) compared to general medical imaging.

2. Methodology: NeuroVLM-Bench

The authors introduce NeuroVLM-Bench, a comprehensive benchmark designed to evaluate 20 frontier MLLMs on 2D neuroimaging data.

A. Dataset Construction

• Source: Curated from publicly available datasets including Brain Tumor datasets, Multiple Sclerosis MRI, Acute Ischemic Stroke (AISD), and Stroke CT datasets.
• Scale: Total of 29,538 samples comprising 18,418 MRI and 11,120 CT scans.
• Classes: Covers five main categories: Brain Tumors (various subtypes), Multiple Sclerosis, Stroke (Ischemic/Hemorrhagic), Other Abnormalities (mimickers like abscesses/cysts), and Normal controls.
• Labels: Includes hierarchical labels for primary diagnosis, detailed subtype, imaging modality, specialized MRI sequence (T1, T2, FLAIR, T1C+), and anatomical plane (Axial, Sagittal).

B. Evaluation Protocol (Four-Phase Tiered Approach)

To ensure fairness and control selection bias, the study employs a progressive filtering pipeline:

1. Phase 0 (Calibration): Locks hyperparameters (Temperature=0.0, Top-p=1.0) and finalizes the prompt schema.
2. Phase 1 (Screening): Evaluates all 20 models on a 30% stratified subset to filter out non-competitive models.
3. Phase 2 (Stability): Evaluates the top 11 candidates on a 45% development split to confirm stability at scale.
4. Phase 3 (Generalization): Final unbiased evaluation of the top 6 models on a held-out 25% test split under Zero-shot and Few-shot (4 examples per class) prompting.

C. Prompting and Output Schema

The benchmark uses a Unified Neuro-Imaging Prompting Protocol (UNIPP). Models are required to output a strict JSON schema containing:

• Image Metadata: Modality, Specialized Sequence, Anatomical Plane.
• Diagnostic Reasoning: Primary Diagnosis, Detailed Diagnosis (Subtype).
• Quantitative Assessment: Diagnosis Confidence score.
• Constraints: No free-text explanations; invalid JSON or missing fields are treated as errors.

D. Multi-Dimensional Performance Metrics (MDPM)

Performance is evaluated across four complementary dimensions:

1. Discriminative Classification: Macro-F1 (abstention-penalized), Weighted F1, Accuracy, Precision, Recall, AUC.
2. Output Reliability: Valid JSON rate and Abstention rate.
3. Statistical Calibration: Expected Calibration Error (ECE) and Brier Score.
4. Operational Efficiency: Token usage, latency, and estimated inference cost.

3. Key Contributions

1. Clinically Grounded Benchmark: The first systematic benchmark for MLLMs in neuroimaging that prioritizes structured, radiology-style reporting over generic VQA.
2. Hierarchical Evaluation: Introduces a rigorous four-phase protocol to filter models and ensure generalization, controlling for selection bias.
3. Multi-Dimensional Analysis: Moves beyond accuracy to evaluate calibration, structured-output validity, and cost-efficiency, providing a holistic view of clinical readiness.
4. Comprehensive Model Comparison: Evaluates 20 models spanning proprietary (GPT-5, Gemini, Claude), open-weight (LLaMA, Qwen), and medical-specialized (MedGemma) architectures.
5. Insights into Failure Modes: Identifies specific limitations, such as the difficulty of fine-grained subtype prediction and the instability of rare class detection.

4. Key Results

A. Performance Hierarchy

• Technical Attributes vs. Clinical Reasoning: Models perform near-perfectly on metadata extraction (Modality, Plane) but struggle significantly with diagnostic reasoning, particularly detailed subtype prediction.
  • Modality/Plane: F1 scores often >0.98.
  • Primary Diagnosis: Moderate performance (Macro-F1 ~0.50–0.60 for top models).
  • Detailed Subtype: Lowest performance (Macro-F1 often <0.35), indicating this is the primary bottleneck.
• Task Difficulty:
  • Tumor Detection: Most reliable task (F1 >0.90 for top models).
  • Stroke: Moderately solvable (F1 ~0.60–0.75).
  • Multiple Sclerosis (MS): Remains highly challenging (F1 ~0.50–0.60) due to subtle, diffuse lesions.
  • Rare Abnormalities: Models consistently fail (F1 near 0) on "Other" classes (mimickers), highlighting safety risks.

B. Model Rankings

• Top Performers: Gemini 2.5 Pro and GPT-5 Chat achieved the strongest overall diagnostic performance.
• Best Balance: Gemini 2.5 Flash offered the optimal trade-off between high diagnostic accuracy, calibration, and computational efficiency.
• Open-Weight Leaders: MedGemma 1.5 4B (medical-specialized) showed the most promising results among open models, approaching the zero-shot performance of proprietary models under few-shot prompting while maintaining perfect JSON validity. LLaMA 4 Maverick also showed moderate success.
• Failures: LLaMA 3.2 Vision variants exhibited near-zero accuracy due to excessive abstention. Qwen 2.5-VL 32B showed a significant performance drop under few-shot prompting.

C. Impact of Few-Shot Prompting

• Performance: Few-shot prompting generally improved Macro-F1 scores for primary diagnosis (e.g., MedGemma 1.5 4B improved significantly).
• Trade-offs: It increased token usage, latency, and inference costs substantially.
• Inconsistency: Gains were not uniform; some models (e.g., Qwen 2.5-VL) degraded, and performance on rare classes or MS did not improve consistently.

D. Calibration and Reliability

• Gemini 2.0 Flash demonstrated the best calibration (lowest ECE and Brier score), indicating more reliable confidence estimates.
• Structured Output: Most top models maintained >99% valid JSON rates, whereas some models (Grok 4, Claude Sonnet 4.5) had lower compliance, posing deployment risks.

5. Significance and Implications

• Clinical Readiness: The study concludes that current MLLMs are not yet ready as general-purpose neuroimaging diagnosticians. They are best suited for assistive roles (e.g., tumor triage, metadata extraction) rather than definitive diagnosis of complex or rare conditions.
• Safety & Reliability: The inability to reliably detect rare abnormalities (mimickers) and the tendency to hallucinate subtypes highlight critical safety gaps. Explicit safeguards and human-in-the-loop workflows are essential.
• Future Directions:
  • Volumetric Reasoning: Current 2D slice-based benchmarks overestimate capabilities; 3D reasoning is required for conditions like MS.
  • Context Integration: Future models must incorporate structured clinical metadata (history, labs) to match real-world diagnostic workflows.
  • Specialist Routing: Instead of a single "general" model, clinical systems should route cases to specialized models based on the specific pathology (e.g., routing stroke cases to models with high vascular sensitivity).
• Policy & Education: The benchmark provides a standardized framework for regulators to assess AI safety and for medical educators to categorize imaging cases for training.

In summary, NeuroVLM-Bench establishes that while MLLMs have mastered technical image attributes, they face significant hurdles in nuanced clinical reasoning. The path forward requires domain-specific adaptation, volumetric analysis, and a shift toward specialized, routed AI systems rather than monolithic generalists.