Performance and Failure Patterns of Multimodal Large Language Models for Hepatocellular Carcinoma Detection on Portal Venous Phase CT

Problem Statement

While multimodal large language models (LLMs) have demonstrated proficiency in language-based tasks and clinical reasoning benchmarks, their capability to interpret medical imaging remains inconsistent. Specifically, the detection of hepatocellular carcinoma (HCC) on computed tomography (CT) requires subtle visual discrimination, spatial reasoning, and the integration of clinical context. Previous studies suggest that earlier models (e.g., GPT-4) struggle with identifying common liver diseases on CT, often performing significantly worse than trained radiologists. There is a lack of standardized benchmark studies that directly compare newer, advanced multimodal LLMs against human readers under controlled, identical conditions using real clinical images.

Methodology

The study employed a retrospective design to evaluate the diagnostic performance of current multimodal LLMs against human experts in a constrained visual task.

• Dataset: The study utilized 211 single-slice portal venous phase CT images: 106 images containing pathologically confirmed HCC lesions (from the HCC-TACE-SEG dataset) and 105 images of normal livers (from the CPTAC-UCEC dataset).
• Image Standardization: To prevent dataset leakage and shortcut learning, all images were converted to standardized PNG format (512 × 512 pixels) with fixed window settings (WW 400, WL 50). All DICOM metadata, institutional identifiers, and annotations were removed. No segmentation masks or lesion markers were provided.
• Models Evaluated: Four multimodal LLMs were tested under zero-shot conditions (no fine-tuning):
  • ChatGPT-4o (December 2024)
  • ChatGPT-4.5 (February 2025)
  • Perplexity AI (configured with GPT-4.1, May 2025)
  • ChatGPT-5.1 (December 2025)
• Prompting Strategy:
  • Run 1 (Baseline): All models received a standardized prompt without additional context.
  • Run 2 (Anatomical Guidance): ChatGPT-5.1 was re-evaluated with a specific "anatomical orientation" prompt added to the first image input, instructing the model to focus on the liver's location relative to the lung and the image orientation. Subsequent images in the session followed without this specific cue.
• Human Readers: A senior radiologist (15 years experience) and an abdominal imaging fellow served as the human control group, receiving the same static images and task instructions.
• Evaluation: Outputs were scored as binary decisions (focal lesion present vs. absent). Performance metrics (sensitivity, specificity, accuracy) were calculated, and statistical comparisons were made using McNemar's test with Bonferroni correction. Error analysis was conducted on misclassified cases to identify failure patterns.

Key Results

• Human Performance: Both human readers outperformed all multimodal LLMs. The senior radiologist achieved 100% sensitivity and 91.5% accuracy. The fellow achieved 98.1% sensitivity and 91.0% accuracy.
• Best Performing LLM: ChatGPT-5.1 with the anatomical guidance prompt (Run 2) was the top-performing model, achieving:
  • Sensitivity: 79.2%
  • Specificity: 87.6%
  • Accuracy: 83.4%
  • This performance was significantly higher than all other LLMs tested (p < 0.05) but remained significantly lower than both human readers (p < 0.05).
• Impact of Prompt Engineering: Adding a single anatomical orientation prompt to ChatGPT-5.1 resulted in significant improvements over its baseline (Run 1):
  • Sensitivity increased by 12.2 percentage points (67.0% → 79.2%).
  • Specificity increased by 6.6 percentage points (81.0% → 87.6%).
  • Accuracy increased by 9.5 percentage points (73.9% → 83.4%).
• Model Variability: Performance varied significantly across models. ChatGPT-4.5 had high sensitivity (79.2%) but poor specificity (50.5%). ChatGPT-4o had high specificity (90.5%) but low sensitivity (43.4%).

Failure Pattern Analysis

The study identified distinct error modes for both the best-performing LLM and human readers:

• False Negatives (Missed HCC): The primary cause for the LLM was isoattenuating lesions (lesions with density similar to surrounding liver tissue), accounting for 36.4% of LLM errors. Other causes included large heterogeneous lesions, lesions at the liver edge/fissure, and lesions mimicking gastrointestinal structures. Notably, one isoattenuating lesion was missed by both the LLM and the fellow.
• False Positives (Normal Livers flagged as HCC): The most common cause for both LLMs and humans was vascular misidentification (e.g., cross-sections of portal veins or hepatic veins mistaken for lesions).
  • LLMs showed fewer false positives due to technical artifacts (partial volume averaging) compared to humans.
  • However, the LLM exhibited AI-specific errors, such as over-reliance on density thresholds, which did not correlate with human reasoning patterns.

Significance and Claims

The authors conclude that while multimodal LLMs are rapidly improving, they currently lack the visual lesion detection capability required for independent clinical diagnosis of HCC on single-slice CT images.

• Current State: The best-performing model (ChatGPT-5.1 with prompt augmentation) achieved high specificity comparable to radiologists but significantly lower sensitivity.
• Prompt Sensitivity: The study demonstrates that LLM performance in medical imaging is highly sensitive to prompt contextualization and anatomical orientation. Simple guidance can yield substantial performance gains.
• Role in Clinical Workflow: The paper argues that these models should not be viewed as autonomous diagnostic tools. Instead, they are best regarded as research tools or adjuncts for hypothesis generation and education, requiring meticulous prompt design, rigorous evaluation, and strict human supervision.
• Limitations: The authors acknowledge that the study was a "stress test" using single-slice, single-phase images, which does not reflect the multi-phase, multi-slice nature of real-world HCC diagnosis. Consequently, the results likely underestimate the potential performance of both humans and AI in a full clinical workflow.

The paper emphasizes that summary performance metrics alone are insufficient; understanding specific failure patterns (such as isoattenuation and vascular confusion) is essential for future development and safe integration into radiology workflows.