Technical Summary: Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessment

Problem Statement

Pulmonary embolism (PE) is a high-risk cardiopulmonary condition requiring timely diagnosis and reliable assessment of future clinical risk. Current clinical practice relies on a multimodal approach, integrating Computed Tomography Pulmonary Angiography (CTPA) imaging with longitudinal Electronic Health Record (EHR) data, including patient history, vital signs, and laboratory results. However, existing deep learning approaches for PE risk prediction often rely on conventional score-based systems or specialized 3D CNNs that lack the ability to generate explanations or support interactive clinical reasoning. Furthermore, large-scale Multimodal Large Language Models (MLLMs) capable of such reasoning often require cloud-based inference, raising privacy concerns, or demand computational resources unavailable in many clinical settings.

There is a need to evaluate compact (efficient) multimodal language models that can operate locally while effectively integrating visual (CTPA) and textual (EHR) evidence to answer specific clinical questions regarding PE diagnosis and prognosis.

Methodology

Dataset: INSPECT

The study utilizes the INSPECT dataset, a large-scale benchmark containing 23,248 CTPA studies from 19,402 patients. The dataset links index CTPA studies with radiology report impressions and structured longitudinal EHR records.

• Tasks: The authors reformulate eight endpoints as structured clinical question-answering (QA) tasks:
  • 1 Diagnostic Task: Presence of PE at the index CTPA.
  • 7 Prognostic Tasks: In-hospital mortality (1, 6, 12 months), readmission (1, 6, 12 months), and pulmonary hypertension (12 months).
• Data Handling: Prognostic samples with insufficient follow-up are marked as censored and excluded from specific task evaluations. The dataset is split at the patient level to prevent data leakage.

Experimental Setup

• Models: Four compact MLLMs were evaluated without parameter updates (zero-shot or few-shot):
  • Qwen3.5 4B
  • Gemma4 E4B (4B)
  • Gemma4 E2B (2B)
  • MedGemma (4B)
• Input Modalities: Three settings were tested:
  1. CTPA-Only: Volumetric CTPA converted to a deterministic 2D montage of axial slices.
  2. EHR-Only: Serialized structured EHR data (demographics, diagnoses, medications, etc.) available prior to the index scan.
  3. CTPA+EHR: Combined visual and textual evidence.
• Prompting: Experiments compared zero-shot and four-shot prompting. Demonstrations were drawn strictly from the training split with patient-level separation.
• Output: Models were instructed to return a constrained structured response containing a binary prediction and a confidence score.
• Metrics: Performance was measured using AUROC, AUPRC (crucial due to class imbalance), Accuracy, and F1 score. Macro-averaging was used across the eight tasks.

Key Contributions

1. Compact MLLM Benchmark: The authors established a benchmark for multimodal PE diagnosis and prognosis using the INSPECT dataset, reformulating eight endpoints as structured clinical QA tasks across CTPA-Only, EHR-Only, and CTPA+EHR settings.
2. Evaluation of Prompting Strategies: The study evaluated zero-shot and four-shot prompting across four compact models, demonstrating that in-context examples improve performance primarily when the model already possesses the capability to interpret the relevant clinical evidence.
3. Identification of Failure Modes: The work identifies specific failure modes in compact multimodal medical QA, including:
   • Majority-class collapse: Models achieving high accuracy by predicting the negative class due to class imbalance.
   • Weak CTPA-Only performance: Compact models struggle to utilize volumetric imaging data without clinical context.
   • Task-level heterogeneity: Significant performance gaps between diagnostic tasks and longitudinal prognostic tasks.

Results

Aggregate Performance

• Model Variance: Qwen3.5 4B and MedGemma performed close to chance level across all modalities, often exhibiting majority-class behavior (high accuracy, near-zero F1).
• Gemma Models: Gemma4 E4B and Gemma4 E2B demonstrated the ability to extract meaningful clinical signals, particularly when EHR evidence was available.
• Best Performance: The strongest zero-shot performance was achieved by Gemma4 E4B with CTPA+EHR input (AUROC: 0.6893, F1: 0.3286). Four-shot prompting further improved positive-case recovery for the strongest Gemma configurations.

Task-Specific Findings

• PE Diagnosis: This was the most accessible task. Gemma4 E4B achieved an AUROC of 0.9683 (zero-shot, CTPA+EHR) and 0.9706 (four-shot, CTPA+EHR). Notably, EHR-Only input also yielded high performance, suggesting serialized clinical context contains strong diagnostic signals. CTPA-Only performance remained weak.
• Mortality Prediction: More difficult than diagnosis but showed meaningful discrimination for Gemma models with clinical context. Gemma4 E4B achieved an AUROC of 0.7242 (zero-shot, CTPA+EHR).
• Readmission Prediction: The most difficult task group. Even the best configurations (Gemma4 E2B, four-shot, CTPA+EHR) achieved only an AUROC of 0.6141, indicating that readmission is shaped by complex factors not fully captured by the available data.
• Pulmonary Hypertension: Showed intermediate difficulty, with Gemma4 E4B achieving an AUROC of 0.6952 (zero-shot, CTPA+EHR).

Failure Modes

• Majority Class Collapse: Models like Qwen3.5 and MedGemma often predicted the negative class for most cases, resulting in deceptively high accuracy but poor clinical utility (low F1/AUPRC).
• Modality Misalignment: Providing more evidence (CTPA+EHR) did not improve performance for all models; it only benefited models (like Gemma) capable of aligning the evidence with the task.
• Limited Impact of Few-Shot: Four-shot prompting refined task alignment and positive-case recovery for capable models but could not compensate for models lacking the underlying ability to interpret the evidence.

Significance and Claims

The paper claims that compact multimodal models have the potential for early-stage PE risk detection and explanation, but their success is highly dependent on the alignment between model capability, input modality, and the clinical endpoint.

• Clinical Relevance: The study highlights that robust PE assessment is inherently multimodal. While imaging is critical for diagnosis, prognostic outcomes (mortality, readmission) rely heavily on longitudinal clinical context found in EHRs.
• Efficiency vs. Capability: The results suggest that while compact models can be effective, they are fragile. They require appropriate evidence sources (EHR or combined) to function correctly; relying solely on CTPA images leads to poor performance for these specific architectures.
• Evaluation Rigor: The authors emphasize that standard accuracy metrics are insufficient for imbalanced medical tasks. They advocate for evaluating discrimination (AUROC), positive-class recovery (AUPRC/F1), and task-level heterogeneity to avoid misleading conclusions about model utility.

The study concludes that while compact MLLMs offer a path toward local, privacy-preserving clinical AI, significant challenges remain in ensuring they do not collapse into majority-class predictors and can effectively utilize complex multimodal inputs for long-horizon prognostic tasks.