A Multimodal AI Framework for Pulmonary Embolism Detection, Segmentation, and Patient Risk Assessment

Technical Summary: A Multimodal AI Framework for Pulmonary Embolism Detection, Segmentation, and Patient Risk Assessment

Problem Statement
Pulmonary Embolism (PE) is a critical cardiovascular condition requiring rapid diagnosis and accurate risk assessment to reduce mortality. While Computed Tomography Pulmonary Angiography (CTPA) is the standard diagnostic tool, current AI-driven approaches largely focus on a single aspect: either detecting PE or segmenting clots from medical images. These existing solutions often fail to incorporate patient-specific physiological and clinical data, which is essential for determining disease severity, treatment pathways, and predicting future complications. Consequently, there is a gap in methodologies that integrate visual analysis with longitudinal physiological data to provide a comprehensive diagnostic and prognostic tool.

Methodology
The authors propose a multimodal Artificial Intelligence framework that integrates three distinct modules to address detection, segmentation, and risk assessment simultaneously. The system architecture combines image processing capabilities with clinical data analysis:

1. Imaging Module (Clot Detection and Segmentation):

   • Data Source: CTPA images from the FUMPE dataset (DICOM format) with corresponding clot masks (MATLAB format).
   • Preprocessing: Images undergo Min-Max normalization and are converted into pseudo-RGB format (three-channel) to suit the model input.
   • Architecture: A custom Mask R-CNN model is employed. It utilizes a ResNet-50 backbone for feature extraction, followed by Feature Pyramid Network (FPN) layers for multi-scale learning, a Region Proposal Network (RPN) for localization, and ROI Align for precise feature extraction. Two heads handle detection and segmentation tasks.
   • Training: The model was trained for 20 epochs using the Adam optimizer.

2. Future PE Prediction Module:

   • Data Source: Physiological time-series data from the PhysioNet Challenge Dataset (48-hour ICU records including heart rate, oxygen saturation, blood pressure, respiratory rate, and temperature).
   • Preprocessing: Missing data is imputed using mean values, and data is standardized.
   • Architecture: A hybrid LSTM-GRU neural network. The Long Short-Term Memory (LSTM) layer captures long-term physiological trends, while the Gated Recurrent Unit (GRU) layer learns short-term variations. This combination aims to model complex temporal dependencies indicative of future PE occurrence.

3. Deterioration Risk Assessment Module:

   • Feature Engineering: Statistical characteristics (mean, max, min, standard deviation, and quartiles) are extracted from the physiological time-series data to create a feature vector representing the patient's state.
   • Architecture: An Artificial Neural Network (ANN) with fully connected layers, ReLU activation, and dropout regularization. This module learns non-linear correlations between clinical variables to predict the probability of clinical deterioration.

4. Multimodal Fusion:

   • The outputs from the Mask R-CNN (clot status/segmentation), LSTM-GRU (future PE risk), and ANN (deterioration risk) are fused to generate a comprehensive patient assessment report.

Key Results
The framework was evaluated using Python-based deep learning libraries (PyTorch, TensorFlow, Keras) on Google Colab GPU architecture.

• Segmentation Performance: The custom Mask R-CNN achieved a Dice Score of 0.88 and an Intersection over Union (IoU) of 0.77, indicating strong similarity between predicted and ground-truth clot regions.
• Prediction Performance: The hybrid LSTM-GRU model demonstrated an accuracy of 86.50% in predicting future PE occurrences based on physiological signals.
• Overall System Accuracy: The multimodal fusion framework achieved an overall system accuracy of 91.81% for future prediction tasks. This represents a significant improvement over a baseline pretrained system, which achieved 82.20% accuracy.
• Qualitative Output: A visualization dashboard was developed to display clot localization, severity (e.g., 0.11% severity in a sample case), future PE risk (e.g., 9.84%), and deterioration risk (e.g., 21.55%).

Significance and Claims
The paper claims that the primary significance of this work lies in its multimodal approach, which moves beyond the limitations of image-only analysis. By integrating CTPA imaging with ICU physiological time-series data, the framework offers a more holistic view of the patient.

• Comprehensive Assessment: Unlike traditional tools that only identify the presence of a clot, this system provides clot segmentation, estimates clot severity, predicts the likelihood of future PE events, and assesses the risk of clinical deterioration.
• Clinical Utility: The authors posit that the system serves as an AI-assisted decision support tool, offering an automated, unified report that can aid radiologists and clinicians in developing appropriate treatment strategies and risk management plans.
• Performance Validation: The study asserts that the custom training on domain-specific data and the integration of diverse data modalities are responsible for the observed accuracy improvements over baseline models.

Limitations and Future Work
The authors acknowledge that while the results are encouraging, the framework requires validation on larger, multicenter databases to ensure generalizability. Future work aims to incorporate additional clinical parameters and data from multiple centers to further enhance model performance.