Technical Summary: Multimodal Machine Learning Integrating Systemic and Imaging Data for ROP Surgical Triage

Problem Statement

Retinopathy of prematurity (ROP) is a leading cause of irreversible childhood blindness. Current clinical management relies heavily on clinician experience and subjective interpretation of fundus images to determine the timing of surgical intervention and assess prognosis. Existing artificial intelligence approaches have primarily focused on unimodal analysis of fundus images, often failing to leverage the rich systemic clinical data available for preterm infants. Furthermore, few studies have attempted to predict the specific need for surgical intervention or the risk of reoperation (secondary surgery) by integrating heterogeneous data sources, such as cardiovascular indicators, perinatal metabolic factors, and temporal sequences of retinal images. The lack of systematic integration limits the objectivity and accuracy of current decision-support tools.

Methodology

Study Design and Data

This retrospective study enrolled 213 preterm infants from Xiamen Maternal and Child Health Hospital. The dataset included:

• Structured Clinical Variables: Six indicators were selected based on clinical relevance: congenital heart disease (CHD), patent ductus arteriosus (PDA), persistent pulmonary hypertension of the newborn (PPHN), the lowest hemoglobin level during hospitalization, maternal gestational diabetes mellitus (GDM), and infant of diabetic mother syndrome.
• Imaging Data: A series of fundus images for each patient. To manage computational load while preserving patient-level information, a maximum of 200 images per patient was used (randomly sampled if exceeding this limit). Images were resized to 224 × 224 pixels and normalized.

Prediction Tasks

The study defined two binary classification tasks at the patient level:

1. Surgical Decision Prediction: Classifying patients as "Not Needed" or "Needed" for surgical/interventional treatment.
2. Surgical Prognosis Prediction: Among patients labeled "Needed," classifying them as "Primary" (single surgery) or "Secondary" (requiring reoperation).

Multimodal Architectures

Four distinct multimodal fusion models were designed and compared:

1. EarlyFusion_Attention: Utilized feature-level fusion. A convolutional encoder processed images, and a fully connected encoder processed clinical variables. Image and clinical features were concatenated per image, and an attention mechanism assigned learnable weights to these joint features before weighted summation to form the patient representation.
2. LateFusion_LSTM: Treated images as a sequence. Images were encoded individually, processed by a bidirectional LSTM to capture temporal dynamics, and then concatenated with an independently encoded clinical representation.
3. Transformer_Fusion: Encoded images and clinical variables into tokens within a shared embedding space. A clinical token and image tokens were processed by a two-layer Transformer encoder with self-attention to generate a global patient representation.
4. TwoTower: Employed separate encoders for images (average pooling of image features) and clinical data, concatenating the resulting representations for final classification.

Training and Evaluation

Models were trained using a multitask learning strategy with weighted cross-entropy loss to address class imbalance. The dataset was split at the patient level into training, validation, and test sets, with stratified allocation to ensure the presence of rare secondary surgery cases across subsets. Performance was evaluated using accuracy, weighted F1-score, sensitivity, specificity, and Area Under the Curve (AUC).

Key Results

Model Performance

Among the four architectures, the EarlyFusion_Attention model demonstrated the best overall performance on the independent test set (43 patients: 37 "Not Needed," 6 "Needed").

• Surgical Decision Prediction:
  • Accuracy: 62.79%
  • Weighted F1 Score: 0.6829
  • ROC AUC: 0.80
  • Sensitivity: 1.000 (Correctly identified all 6 patients requiring surgery; 0 false negatives).
  • Specificity: 0.568 (16 "Not Needed" patients were misclassified as "Needed").
• Prognosis Prediction (Surgical Subgroup):
  • Evaluated on the 6 patients who required surgery (5 Primary, 1 Secondary).
  • Accuracy: 83.33% (5/6 correct).
  • Weighted F1 Score: 0.8519.
  • ROC AUC: 1.00 (Note: Based on a single Secondary case).

Subgroup Analysis

The model's performance varied by patient characteristics:

• Gestational Age: The model performed well in the 28–32 weeks (AUC 0.964) and ≥32 weeks (AUC 0.933) groups but showed limited discrimination in the <28 weeks group (AUC 0.542).
• Birth Weight: Performance was more reliable in infants with birth weight ≥1500g (AUC 0.889) compared to those <1000g (AUC 0.583).

Key Contributions

1. Multimodal Integration: The study successfully integrated cardiovascular-metabolic clinical indicators with serial temporal fundus images, moving beyond unimodal image analysis to a patient-level multimodal framework.
2. Dual-Task Framework: It addressed two critical clinical questions simultaneously: the initial need for surgery and the risk of recurrence requiring secondary surgery.
3. Architectural Comparison: The paper provides a structured comparison of four different fusion strategies (Early, Late, Transformer, Two-Tower) for ROP triage, identifying EarlyFusion_Attention as the superior architecture for this specific data distribution.
4. High Sensitivity Screening: The proposed model demonstrated perfect sensitivity in the test set for identifying patients requiring surgery, suggesting utility as a screening tool to flag high-risk infants.

Significance and Claims

The authors claim that this study provides preliminary evidence for the feasibility of using multimodal machine learning to support ROP treatment decision-making. The EarlyFusion_Attention model's high sensitivity suggests it could serve as an auxiliary tool to help clinicians flag infants who require closer monitoring or surgical evaluation, potentially reducing missed cases.

However, the paper maintains a modest stance regarding its clinical readiness:

• Specificity Limitations: The moderate specificity (0.568) indicates a tendency toward false positives, meaning the model should not replace ophthalmologist assessment but rather support risk stratification.
• Data Scarcity: The prognosis prediction results, while promising (AUC 1.00), are explicitly described as "exploratory" due to the extremely small sample size of the surgical subgroup (6 patients) and the single secondary surgery case.
• Validation Needs: The authors emphasize that external validation in larger, multicenter cohorts is required before the model can be considered for independent clinical application.

In summary, the study demonstrates that integrating systemic clinical data with imaging features can enhance the objectivity of ROP triage, particularly in identifying patients who need intervention, but further research is needed to refine specificity and validate prognostic capabilities.