An Intelligent Multimodal Deep Learning Framework for Early Diagnosis of Diabetic and Hypertensive Retinopathy Using Retinal Fundus Images

Technical Summary: An Intelligent Multimodal Deep Learning Framework for Early Diagnosis of Diabetic and Hypertensive Retinopathy

Problem Statement
Diabetic Retinopathy (DR) and Hypertensive Retinopathy (HR) are leading causes of irreversible vision loss globally. While early diagnosis is critical for preventing blindness, current diagnostic practices face significant limitations. Most existing systems rely exclusively on retinal fundus image analysis, often ignoring vital systemic clinical biomarkers such as blood glucose levels (glycated hemoglobin) and blood pressure. This "image-only" approach can lead to reduced diagnostic accuracy, particularly in early-stage cases or when image quality varies. Furthermore, many existing models focus on either DR or HR in isolation, rather than addressing both conditions simultaneously within a unified framework. There is also a gap in the clinical implementation of multimodal systems that integrate imaging with electronic health records (EHR) using accessible prototyping tools like MATLAB.

Methodology
The authors propose a hybrid, multimodal deep learning framework implemented in MATLAB (R2023a) to diagnose both DR and HR by fusing retinal fundus images with systemic clinical data.

1. Data Acquisition: The study utilized a dataset of 1,200 fundus images from a hospital in Pune, India, accompanied by corresponding clinical records. The dataset included 300 normal cases, 825 DR cases (mild, moderate, severe), and 35 HR cases. Clinical attributes included glycated hemoglobin (referred to as electronic health records in the text), systolic blood pressure (referred to as systolic electronic health records), and disease duration.
2. Preprocessing:
   • Images: Fundus images underwent contrast enhancement via histogram equalization, noise reduction using median filtering, conversion to grayscale, and brightness normalization.
   • Clinical Data: Numerical clinical variables were normalized using z-score scaling.
3. Segmentation and Feature Extraction:
   • Image Processing: The framework employed U-Net for blood vessel and optic disc segmentation, active contour models for lesion contours, and region growing for exudate identification.
   • Feature Extraction: Deep features were extracted from the segmented regions using Convolutional Neural Networks (CNNs).
4. Multimodal Fusion and Classification:
   • The system utilizes an Intermediate (Joint) Fusion strategy. Image features (extracted via CNN/Vision Transformer) and clinical features (encoded via MLP or text encoders) are concatenated or fused using attention mechanisms to create a joint latent representation.
   • The fused features are passed through a classification layer (Softmax) to predict disease labels.
   • The model architecture integrates deep learning (CNN) with clinical data processing using MLP encoders and gated fusion mechanisms.
5. Validation: The dataset was split into training (70%), validation (15%), and testing (15%) sets. The model was evaluated using 10-fold cross-validation to ensure stability and prevent overfitting.

Key Contributions

• Dual-Pathology Detection: Unlike many prior works that focus on a single disease, this framework simultaneously diagnoses both Diabetic and Hypertensive Retinopathy.
• Multimodal Integration: The study demonstrates the efficacy of fusing retinal imaging with systemic biomarkers (glycated hemoglobin and blood pressure) to overcome the limitations of image-only models.
• MATLAB Implementation: The framework is developed entirely in MATLAB, offering a scalable, low-cost solution suitable for medical prototyping and resource-constrained environments.
• Hybrid Architecture: The integration of deep learning (CNN) with clinical data processing (MLP) creates a robust diagnostic tool that captures both local retinal lesions and systemic physiological context.

Results
The proposed multimodal framework demonstrated superior performance compared to single-modality baselines:

• Accuracy: The hybrid model achieved an overall accuracy of 98%.
• Comparison: This outperformed image-only CNN models (91.82% accuracy) and clinical-data-only Random Forest models (85.90% accuracy).
• Metrics: The model achieved a Sensitivity of 96.04%, Specificity of 98.10%, an F1-Score of 0.976, and an AUC of 0.98.
• Error Reduction: The integration of clinical data reduced false negatives by approximately 40% compared to baseline CNN approaches, particularly improving the detection of early-stage DR and HR.
• Statistical Significance: A paired t-test confirmed that the inclusion of clinical features resulted in a statistically significant improvement in accuracy (p < 0.001). ANOVA testing also showed a significant main effect of modality type (F(2, 27) = 23.6, p < 0.0005).
• Generalization: The model maintained accuracy above 95% across different demographic groups (ages 35–75) and disease stages.

Significance and Claims
The paper claims that this multimodal framework represents a "quantum leap" from traditional image-only methods by significantly enhancing diagnostic sensitivity, especially for early-stage cases where visual signs are subtle. By incorporating systemic context (glycated hemoglobin and blood pressure), the system reduces false negatives and improves risk stratification.

The authors assert that the framework is particularly valuable for:

• Screening and Telemedicine: Providing a scalable, low-cost solution for large-scale ophthalmic screening, especially in areas with limited specialist access.
• Clinical Decision Support: Offering a complete risk evaluation that aids in faster and more accurate decision-making.
• Dual Diagnosis: Enabling the screening of both DR and HR through a single image acquisition pathway.

The study concludes that while the current results are promising, future work should focus on validating the model with multi-center data, implementing the system on mobile devices for point-of-care diagnosis, and conducting longitudinal studies to track disease progression and treatment response.