Triple-Phase Multimodal Knowledge Aggregation Framework for Microbial Keratitis Subtype Diagnosis on Slit-Lamp Photography

Technical Summary: Triple-Phase Multimodal Knowledge Aggregation Framework for Microbial Keratitis Subtype Diagnosis

Problem Statement
Microbial keratitis (MK) is a severe corneal infection requiring rapid identification of the causative pathogen (bacterial vs. fungal) to guide targeted treatment. Current diagnostic standards, relying on culture and PCR, are slow (taking several days) and fail to identify pathogens in over 50% of cases. This delay often leads to suboptimal empirical treatment, worsening clinical outcomes. While slit-lamp photography (SLP) is a widely available, low-cost imaging modality that captures specific biomarkers (e.g., stromal infiltrates, hypopyon) under varying illumination, distinguishing between bacterial and fungal subtypes based solely on these images is challenging due to subtle visual differences and significant heterogeneity in imaging conditions (illumination modes, magnification). Existing deep learning approaches often suffer from poor generalization when applied to external cohorts and tend to be evaluated on pooled multi-center data, which can mask geographic and distributional biases.

Methodology
The authors propose a Triple-Phase Multimodal Knowledge Aggregation Framework designed to classify bacterial versus fungal MK using SLP images and clinical metadata. The framework integrates data from three illumination modalities: diffuse blue light (with fluorescein), sclerotic scatter, and diffuse white light.

1. Data Curation & Preprocessing:

   • A large-scale, multi-center dataset was curated comprising 1,645 patients (1,048 fungal, 597 bacterial) and 17,158 images from the US and India.
   • Limbus-based Cropping: A connectivity-based segmentation network (BiconNet) was used to crop the limbus region, removing irrelevant background noise (sclera, eyelids) to focus the model on lesion-relevant areas.
   • Metadata Integration: Clinical variables (trauma history, organic/water exposure, contact lens use, visual acuity) were embedded and concatenated with image features.

2. Phase One: Cross-Modality Contrastive Learning (CMCL):

   • A universal model is pretrained using contrastive learning. Images from the same patient, regardless of illumination modality or magnification, are treated as positive pairs, while images from different patients are negative pairs.
   • This phase aims to learn modality-invariant representations, capturing disease-specific features that persist across different imaging conditions.

3. Phase Two: Modality-Specific Fine-Tuning:

   • The pretrained weights are transferred to three separate networks, each fine-tuned on a specific illumination modality (Blue, Sclerotic Scatter, White).
   • A regularization term (the contrastive loss) is added to the classification loss during fine-tuning to maintain the learned invariant features while adapting to modality-specific cues.

4. Phase Three: Feature-Level Multimodal Ensemble:

   • Instead of simple majority voting, the framework extracts feature vectors from the layer preceding the classification head for each modality-specific network.
   • These vectors are averaged per modality (handling missing modalities via zero-vector initialization) and concatenated into a single feature matrix.
   • A Multi-Layer Perceptron (MLP) fuses these features to produce the final patient-level prediction.

Key Contributions

1. Dataset Construction: The creation of a large-scale, multi-center, multi-modality dataset (17,158 images from 1,645 patients) with paired clinical metadata, addressing the scarcity of annotated SLP datasets for MK.
2. Novel Framework: The introduction of a triple-phase architecture that explicitly addresses imaging heterogeneity through contrastive pre-training, modality-specific fine-tuning, and feature-level ensemble learning.
3. Rigorous Evaluation Protocol: The authors expose a critical pitfall in prior multi-center studies: aggregate metrics on pooled data often yield overly optimistic results that mask severe performance disparities across geographic sites. They introduce site-stratified evaluations under three protocols (original distribution, resampling, and fully balanced subsets) to provide a more realistic assessment of cross-site generalization.

Results

• Overall Performance: The proposed framework achieved state-of-the-art performance with 85.84% accuracy, 84.46% average F1-score, and 0.885 AUC, outperforming baselines including ResNet-50, DenseNet-121, SwinV2-T, and DeepIK.
• Bacterial Classification: The framework notably improved the recognition of bacterial cases, which are historically difficult. The bacterial F1-score increased from ~76.7% (best baseline) to 80.0%, while maintaining a high fungal F1-score of 89.0%.
• Modality Analysis: The model performed best under sclerotic scatter illumination (84.81% accuracy) and showed consistent gains across all modalities compared to single-modality baselines.
• Site-Specific Generalization:
  • On the original dataset, all methods showed high fungal performance in India but poor bacterial performance, and vice versa in the US, indicating geographic bias.
  • Under resampling and balanced evaluation settings, the proposed framework remained the top performer, demonstrating greater robustness and fairness across sites compared to baselines.
  • An ablation study removing explicit location metadata ("No Location") still showed geographic bias, suggesting the model implicitly learns site-specific visual cues (e.g., sensor noise, population demographics) from the images themselves.

Significance and Claims
The paper claims that this work represents a significant step toward accessible, AI-assisted diagnosis for microbial keratitis. By leveraging a triple-phase framework, the authors demonstrate that it is possible to extract robust, modality-invariant features from heterogeneous slit-lamp images, thereby improving diagnostic accuracy for the challenging bacterial subtype.

Crucially, the authors argue that their evaluation methodology provides a more faithful benchmark for real-world deployment. They highlight that standard pooled-data evaluation can be misleading, concealing significant weaknesses in specific geographic domains. Their framework's ability to maintain top performance under stringent, balanced, and resampled evaluation settings suggests a higher potential for fair and generalizable cross-site deployment.

The authors remain modest regarding clinical deployment, noting that while offline performance is strong, prospective validation in real-world point-of-care workflows is essential. They also acknowledge current limitations, including the binary nature of the classification (bacterial vs. fungal), the need for broader geographic representation to mitigate shortcut learning, and the necessity of standardized imaging protocols in future prospective studies.