Technical Summary: Quasi-multimodal-based Pathophysiological Feature Learning for Retinal Disease Diagnosis

Problem Statement

Retinal diseases span a broad spectrum and are effectively identified using complementary signals from multimodal data, such as Color Fundus Photography (CFP), Fundus Fluorescein Angiography (FFA), and Multispectral Imaging (MSI). However, the physical deployment of true multimodal imaging in clinical practice faces significant hurdles:

• Invasiveness and Risk: Techniques like FFA require dye injection, posing potential risks.
• Data Heterogeneity and Alignment: Different modalities suffer from spatial misalignment due to eye movements (e.g., saccades) and lack standardized protocols.
• Registration Complexity: Aligning raw data or learned features across modalities is computationally and technically challenging.
• Information Redundancy: Simply fusing multiple modalities without adaptive calibration can lead to redundancy and mutual exclusion, degrading diagnostic performance.

Existing quasi-multimodal approaches often lack specific modalities (e.g., arterial-phase FFA) or fail to effectively handle information redundancy and generalizability gaps between fine-grained pixel synthesis and downstream diagnostic tasks.

Methodology

The authors propose a unified, two-stage learning framework that integrates multimodal data synthesis and adaptive feature fusion for retinal disease classification and grading.

Stage I: Modality-Specific Representation Learning

Parallel models, based on an encoder-decoder backbone, are trained to map a single input CFP image to various pathophysiological representations:

1. FFA Synthesis: Generates both arterial and arteriovenous phase FFA images. This utilizes a pix2pix network trained on aligned CFA-FFA pairs to capture vascular leakage and non-perfusion areas.
2. MSI Synthesis: Generates a multispectral datacube (visible to near-infrared) using a DRCR-Net backbone. To address sensor variability, the training process introduces Spectral Response Function (SRF) randomization and employs cycle-consistency loss to ensure reconstruction fidelity.
3. Saliency Map Synthesis: Generates segmentation maps highlighting latent lesions (hemorrhages, exudates, microaneurysms) and optic disc/cup regions using U-Net models trained on DDR and REFUGE datasets.

Stage II: Multimodal Diagnosis and Adaptive Calibration

Learned representations are transferred as high-level features for diagnosis. This stage introduces two key innovations:

1. Multimodal Feature Finetuning Module (MFFM): The encoders responsible for FFA and MSI synthesis are jointly finetuned with the diagnostic pipeline using task-specific losses and fidelity regularizations (adversarial and cycle-consistency losses). This aligns the synthesis process with diagnostic objectives without requiring large-scale paired data for the finetuning phase. Saliency map encoders remain frozen to preserve structural integrity.
2. Cross-Pathophysiology Attention Module (CPAM): To address information redundancy and enable flexible integration, CPAM adaptively calibrates features through a hierarchical attention mechanism:
   • Modal-Specific Feature Calibration (MSFC): Uses stacked Squeeze-and-Excitation (SE) pipelines to capture unique semantic information within each modality, suppressing irrelevant responses.
   • Modal-Dependent Feature Calibration (MDFC): Employs recursive cross-attention units to model interactions between modalities at multiple scales, capturing heterogeneous yet complementary pathophysiological cues.

The final features are concatenated and passed to a classifier for multi-label classification or diabetic retinopathy (DR) grading.

Key Contributions

1. Unified Architecture: The first quasi-multimodal model to integrate arterial-phase FFA, arteriovenous-phase FFA, and MSI synthesized solely from CFP, combined with a task-driven finetuning strategy that aligns synthesis with diagnosis.
2. Cross-Pathophysiology Attention Module (CPAM): A novel mechanism that hierarchically calibrates features using both modal-specific and modal-dependent components, enabling fine-grained, multi-scale alignment of pathophysiological signatures.
3. Performance Superiority: Extensive experiments demonstrate the approach outperforms state-of-the-art methods in both multi-label classification and DR grading.

Experimental Results

The framework was evaluated on two public datasets: MuReD (multi-label classification) and DDR (DR grading).

• Multi-label Classification (MuReD): The proposed method achieved an F1-score of 0.683 and an AUC of 0.953, outperforming the best baseline (C-Tran) by 4.11% in F1-score and 1.92% in AUC. Notable improvements were observed in vascular diseases (e.g., BRVO, CRVO) and choroidal pathologies (e.g., DN, MYA), attributed to the integration of FFA and MSI features.
• Diabetic Retinopathy Grading (DDR): The method achieved an Accuracy of 0.842 and a Kappa coefficient of 0.861, surpassing the second-best method (MuR-CAN) by 2.99% in accuracy and 0.48% in Kappa.
• Ablation Studies:
  • Modality Importance: Adding FFA and MSI to the baseline CFP significantly improved performance, with all-modality fusion yielding the highest gains (e.g., +13.64% F1-score in multi-label classification).
  • Calibration Nodes: The full CPAM (MSFC + MDFC) significantly outperformed models using only the baseline MFFM, confirming the necessity of adaptive calibration to handle redundancy.
  • Fusion Strategy: Feature-level fusion with unfrozen encoders proved superior to image-level fusion or frozen encoders, highlighting the benefit of task-driven finetuning.
• Robustness: The model maintained strong performance under simulated clinical perturbations (blur, sharpness, distortion), with performance drops generally remaining below 3.25%.

Significance and Claims

The paper claims that this work offers a scalable framework for retinal disease screening that enhances accuracy and efficiency without requiring invasive or complex multimodal data acquisition in the clinical setting. By synthesizing complementary modalities from a single CFP image and adaptively fusing them, the system bridges the gap between data heterogeneity and diagnostic needs.

The authors emphasize that their approach departs from the conventional "synthesis-then-diagnosis" paradigm by:

1. Jointly finetuning modality-specific encoders with diagnostic supervision to align representation learning with clinical objectives.
2. Explicitly modeling pathophysiological correspondences at multiple scales via CPAM, shifting from generic feature aggregation to pathology-aware calibration.
3. Propagating high-level, task-adapted features to impute missing modalities, thereby reducing the mismatch between pixel-level synthesis complexity and diagnostic prediction.

The study concludes that while the method shows promising results, future work will focus on incorporating additional clinical modalities (e.g., ICG angiography), leveraging advanced generative models, and conducting multi-center validations to further improve generalizability.