Diabetic Retinopathy Grading with CLIP-based Ranking-Aware Adaptation:A Comparative Study on Fundus Image

1. Problem Statement

Diabetic Retinopathy (DR) is a leading cause of preventable blindness, requiring early and accurate severity grading (ranging from "No DR" to "Proliferative DR") for effective clinical intervention. Current manual grading is expensive, time-consuming, and prone to inter-reader variability. While deep learning (CNNs) has advanced automated grading, class imbalance (few severe cases) and subtle inter-class differences remain significant challenges.

The paper investigates whether Vision-Language Foundation Models (specifically CLIP), known for strong zero-shot capabilities, can be effectively adapted for this specialized medical task. It addresses three specific questions:

1. Can CLIP perform DR grading without domain-specific training (Zero-shot)?
2. Can CLIP features be integrated with lesion-localization mechanisms (Hybrid models)?
3. Can the ordinal nature of DR severity (mild $\to$ severe) be explicitly encoded to improve performance (Ranking-aware)?

2. Methodology

The study compares three distinct CLIP-based strategies trained and evaluated on a combined dataset of APTOS 2019 and Messidor-2 ($n=5,406$ images).

A. Data Preparation

• Dataset: Merged APTOS 2019 and Messidor-2 to maximize minority class representation.
• Imbalance Handling: The dataset is highly imbalanced (No DR: 52.2% vs. Severe/Proliferative: ~11%).
  • Resampling: Combined undersampling of majority classes and oversampling of minority classes to create a balanced training set.
  • Augmentation: Standard augmentations for all classes; stronger augmentations (higher flip probability, wider rotation, random erasing) applied to minority classes.
  • Threshold Calibration: Class-specific optimal thresholds were calculated on the validation set to convert multi-class classification into five one-vs-rest decisions, improving recall for minority classes.

B. Three Proposed Approaches

1. Zero-Shot Baseline:

   • Uses the frozen ResNet-based CLIP image encoder.
   • Relies on manually engineered text prompts (e.g., "a fundus photograph showing mild diabetic retinopathy").
   • Prediction is based on the highest cosine similarity between image and text embeddings. No domain-specific training occurs.

2. Hybrid FCN-CLIP with CBAM Attention:

   • Architecture: Uses the frozen CLIP image encoder to extract feature maps, which are fed into a Fully Convolutional Network (FCN) decoder.
   • Attention Mechanism: Integrates CBAM (Convolutional Block Attention Module) to apply sequential channel and spatial attention, forcing the model to focus on clinically relevant lesions (microaneurysms, hemorrhages, exudates) rather than global anatomy.
   • Training: Supervised fine-tuning with weighted cross-entropy and focal loss ($\gamma=2.0$) to handle imbalance.

3. Ranking-Aware Prompting Model:

   • Concept: Exploits the ordinal structure of DR grades (0 to 4).
   • Mechanism: Introduces learnable prompt vectors for each grade. The model optimizes these prompts so that similarity scores respect the ordinal rank (i.e., a "Severe" image should score higher for "Severe" than "Mild," and "Mild" higher than "No DR").
   • Loss Function: Combines standard cross-entropy with a ranking loss (weighted $\alpha=0.7$) to enforce ordinal consistency.

3. Key Contributions

• Systematic Comparison: Provides a controlled evaluation of zero-shot, supervised hybrid, and prompt-learning strategies under identical data and evaluation conditions.
• Ordinal Adaptation: Demonstrates the efficacy of encoding the ordinal severity of DR directly into the prompt learning process.
• Imbalance Mitigation Analysis: Quantifies the impact of combining resampling with class-specific threshold calibration on minority class recall.
• Complementary Strengths: Identifies that different model architectures excel at different clinical tasks (e.g., one is better at general accuracy, the other at detecting the most critical proliferative stage).

4. Experimental Results

The models were evaluated on a held-out test set (15% of data) using Accuracy, AUROC, Precision, Recall, and F1-score.

• Performance: Both fine-tuned models vastly outperformed the zero-shot baseline.
• Error Analysis: The zero-shot baseline suffered from frequent "off-by-two-or-more" errors (10%), which are clinically dangerous. The fine-tuned models reduced this to <3%.
• Attention Visualization: Qualitative analysis showed the Hybrid FCN-CLIP model successfully localized lesions (microaneurysms), whereas the zero-shot model focused on diffuse anatomical features (optic disc).

5. Significance and Discussion

• Clinical Implications:
  • Screening Context: The Ranking-Aware model is superior for population-level screening due to its high overall accuracy and excellent recall for severe cases (83%), minimizing the risk of missing critical patients.
  • Confirmatory Context: The Hybrid FCN-CLIP model achieves higher precision and significantly better recall for Proliferative DR (74% vs. 50%). Since Proliferative DR is the stage most likely to cause blindness, this model is better suited for confirmatory diagnosis or high-risk triage.
• Ensemble Potential: The authors suggest that combining these two approaches could yield a system that leverages the ordinal sensitivity of the ranking model and the lesion-localization precision of the hybrid model.
• Limitations: The study notes limitations regarding external validation (different devices/demographics), the specific geographic scope of the dataset, and the computational cost of the hybrid model for resource-constrained settings.

Conclusion

The paper concludes that while CLIP has potential, domain-specific adaptation is essential for effective DR grading. The study establishes that incorporating ordinal structure (Ranking-Aware) and lesion localization (Hybrid FCN-CLIP) are the two most effective strategies, offering complementary strengths that can be leveraged for robust clinical deployment.