From Pixels to Explanations: Interpretable Diabetic Retinopathy Grading with CNN-Transformer Ensembles, Visual Explainability and Vision-Language Models

Technical Summary: From Pixels to Explanations: Interpretable Diabetic Retinopathy Grading

1. Problem Statement

Diabetic Retinopathy (DR) is a leading cause of vision loss globally. While deep learning (DL) models have achieved high accuracy in grading DR severity (from "No DR" to "Proliferative DR"), they suffer from two critical clinical limitations:

• The "Black Box" Nature: Most high-performing models lack interpretability, providing a grade without explaining the underlying pathological evidence (e.g., microaneurysms or hemorrhages).
• Limitations of Current Explainability: Existing visual attribution methods (like Grad-CAM) often produce diffuse, coarse heatmaps that are difficult for clinicians to translate into concrete diagnostic reasoning. Furthermore, current methods lack a systematic way to evaluate the quality and clinical faithfulness of these explanations.

2. Methodology

The authors propose a unified pipeline that transitions from raw pixels to multimodal explanations using the APTOS 2019 benchmark. The study is structured into three experimental stages:

• Stage 1: Backbone Benchmarking: The researchers evaluated six representative architectures—four CNN-based (ResNet-50, DenseNet-121, EfficientNet-B3, ConvNeXt-Tiny) and two Transformer-based (DeiT-Small, Swin Transformer) — using stratified five-fold cross-validation.
• Stage 2: Ensemble Learning: To improve robustness against class imbalance and subtle grade boundaries, they compared four fusion strategies:
  • Hard Voting (majority rule).
  • Weighted Soft Voting (averaging probabilities using validation-derived weights).
  • Stacking (using a multinomial logistic regression meta-learner).
  • Hybrid Class-Level Fusion (a novel approach assigning different model weights to different DR grades).
• Stage 3: Multimodal Explainability Pipeline:
  • Visual: Generation of Grad-CAM++ saliency maps to localize influential retinal regions.
  • Textual: Utilizing Vision-Language Models (VLMs)—specifically a general-purpose model (LLaVA-1.5-1B) and a biomedical-adapted model (LLaVA-Med-1.5-Mistral-7B)—to generate concise clinical rationales.
  • Prompt Engineering: The authors used "conservative prompting" to force the VLMs to describe only visible findings, mention image quality, and avoid hallucinations.
  • Quantitative Evaluation: Explanations were measured using BERTScore (semantic similarity to templates), CLIPScore (image-text alignment), and a custom Coverage metric (presence of 10 key clinical descriptors).

3. Key Contributions

1. Systematic Backbone Comparison: A controlled benchmark showing how convolutional vs. attention-based inductive biases affect ordinal DR grading.
2. Novel Hybrid Fusion Investigation: An exploration of whether specialized class-level weighting can outperform global probability fusion.
3. Multimodal Interpretability Framework: A pipeline that combines visual localization with textual reasoning.
4. Quantitative Explanation Assessment: Moving beyond qualitative "eye-balling" to provide measurable metrics for the quality of generated medical text.

4. Key Results

• Grading Performance: Modern CNNs were the strongest single-model baselines. ResNet-50 (QWK 0.919) and ConvNeXt-Tiny (QWK 0.914) outperformed transformers.
• Ensemble Superiority: Ensembling improved ordinal agreement. Weighted Soft Voting was the most consistent and effective strategy (QWK 0.934). The proposed Hybrid Class-Level Fusion was competitive but did not yield statistically significant improvements over standard soft voting.
• Explanation Quality:
  • Visual: Grad-CAM++ provided plausible but coarse localization.
  • Textual Trade-off: A significant trade-off was observed between the two VLMs: the biomedical-adapted model (LLaVA-Med) achieved higher clinical coverage (more medical terms), while the general-purpose model (LLaVA) achieved higher BERTScore (closer adherence to template phrasing).
  • Alignment: Both models showed comparable image-text grounding (CLIPScore).

5. Significance and Clinical Implications

The study demonstrates that AI for DR grading can move toward a "decision-support" role rather than just a "classification" role. By providing both a heatmap and a written rationale, the system acts as a triage and review aid for ophthalmologists.

Clinical Insight: The authors emphasize that this system should be used as an "expert-in-the-loop" tool. The multimodal output helps clinicians quickly verify if a model's decision is based on legitimate lesions or imaging artifacts, thereby increasing trust and efficiency in high-volume screening environments.