Forgetting-Resistant and Lesion-Aware Source-Free Domain Adaptive Fundus Image Analysis with Vision-Language Model

This paper proposes a novel Forgetting-Resistant and Lesion-Aware (FRLA) method for source-free domain adaptation in fundus image analysis that leverages vision-language models to preserve confident target predictions and incorporate fine-grained lesion knowledge, thereby outperforming existing state-of-the-art approaches.

The Gist

Imagine you are a highly skilled doctor who has spent years studying thousands of eye scans from Hospital A (the Source Domain). You are an expert at spotting diseases like glaucoma or diabetic retinopathy in those specific scans.

Now, you are transferred to Hospital B (the Target Domain). The cameras are different, the lighting is different, and the patients look slightly different. You have no new training data from Hospital B, and you can't look at the old Hospital A files anymore (due to privacy rules). You need to adapt your skills to diagnose patients at Hospital B immediately.

This is the challenge of Source-Free Domain Adaptation (SFDA).

Recently, scientists tried to help this doctor by giving them a "Super AI Assistant" (a Vision-Language Model). This assistant has read millions of medical textbooks and knows everything about eyes in general. However, the paper you shared points out that simply letting the doctor and the AI work together has two big problems:

1. The "Over-Correction" Problem: The AI is so confident in its general knowledge that it sometimes talks the doctor out of their own correct instincts. If the doctor is 100% sure a patient has a specific disease, but the AI says "maybe not," the doctor might start doubting themselves and make mistakes. The doctor "forgets" their own good judgment.
2. The "Blurry Photo" Problem: The AI usually looks at the whole eye scan and gives a general opinion. But eye diseases often hide in tiny, specific spots (like a tiny leak or a small spot of damage). The AI's general advice misses these fine details, which are crucial for a correct diagnosis.

The Solution: FRLA (Forgetting-Resistant & Lesion-Aware)

The authors propose a new method called FRLA to fix these issues. Here is how it works, using simple analogies:

1. The "Memory Bank" (Forgetting-Resistant Adaptation)

Imagine the doctor keeps a personal diary of their best diagnoses.

• The Old Way: The doctor and the AI argue over every single case. If the AI disagrees, the doctor changes their mind, even if the doctor was right.
• The FRLA Way: The doctor writes down their most confident diagnoses in a diary before talking to the AI. When they consult the AI, they check the diary first. If the doctor was already 99% sure about a case, they lock that decision in the diary and tell the AI, "I know this one; don't try to change my mind."
• The Result: The doctor learns from the AI's general knowledge but never loses their own hard-earned expertise. They don't "forget" what they already know.

2. The "Magnifying Glass" (Lesion-Aware Adaptation)

Imagine the AI usually looks at the eye scan from a distance and says, "There is a problem here."

• The Old Way: The doctor tries to guess where the problem is based on that vague hint.
• The FRLA Way: The AI is forced to use a magnifying glass. Instead of just giving a general opinion, it points to specific tiny patches of the image and says, "Look right here, at this specific spot, there is a lesion."
• The Result: The doctor learns to look at the exact same tiny spots the AI is highlighting. This helps the doctor spot the tiny, dangerous details (lesions) that they might have missed otherwise.

How They Work Together

The method uses a clever training schedule:

• Early in training: The "Magnifying Glass" (Lesion-Aware) is very loud. The doctor needs to learn where to look, so the AI gives strong hints about specific spots.
• Later in training: The "Magnifying Glass" gets quieter. The doctor has learned where to look, so now they focus on making the final diagnosis without being distracted by too many tiny details.

The Outcome

When the researchers tested this new method:

• The doctor (the model) became much better at diagnosing eye diseases in the new hospital than they were before.
• They outperformed other methods that tried to mix the doctor and the AI.
• Most importantly, the doctor didn't lose their original skills; they just got smarter and more detailed.

In short: This paper teaches us how to combine a human expert's specific experience with an AI's general knowledge without letting the AI confuse the expert, while also teaching the expert to look at the tiny details that matter most.

Technical Summary

1. Problem Statement

The paper addresses Source-Free Domain Adaptation (SFDA) in the context of fundus image diagnosis (screening for retinal diseases like diabetic retinopathy, glaucoma, etc.).

• Context: Deep learning models trained on source domain data (e.g., specific imaging devices) often suffer performance degradation when applied to target domains due to domain shifts (different scanners, protocols).
• Constraint: In SFDA, the source data is unavailable during adaptation to protect patient privacy; only the pre-trained source model and unlabeled target data are accessible.
• Current Limitations: Recent approaches utilize Vision-Language (ViL) foundation models (e.g., CLIP, FLAIR) to assist adaptation. However, the authors identify two critical flaws in existing ViL-based SFDA methods:
  1. Forgetting of Superior Predictions: While Mutual Information (MI) loss is used to align target and ViL predictions, it inadvertently causes the target model to "forget" its own high-confidence, correct predictions, leading to accuracy drops in specific classes.
  2. Lack of Fine-Grained Knowledge: Existing methods only utilize global image-level features from ViL models, ignoring the rich, patch-wise (local) knowledge regarding lesion locations (e.g., exudates, drusen) which is crucial for medical diagnosis.

2. Methodology: Forgetting-Resistant and Lesion-Aware (FRLA)

The authors propose the FRLA framework, which integrates two novel modules to overcome the identified limitations.

A. Forgetting-Resistant Adaptation (Dual Mutual Information Loss)

To prevent the target model from losing its reliable predictions during the distillation of ViL knowledge:

• Memory Bank: The system periodically caches confident predictions from the target model in a memory bank.
• Dual MI Loss:
  1. Standard MI ($L_{dis}$): Maximizes mutual information between the current target model predictions and the ViL model predictions (standard distillation).
  2. Forgetting-Resistant MI ($L_{fr}$): Selects high-confidence samples from the memory bank and computes a second mutual information loss between the current target predictions and these historical confident predictions.
• Mechanism: This ensures that if the target model previously made a correct, confident prediction, the optimization process explicitly preserves that knowledge rather than overriding it with potentially noisy ViL guidance.
• Loss Function: $L_{im} = L_{dis} + L_{fr}$.

B. Lesion-Aware Adaptation (Adaptive Patch-Level Supervision)

To leverage the fine-grained diagnostic capabilities of ViL models:

• Patch-wise Prediction Generation: The ViL model generates patch-level probability maps (similar to Class Activation Maps) by projecting features before Global Average Pooling (GAP) onto text embeddings.
• Filtering & Rectification:
  • Compatibility Check: Patch predictions inconsistent with the confident image-level labels in the memory bank are discarded to avoid noise.
  • Class Imbalance Correction: The remaining patch probabilities are rectified using class weights to prevent the model from biasing toward dominant lesion classes.
• Adaptive Weighting: A patch-level loss ($L_{la}$) is applied using Mutual Information between target and ViL patch predictions. Crucially, the weight of this loss gradually attenuates to zero during the first half of training and remains zero thereafter. This allows the model to learn lesion locations early without being distracted from the primary image-level classification task in later stages.
• Total Loss: $L = L_{im} + L_{la}$.

3. Key Contributions

1. Identification of "Forgetting" Phenomenon: The paper highlights that standard MI-based distillation in SFDA can degrade the performance of the target model on specific classes, a previously overlooked issue.
2. Forgetting-Resistant Module: Introduces a memory-bank-based dual MI loss that explicitly protects high-quality historical predictions of the target model.
3. Lesion-Aware Module: Proposes a novel strategy to extract and utilize patch-wise supervision from foundation models, enabling the target model to localize lesions (critical for fundus diagnosis) rather than just classifying the whole image.
4. Adaptive Training Strategy: Implements a dynamic loss weighting scheme for patch supervision to balance fine-grained learning with global classification stability.

4. Experimental Results

The method was evaluated on two cross-domain scenarios using three datasets: ODIR (Source), FIVES, and VietAI (Targets). The task involved classifying four categories: Normal, Age-related Macular Degeneration (ARMD), Diabetic Retinopathy (DR), and Glaucoma (GLC).

• Performance: FRLA achieved the State-of-the-Art (SOTA) performance in both adaptation settings.
  • ODIR $\to$ FIVES: FRLA achieved 80.4% average accuracy, outperforming the previous best (DIFO) at 78.6% and the raw ViL model (FLAIR) at 73.5%.
  • ODIR $\to$ VietAI: FRLA achieved 80.8% average accuracy, significantly outperforming DIFO (78.4%) and FLAIR (72.1%).
• Ablation Studies:
  • Adding $L_{fr}$ (Forgetting-Resistant) significantly boosted accuracy, particularly preventing drops in Glaucoma and Normal class performance.
  • Adding $L_{la}$ (Lesion-Aware) improved early-stage learning and final accuracy, confirmed by Class Activation Maps (CAMs) showing better localization of lesions.
• Comparison: The method outperformed traditional SFDA methods (SHOT, COWA) and foundation-model-based methods (Co-learn, DIFO).

5. Significance

• Medical Relevance: By incorporating patch-wise supervision, the model mimics the diagnostic process of ophthalmologists who look for specific lesion patterns, leading to more robust and interpretable AI in retinal screening.
• Privacy Preservation: The SFDA approach ensures that sensitive source patient data is never exposed during the adaptation to new clinical sites, addressing a major barrier in medical AI deployment.
• Generalizability: The proposed "forgetting-resistant" mechanism offers a new perspective on how to safely distill knowledge from foundation models without destabilizing the target model's existing capabilities, applicable beyond just fundus imaging.