FunduSegmenter: Leveraging the RETFound Foundation Model for Joint Optic Disc and Optic Cup Segmentation in Retinal Fundus Images

1. Problem Statement

The segmentation of the Optic Disc (OD) and Optic Cup (OC) in retinal fundus images is critical for diagnosing systemic diseases (e.g., glaucoma, diabetes) and discovering biomarkers. However, current state-of-the-art supervised Convolutional Neural Network (CNN) models face three major limitations:

• Data Scarcity & Annotation Cost: Medical image annotation is labor-intensive, leaving many large datasets unlabeled.
• Poor Generalization: Supervised models often fail when tested on out-of-distribution (OOD) data (e.g., images from different cameras or clinical centers).
• Synthetic Data Reliance: Many models rely on synthetic data augmentation (phantoms) to improve performance, which reduces clinical trust compared to models trained on real images.

While Foundation Models (FMs) like RETFound (trained on nearly 1 million unlabeled retinal images via Self-Supervised Learning) have shown success in classification tasks, they had not yet been adapted for segmentation tasks, particularly for the complex joint segmentation of OD and OC.

2. Methodology: FunduSegmenter

The authors propose FunduSegmenter, a novel architecture that adapts the RETFound foundation model for segmentation. The model freezes the pre-trained RETFound encoder (ViT-large) and introduces a series of custom modules to bridge the gap between general image representation and specific segmentation tasks.

Core Architecture Components:

1. Pre-adapter:
   • Function: Accommodates input images of varying sizes without relying on position interpolation (which is limited in size and efficiency).
   • Design: A lightweight, inverted residual architecture (inspired by MobileNetV2) with ~40k parameters. It processes input images and downsamples them to 224×224 while preserving features through a weighted residual connection.
2. RETFound Encoder (Frozen):
   • Uses the pre-trained ViT-large encoder from RETFound to extract high-level latent representations from the fundus images.
3. Segmenter Decoder:
   • Adapts the decoder from the "Segmenter" model (originally for natural images) to work with the RETFound encoder. It utilizes a Mask Transformer to generate mask maps.
4. Post-adapter:
   • Function: Addresses the issue of unclear boundaries between OD and OC, which often causes errors in direct bilinear upsampling.
   • Design: A progressive upsampling module that reconstructs spatial maps from 14×14 to the final image size (224×224) using stacked CNN blocks and bicubic interpolation, improving convergence speed and accuracy.
5. Skip Connections with CBAM:
   • Extracts feature maps from intermediate layers (6, 12, 18, 24) of the RETFound encoder.
   • These maps are upsampled and processed through Convolutional Block Attention Modules (CBAM) before being concatenated with the decoder features. This injects multi-scale information, crucial for segmenting the smaller Optic Cup.
6. ViT Block Adapter:
   • Function: Allows the frozen encoder to adapt to the specific segmentation task without full fine-tuning.
   • Design: A bottleneck adapter (Linear → GELU → Linear) inserted between the Layer Normalization and Multi-Head Attention blocks of the transformer. This enables the model to tune specific parameters while retaining the strong general representations learned during pre-training.

Training Strategy:

• Loss Function: A combination of Dice Loss and Cross-Entropy Loss.
• Datasets: Trained and evaluated on a proprietary dataset (GoDARTS) and four public datasets (IDRiD, Drishti-GS, RIM-ONE-r3, REFUGE).
• Experiments: Conducted Internal Verification, External Verification (cross-dataset testing), and Domain Generalization experiments.

3. Key Contributions

1. First RETFound Adaptation for Segmentation: To the authors' knowledge, this is the first work to adapt RETFound for anatomical structure segmentation in fundus images.
2. Novel Adapter Architecture: The integration of Pre-adapter, Post-adapter, CBAM skip connections, and ViT Block Adapters creates a robust framework for translating foundation model representations into precise segmentation masks.
3. Superior Generalization: The model demonstrates exceptional stability across in-distribution and out-of-distribution data, outperforming specialized baselines without relying on synthetic data augmentation.
4. Generalizability: The proposed modules are designed to be generic, applicable to other Vision Transformer (ViT) based foundation models trained via Self-Supervised Learning (SSL).

4. Results

The model was evaluated against six state-of-the-art baselines: nnU-Net, DUNet, TransUNet, DoFE, RAM-DSIR, and TVConv.

• Internal Verification:
  • Achieved an average Dice Similarity Coefficient (DSC) of 90.51%.
  • Outperformed all baselines significantly (e.g., nnU-Net: 82.91%, DUNet: 89.17%, TransUNet: 87.91%).
  • Achieved the best scores for OD segmentation on Drishti-GS (97.34%) and OC on REFUGE (87.19%).
• External Verification:
  • When tested on unseen datasets, FunduSegmenter's average results were approximately 3% higher than the best baseline.
  • Demonstrated high stability; unlike nnU-Net which dropped to 17.96% DSC on the GoDARTS dataset (due to distribution shift), FunduSegmenter maintained ~79% DSC.
• Domain Generalization:
  • Performed competitively against specialized domain generalization models (DoFE, RAM-DSIR, TVConv).
  • While slightly behind TVConv in some metrics, FunduSegmenter achieved this without using prior positional knowledge (like OD center cropping), making it more adaptable to complex layouts (e.g., vessel segmentation).
• Ablation Studies:
  • Confirmed that the Post-adapter and ViT Block Adapter were critical for solving convergence issues (where baseline models stalled for tens of thousands of epochs) and improving accuracy.
  • The Pre-adapter proved more efficient and effective than standard position interpolation for handling variable input sizes.

5. Significance and Translational Relevance

• Clinical Stability: The model provides stable segmentation across diverse clinical centers and imaging devices, a prerequisite for automated biomarker discovery and longitudinal monitoring.
• Efficiency: With only 35.57M tunable parameters, the model is computationally efficient compared to training massive models from scratch.
• Paradigm Shift: This work validates that foundation models pre-trained on massive unlabeled datasets can be effectively fine-tuned for complex medical segmentation tasks using limited labeled data, reducing the dependency on expert annotation and synthetic data.
• Future Impact: The architecture serves as a benchmark for adapting SSL foundation models to medical imaging and opens pathways for applying these techniques to other retinal tasks (vessel segmentation) and systemic disease prediction.

Code Availability: The source code and trained weights are publicly available at: https://github.com/JusticeZzy/FunduSegmenter.