GuiDINO: Rethinking Vision Foundation Model in Medical Image Segmentation

GuiDINO introduces a framework that leverages DINOv3 as a visual guidance generator to produce spatial guide masks via a lightweight TokenBook mechanism, effectively enhancing medical image segmentation across diverse datasets and backbones without requiring full fine-tuning of the foundation model.

The Gist

Imagine you are trying to find a specific, tiny needle in a massive, messy haystack.

In the world of medical imaging, the "haystack" is a complex scan (like an MRI or ultrasound), and the "needle" is a tumor or a polyp that a doctor needs to see clearly. Traditionally, to find this needle, we build a specialized robot (a Medical AI) trained from scratch just to look at haystacks. It learns the specific texture of hay and the shape of needles very well, but it's slow to teach and needs a lot of examples.

Recently, scientists built a super-smart, general-purpose robot (a Foundation Model, like DINOv3) that has seen everything in the world—cats, cars, landscapes, and clouds. It's incredibly good at understanding shapes and textures. However, if you just ask this general robot to find the medical needle, it gets confused. It doesn't know what a "medical needle" looks like, and retraining it to do so is expensive and requires huge amounts of data we don't always have.

GuiDINO is a clever new idea that says: "Why don't we let the super-smart robot be the guide, and let the specialized robot do the actual work?"

Here is how it works, using a simple analogy:

1. The "Flashlight" (The Guide Generator)

Think of the Foundation Model (DINOv3) as a flashlight. It doesn't know exactly what the needle is, but it's great at spotting where interesting things are. It scans the image and says, "Hey, look over here! There's a weird shape, a texture change, or a boundary."

In the paper, this flashlight is called the TokenBook. It takes the general "knowledge" the robot learned from the internet and turns it into a simple, glowing map (a Guide Mask). This map doesn't draw the final picture; it just highlights the rough area where the doctor should look.

2. The "Specialized Surgeon" (The Medical Backbone)

Now, imagine you have a highly skilled Surgeon (the Medical AI, like nnUNet). This surgeon has spent years learning specifically how to cut and stitch medical tissue. They know exactly how to handle the delicate details.

In the old way, we tried to force the Surgeon to also be a flashlight, which confused them and made them slower.
In GuiDINO, we let the Surgeon stay focused on their job. We just hand them the Flashlight Map created by the general robot.

3. The "Gatekeeper" (How they work together)

The Flashlight Map acts like a gatekeeper. When the Surgeon looks at the image, the Gatekeeper says, "Ignore the empty space on the left; focus your energy on the glowing spot on the right."

This allows the Surgeon to:

• Ignore distractions: They don't waste time looking at the background.
• Focus on details: They can use their specialized medical knowledge to define the exact edges of the tumor.
• Stay efficient: They don't need to be retrained from scratch; they just get a little nudge in the right direction.

The Result: A Perfect Team-Up

The paper tested this team-up on three different medical challenges:

• Polyps in the colon (like finding a small bump in a tunnel).
• Skin lesions (finding a spot on a photo of skin).
• Thyroid nodules (finding lumps in an ultrasound).

The findings were impressive:

• Better Accuracy: The team found the "needles" more accurately than the Surgeon working alone or the Flashlight working alone.
• Sharper Edges: Because the Surgeon could focus, the boundaries of the tumors were drawn much more precisely (like a sharp pencil line instead of a smudged crayon).
• No Heavy Lifting: They didn't need to retrain the super-smart robot. They just used its "intuition" to guide the specialist.

Why This Matters

Think of it like a GPS and a Local Driver.

• The Foundation Model is the GPS. It knows the general map of the world and can tell you, "The destination is roughly in this neighborhood."
• The Medical AI is the Local Driver. They know the specific streets, the potholes, and the one-way signs of the medical world.

Before GuiDINO, we tried to make the GPS drive the car (which is slow and expensive) or let the Local Driver guess the neighborhood (which is risky). GuiDINO lets the GPS shout, "Go that way!" and lets the Local Driver take the wheel with perfect precision.

In short: GuiDINO is a smart way to combine the "big picture" knowledge of AI with the "specialized skills" of medical AI, making medical scans easier to read without needing massive amounts of new data or computing power.

Technical Summary

1. Problem Statement

Vision Foundation Models (VFMs), such as DINOv3, have demonstrated remarkable capabilities in learning generalizable visual representations from large-scale natural image datasets. However, their direct application to medical image segmentation faces significant challenges:

• Domain Shift: The semantic features learned from natural images do not align directly with the specific characteristics of medical images (e.g., specific textures, modalities like ultrasound or endoscopy).
• Resource Constraints: Fully fine-tuning these massive models for medical tasks requires extensive computational resources and large annotated datasets, which are scarce in the medical domain.
• Loss of Inductive Bias: Replacing specialized medical architectures (like nnUNet) with generic foundation backbones often discards the inductive biases (e.g., convolutional priors) that make medical networks efficient and effective.

The core question addressed is: How can the rich token features of foundation models be utilized to guide medical segmentation without full fine-tuning or sacrificing the efficiency of dedicated medical architectures?

2. Methodology: The GuiDINO Framework

The authors propose GuiDINO (Guided-by-DINO), a framework that repositions the foundation model not as the segmentation backbone, but as a visual guidance generator. The pipeline consists of three main components:

A. Architecture Overview

1. Frozen Guide Generator (DINOv3): A pre-trained DINOv3 backbone extracts dense token features from the input medical image. Crucially, this backbone remains frozen (or optionally adapted via LoRA) to preserve generalization and save resources.
2. TokenBook Mechanism: This is the core innovation. It converts the abstract token features into a spatial guide mask.
   • It aggregates token-prototype similarities.
   • It learns a set of prototypes ($P$) and weights ($\alpha_i$) to map token features ($T_i$) to spatial regions of interest.
   • Formula: $G(x) = \sum_{i=1}^{N} \alpha_i \cdot \text{sim}(T_i, P)$.
   • The output is a spatial map highlighting relevant regions, effectively bridging the semantic gap between natural and medical domains.
3. Segmentation Backbone: A dedicated medical segmentation network (e.g., nnUNet, UNet, nnWNet) takes the original image and the generated guide mask as input. The guide mask acts as a gate, modulating feature activations to focus the network on relevant areas while preserving the backbone's native inductive biases.

B. Learning Objectives

The training employs a composite loss function to align the guide generation with the segmentation task:
$$L = L_{seg} + \lambda L_{guide}$$

• $L_{seg}$: Standard segmentation loss (e.g., Dice or Cross-Entropy) between the final prediction and ground truth.
• $L_{guide}$: A guide supervision loss (Binary Cross-Entropy) that forces the generated guide mask to align with the ground-truth segmentation regions.
• Optional Boundary Loss: A hinge loss can be added to sharpen fine structures and improve boundary delineation.

C. Parameter-Efficient Adaptation

The framework supports LoRA (Low-Rank Adaptation) on the DINOv3 guide backbone. This allows for lightweight tuning of the foundation model to better suit specific medical domains without full fine-tuning.

3. Key Contributions

1. New Perspective on VFMs: The paper advocates for using foundation models as spatial guidance generators rather than direct backbones. This leverages their visual feature extraction power while avoiding semantic discrepancies.
2. TokenBook Mechanism: A novel, lightweight module that translates general token representations into task-specific spatial masks, enabling the injection of foundation priors into medical networks.
3. Efficiency and Performance: GuiDINO achieves state-of-the-art or competitive results across diverse datasets while maintaining the efficiency of traditional medical architectures and avoiding the heavy computational cost of full backbone fine-tuning.

4. Experimental Results

The framework was evaluated on three diverse medical datasets: Kvasir-SEG (polyps), ISIC 2017 (skin lesions), and TN3K (thyroid nodules).

• Performance Gains:
  • GuiDINO-W (applied to nnWNet) outperformed standard baselines (nnUNet, SwinUNet, H2Former) and the SegDINO method across all metrics (IoU, Dice, HD95).
  • Notable improvements were seen in boundary delineation (lower HD95 scores), indicating sharper segmentation edges.
  • Ablation Study: Integrating GuiDINO significantly boosted the performance of weaker backbones (e.g., UNet on ISIC 2017 saw a +10.35% Dice improvement).
• LoRA Adaptation:
  • Adding LoRA to the DINOv3 backbone further improved performance in most cases (e.g., GuiDINO-W-LoRA achieved 92.09% Dice on Kvasir vs. 90.86% without LoRA).
  • However, the benefit of LoRA varied by dataset, suggesting its effectiveness depends on specific domain characteristics.
• Qualitative Analysis: Visualizations showed that the guide masks successfully highlighted the rough location of targets, helping the segmentation backbone focus on relevant regions, especially in low-contrast or complex structural cases.

5. Significance and Conclusion

GuiDINO offers a practical alternative to the prevailing trend of fully fine-tuning foundation models for medical imaging.

• Preservation of Inductive Bias: It allows specialized medical architectures to remain the primary learners of semantic features, while the foundation model handles spatial guidance.
• Resource Efficiency: By freezing the heavy backbone and using lightweight adapters (TokenBook/LoRA), it reduces the computational burden and data requirements.
• Generalizability: The approach demonstrates that foundation models can serve medical vision tasks effectively by acting as "guides" rather than "replacements," providing a new paradigm for integrating large-scale pre-trained knowledge into domain-specific medical AI.

Code Availability: The implementation is open-source at https://github.com/Hi-FishU/GuiDINO.