Are Vision Foundation Models Foundational for Electron Microscopy Image Segmentation?

Technical Summary: Are Vision Foundation Models Foundational for Electron Microscopy Image Segmentation?

Problem Statement
While Vision Foundation Models (VFMs) pre-trained on large-scale natural image datasets are increasingly applied to biomedical image analysis, it remains unclear whether their latent representations are sufficiently general to support effective transfer across heterogeneous microscopy datasets. Specifically, the authors investigate whether current VFMs can overcome substantial distribution shifts that occur even within the same imaging modality (Electron Microscopy, or EM). The study focuses on the task of mitochondria segmentation, a critical task in EM analysis, to determine if VFMs can serve as a "foundational" base for training a single robust model across different EM datasets, or if they remain strongly domain-specific.

Methodology
The authors conducted a controlled benchmark using two widely adopted public EM datasets: Lucchi++ and VNC. These datasets were selected for their established use in the literature, allowing for direct comparison with existing methods. The study evaluated three representative VFMs: DINOv2, DINOv3, and OpenCLIP.

The experimental design focused on two practical model adaptation regimes that preserve most pre-trained weights, avoiding full end-to-end fine-tuning:

1. Frozen-Backbone Adaptation: The VFM backbone weights are kept fixed (including normalization layers), and only a lightweight convolutional segmentation head is trained.
2. Parameter-Efficient Fine-Tuning (PEFT) via LoRA: The segmentation head is trained as above, but Low-Rank Adaptation (LoRA) adapters are injected into the backbone to allow for targeted, lightweight updates to the pre-trained weights.

The authors evaluated performance under two training regimes:

• Single-Dataset Training: Models trained and tested on either Lucchi++ or VNC individually.
• Paired-Dataset Training: Models trained on the union of both datasets to assess the ability to learn a unified representation for heterogeneous EM sources.

To diagnose the underlying causes of performance differences, the authors employed representation-space diagnostics, including Principal Component Analysis (PCA) of pooled embeddings, calculation of the Fréchet DINOv2 Distance (FD-DINOv2) to quantify distributional mismatch, and linear probes (logistic regression) to test the linear separability of the dataset labels.

Key Results

• Single-Dataset Performance: All three VFMs achieved strong segmentation performance (measured by foreground Intersection-over-Union, IoUfg) when trained and tested on a single dataset. LoRA-PEFT consistently improved in-domain performance by approximately 10 percentage points over the frozen-backbone baseline across all backbones.
• Paired-Dataset Failure: When training on the union of Lucchi++ and VNC, performance collapsed for all models, dropping to near-random levels (IoUfg < 0.06). LoRA provided only marginal gains in this setting, failing to recover robust performance.
• Domain Mismatch Analysis: Representation space diagnostics revealed a pronounced and persistent domain mismatch between Lucchi++ and VNC. Even after LoRA adaptation, the two datasets occupied disjoint regions in the feature space (FD-DINOv2 distance decreased from 79.32 to 45.02 but remained high), and linear probes could still distinguish the dataset origin with near-perfect accuracy (1.0). This indicates that the failure of paired training is driven by intrinsic domain shifts within the EM modality that lightweight adaptation cannot bridge.
• Comparison to State-of-the-Art: While fully supervised end-to-end baselines (e.g., U-Net variants) remained the strongest performers, VFM-based models with LoRA were competitive in single-dataset scenarios. However, they could not match the performance of specialized models in the paired setting.

Key Contributions

1. Benchmarking: Provided a controlled benchmark of three major VFMs (DINOv2, DINOv3, OpenCLIP) on an EM segmentation task under frozen-backbone and LoRA-PEFT adaptation.
2. Empirical Evidence of Failure: Demonstrated that paired training across heterogeneous datasets from the same microscopy modality (EM) severely fails, despite the visual similarity of the images.
3. Domain-Mismatch Analysis: Linked this failure to persistent representation separation between datasets, showing that PEFT reduces but does not eliminate the domain gap.
4. Performance Context: Reported performance comparisons against representative non-VFM baselines from the EM segmentation literature.

Significance and Claims
The paper concludes that while VFMs can deliver competitive results for EM segmentation within a single domain using lightweight adaptation, they are not yet "foundational" enough to support a single robust model across heterogeneous EM datasets without additional domain-alignment mechanisms. The authors argue that the substantial intra-modality shift within EM imaging is a dominant factor hindering generalization, and that current PEFT strategies are insufficient to align these distinct distributions.

The study suggests that the bottleneck for adopting VFMs in microscopy is not just the shift from natural to biomedical images, but the variability inherent within microscopy datasets themselves. Consequently, existing domain alignment mechanisms, largely developed for natural images, may be inadequate for the more variable image domains found in microscopy. The authors call for further methodological developments to enable VFMs to generalize effectively over imaging modalities and within them.