Revisiting foundation models for cell instance segmentation

1. Problem Statement

Cell instance segmentation is a critical task in microscopy image analysis, essential for applications like drug screening and developmental biology. While deep learning tools exist, the field faces two main challenges:

1. Fragmentation: There is a proliferation of specialized tools for specific tasks (e.g., nuclei vs. whole cells, light vs. electron microscopy), creating a burden on users to select and manage the correct model.
2. Adaptation of Foundation Models: General-purpose Vision Foundation Models (VFMs), particularly the Segment Anything Model (SAM) family (SAM, SAM2, SAM3), have shown promise. However, it remains unclear:
   • What is the optimal strategy for adapting SAM-style models to microscopy?
   • Do general-purpose models (like the newly released SAM3, trained on natural images and some microscopy data) render specialized microscopy foundation models obsolete?
   • How does training data diversity and modality affect performance?

Current automatic segmentation strategies for SAM (like Automatic Mask Generation or AMG) often fail on complex microscopy data due to overlapping cells, varying morphologies, and domain shifts.

2. Methodology

The authors propose a comprehensive evaluation framework and a novel inference strategy called Automatic Prompt Generation (APG).

A. Benchmarking Scope

The study evaluates a diverse set of foundation models across 36 datasets covering four modalities:

• Cell Segmentation: Fluorescence and Label-free (Phase Contrast/Brightfield).
• Nucleus Segmentation: Fluorescence and Histopathology (H&E staining).

Models Evaluated:

• General-Purpose: SAM (with AMG), SAM2 (found inferior in initial tests), and SAM3 (prompted with text like "cell" or "nucleus").
• Microscopy-Specific:
  • CellPoseSAM: Uses SAM's encoder with a CellPose-style decoder.
  • CellSAM: Uses SAM's encoder with a bounding box detector (CellFinder) to generate box prompts for SAM.
  • µSAM / PathoSAM: Fine-tuned SAM architectures with a custom decoder predicting foreground probabilities and distance maps, followed by a watershed algorithm (Automatic Instance Segmentation or AIS).

B. The Novel Method: Automatic Prompt Generation (APG)

The authors introduce APG, a new inference strategy designed to operate on top of fine-tuned SAM models (specifically µSAM and PathoSAM) without retraining.

• Core Concept: Instead of relying solely on the watershed algorithm (used in standard AIS) or fixed grid prompts (used in AMG), APG derives point prompts directly from the model's decoder predictions.
• Workflow:
  1. Prediction: The model predicts foreground probabilities ($fg$), normalized boundary distances ($db$), and center distances ($dc$).
  2. Thresholding: Binary masks are generated by thresholding these three maps.
  3. Component Extraction: Connected components are identified on the intersection of these masks.
  4. Prompt Derivation: For each component, a point prompt is derived by computing the maximum of the boundary distance transform (effectively finding the "center" of the object).
  5. Re-segmentation: These derived point prompts are fed back into the SAM prompt encoder and mask decoder to generate refined masks.
  6. Filtering: Overlapping masks are filtered using Non-Maximum Suppression (NMS) based on predicted Intersection over Union (IoU) scores.
• Advantage: Unlike AIS, which requires a single component per object (risking over/under-segmentation), APG can generate multiple prompts per object and filter them, offering greater flexibility for complex morphologies.

3. Key Contributions

1. Comprehensive Benchmark: The first extensive evaluation of SAM3 on microscopy data, alongside a rigorous comparison of SAM, SAM2, and state-of-the-art microscopy-specific models (CellPoseSAM, CellSAM, µSAM).
2. Automatic Prompt Generation (APG): A novel, training-free strategy that significantly improves instance segmentation performance on top of existing fine-tuned SAM models.
3. Strategic Insights:
   • Demonstrated that SAM3, despite being trained on some microscopy data, is not yet competitive with domain-specific foundation models.
   • Highlighted the sensitivity of SAM3 to text prompts (e.g., shape descriptors like "blob" or "dot" often outperformed biological terms like "nucleus").
   • Showed that the size and relevance of the training dataset are the primary drivers of performance, with models trained on large, diverse microscopy datasets (CellPoseSAM, µSAM) outperforming general models.

4. Results

The evaluation used Mean Segmentation Accuracy (mSA) across various IoU thresholds.

• Performance of APG:
  • APG consistently improved µSAM/PathoSAM performance compared to their native AIS strategy.
  • It achieved substantial gains in label-free microscopy (e.g., TOIAM dataset) and histopathology.
  • APG is competitive with CellPoseSAM, the current state-of-the-art, often ranking in the top 3 across all four modalities.
• Model Rankings:
  • Top Performers: CellPoseSAM and APG (on µSAM/PathoSAM) consistently ranked highest.
  • Mid-Tier: AIS (standard µSAM) and CellPose 3.
  • Lower Tier: CellSAM performed worse than other microscopy models (likely due to reliance on box detection).
  • General Models: SAM (AMG) and SAM3 performed significantly worse than specialized models, particularly on difficult tasks like label-free cell segmentation or dense nuclei. SAM2 was found inferior to SAM in this setting.
• SAM3 Findings:
  • SAM3 showed sensitivity to prompt choice. For example, describing shape ("irregular shape") often yielded better results than the correct biological term ("cell").
  • It failed to recognize specific biological terms like "nucleus" in some contexts without fine-tuning.

5. Significance and Future Directions

• Validation of Domain-Specific Adaptation: The paper confirms that while general foundation models are powerful, domain-specific fine-tuning combined with specialized inference strategies (like APG) remains essential for high-precision microscopy. General models like SAM3 are not yet a "drop-in" replacement for specialized tools.
• Efficiency of APG: APG offers a way to boost the performance of existing fine-tuned models without the computational cost of retraining, making it a highly practical tool for the bioimage analysis community.
• Future Work:
  • Extending APG to incorporate box prompts (which generally perform better than points).
  • Evaluating 3D segmentation capabilities, as current evaluation was limited to 2D slices of 3D data.
  • Investigating few-shot or zero-shot adaptation of SAM3 using example images rather than just text prompts.

In conclusion, the paper establishes that the future of microscopy segmentation lies not just in larger general models, but in specialized foundation models enhanced by intelligent, automatic prompting strategies like APG.