Colony Grounded SAM2: Zero-shot detection and segmentation of bacterial colonies using foundation models

1. Problem Statement

In microbiology, the quantification and identification of bacterial colonies on agar plates are critical for medical diagnostics, risk assessment, and food safety. While manual quantification is accurate, it is labor-intensive and time-consuming. Although Artificial Intelligence (AI) has shown promise in automating this task, existing deep learning models typically require task-specific, manually labeled datasets for training. Creating these datasets is resource-intensive and creates a bottleneck, limiting the deployment of AI solutions to new or diverse imaging settings (e.g., different agar types, bacterial species, or lighting conditions). There is a critical need for a robust, zero-shot solution that can generalize across unseen datasets without requiring further fine-tuning on new data.

2. Methodology

The authors propose Colony Grounded SAM2, a zero-shot inference pipeline that combines two pre-trained foundation models: Grounding DINO and Segment Anything Model 2 (SAM2). The pipeline operates in two sequential stages:

A. Detection (Grounding DINO)

• Base Model: The pipeline utilizes Grounding DINO, an open-vocabulary object detection model based on the DINO vision transformer, which uses grounded pre-training to understand objects in natural images.
• Domain Adaptation: Since microbiological agar plate images differ significantly from the natural images used in Grounding DINO's original pre-training, the authors performed a brief domain-specific fine-tuning.
  • Dataset: Fine-tuned using the AGAR dataset (18,000 photos, 336k colonies).
  • Strategy: To preserve the model's inherent zero-shot robustness, fine-tuning was limited to a single epoch.
• Function: This stage detects individual bacterial colonies and generates bounding boxes. It treats the task as a set of binary classification problems rather than requiring specific species classification at this stage.

B. Segmentation (SAM2)

• Base Model: The bounding boxes generated by Grounding DINO are fed directly into SAM2, a large-scale foundation model designed to segment any object in an image.
• Prompting Strategy: The bounding boxes serve as "prompts" to instruct SAM2 where to focus its attention.
• Zero-Shot Hypothesis: The authors hypothesized that because SAM2 is prompted with precise locations from the detector, no further domain adaptation or fine-tuning of SAM2 is necessary to achieve high-quality instance segmentation.

3. Key Contributions

• Zero-Shot Pipeline: Development of a fully functional pipeline capable of detecting and segmenting bacterial colonies on out-of-distribution (OOD) datasets without retraining on the target data.
• Domain Adaptation Strategy: Demonstrating that a single epoch of fine-tuning on a large, diverse dataset (AGAR) is sufficient to adapt a general-purpose foundation model to the specific domain of microbiology while retaining zero-shot capabilities.
• Open Access: The authors have released the entire pipeline, including model weights and code, to facilitate annotation and classification tasks in the microbiology community.
• Dual-Metric Evaluation: Introduction of a specific evaluation metric, Dice@detection, to isolate segmentation performance from detection errors, providing a clearer picture of the segmentation model's intrinsic capability.

4. Results

The model was evaluated on three open-access datasets: AGAR (fine-tuning), ADBC (detection evaluation), and Hemolysis segmentation (segmentation evaluation).

Detection Performance (ADBC Dataset)

• Metric: Mean Average Precision (mAP) with an Intersection-over-Union (IoU) threshold of 0.2 (chosen to account for loosely labeled ground truth in the dataset).
• Performance: Colony Grounded SAM2 achieved an mAP of 93.1%.
• Comparison: This outperformed five baseline models from the Colony-YOLO suite (ranging from 76.5% to 91.1%), including specialized models like Colony-YOLO (91.1%).
• Observation: The model demonstrated superior precision in drawing bounding boxes, particularly for small colonies, often outperforming the manual annotations themselves.

Segmentation Performance (Hemolysis Dataset)

• Overall Metric: The standard Dice score was 0.60.
• Conditional Metric: The Dice@detection score (segmentation quality given correct detection) was 0.85.
• Analysis: The gap between the overall Dice (0.60) and Dice@detection (0.85) indicates that the primary source of error lies in the detection phase (missing dense central colonies) rather than the segmentation phase.
• Qualitative Findings: The model produced high-quality masks. Errors were often due to the model detecting colonies that were not the primary focus of the plate or missing densely packed colonies in the center of the plate, which human experts also often overlook in favor of separable outer colonies.

5. Significance and Impact

• Scalability: This approach eliminates the need for expensive, time-consuming data annotation for every new bacterial species or agar type, enabling rapid deployment of AI tools in diverse microbiological settings.
• Human-in-the-Loop: The high precision of the bounding boxes makes the model an excellent pre-annotator, significantly reducing the workload for human experts who only need to verify or correct the AI's output.
• Robustness: The system demonstrates strong generalization across different imaging conditions, agar types (e.g., blood, chocolate, Trypticase Soy), and bacterial species, validating the efficacy of adapting foundation models for scientific domains.
• Resource Efficiency: By leveraging pre-trained foundation models with minimal fine-tuning, the research offers a cost-effective alternative to training large models from scratch for niche scientific applications.