CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation

1. Problem Statement

Real-time semantic segmentation is critical for robotic-assisted and computer-guided ophthalmic surgeries (e.g., cataract and glaucoma procedures). However, current solutions face three major bottlenecks:

• Domain Mismatch: General-purpose foundation models like Meta's Segment Anything Model 2 (SAM-2) struggle with the unique visual complexities of surgical videos, such as transparent tissues, glare, low contrast, and occlusion by instruments. Their zero-shot performance is often insufficient for clinical precision.
• Annotation Costs: Creating high-quality ground-truth masks for surgical videos is labor-intensive, often requiring expert clinicians to label frames manually for hours. This limits the scalability of training datasets.
• Limited Generalization: Existing medical adaptations of SAM-2 often focus on specific procedures or lack robustness when applied to unseen surgical phases or different anterior segment surgeries (e.g., glaucoma trabeculectomy).

2. Methodology

A. Data Sources

• Primary Dataset: Cataract-1K, containing 30 cataract surgery videos (2,256 annotated frames) recorded at Klinikum Klagenfurt. It includes 12 classes covering anatomical structures (Iris, Pupil, Lens) and surgical instruments (e.g., Phacoemulsifier Tip, Forceps).
• External Validation: CaDIS dataset (25 videos, 4,613 frames) for cross-dataset testing.
• Zero-Shot Evaluation: Two publicly available trabeculectomy (glaucoma surgery) videos not seen during training.

B. Model Architecture: CataractSAM-2

The authors propose CataractSAM-2, a domain-adapted extension of Meta's SAM-2.

• Architecture: Retains the original SAM-2 Image Encoder (Vision Transformer) to preserve general visual priors.
• Fine-Tuning Strategy: Only the Prompt Encoder and Mask Decoder are fine-tuned. This approach balances computational efficiency with the ability to adapt to the specific spatial complexity and visual challenges of cataract surgery (e.g., distinguishing tools from transparent tissues).
• Training Configuration:
  • Optimizer: AdamW (LR = 0.0001).
  • Mixed-precision training (PyTorch AMP).
  • Gradient accumulation (every 4 steps) to simulate larger batches.
  • Warm-up phase on the first 5 frames of each video to stabilize early predictions.

C. Interactive Annotation Framework

To address the annotation bottleneck, the authors developed a human-in-the-loop pipeline integrated into Jupyter notebooks:

1. Sparse Prompting: Users provide minimal input (positive/negative points) on a single frame via an intuitive UI.
2. Video Propagation: The system uses the SAM2VideoPredictor to propagate these masks across all subsequent frames using temporal continuity.
3. Iterative Correction: Users can visualize results and correct oversegmentation or undersegmentation by adding new points to specific frames, which are then re-propagated.
4. Output: Automated generation of high-quality binary masks saved as PNG files.

3. Key Contributions

1. CataractSAM-2: The first domain-adapted SAM-2 variant specifically fine-tuned for real-time, high-precision segmentation in anterior segment ophthalmic surgeries.
2. Scalable Annotation Tool: An interactive framework that reduces annotation time from hours to minutes per video volume by combining sparse prompts with automated video propagation.
3. External Validation: Successful validation on the CaDIS dataset, demonstrating transferability to diverse instruments and surgical phases beyond the training data.
4. Cross-Procedural Generalization: Demonstration of zero-shot generalization to trabeculectomy (glaucoma) procedures, a surgery type not included in the training set.
5. Open-Source Release: Release of pretrained weights, inference scripts, and the interactive annotation notebook to foster reproducibility.

4. Results

Quantitative Performance

• Metrics: Evaluated using Intersection over Union (IoU) and Pixel Accuracy (PAC).
• Comparison: CataractSAM-2 significantly outperformed baseline models (Base SAM-2, MedSAM-2, SurgSAM-2) across all 6 test videos in the Cataract-1K set.
  • It achieved the highest mean IoU and PAC with the lowest variability, particularly in delineating tool boundaries.
  • Inference Speed: Achieved 15 FPS on an NVIDIA A100 GPU, suitable for real-time intraoperative deployment.
• CaDIS Validation: On the external CaDIS dataset, the model maintained high performance with IoU ranging from 0.88–0.95 and PAC from 0.96–0.99, confirming robustness across heterogeneous data.

Qualitative & Zero-Shot Results

• Trabeculectomy Generalization: The model successfully segmented instruments and anatomical structures in glaucoma surgery videos (e.g., conjunctival incision, scleral flap dissection) without retraining.
• Challenges Handled: The model maintained accuracy despite glare, low contrast, and overlapping geometries, though minor oversegmentation was observed in high-reflectivity zones.

5. Significance and Impact

• Foundation for Surgical AI: CataractSAM-2 establishes a new foundation for expanding anterior ophthalmic surgical datasets, moving beyond the limitations of manual labeling.
• Robotic Integration: The real-time capability (15 FPS) and high precision make it a viable component for robotic-assisted surgery, surgical navigation, and augmented reality (AR) overlays.
• Efficiency: The interactive annotation tool democratizes dataset creation, allowing researchers to generate large-scale, high-quality ground-truth data with minimal expert effort.
• Future Directions: The authors identify the need for multi-center data to improve diversity, potential encoder fine-tuning for glare handling, and model compression (pruning/quantization) for edge deployment in closed-loop robotic systems.

In conclusion, CataractSAM-2 bridges the gap between general vision foundation models and specialized medical applications, offering a scalable solution for both real-time surgical assistance and rapid dataset generation.