Opportunistic Promptable Segmentation: Leveraging Routine Radiological Annotations to Guide 3D CT Lesion Segmentation

Technical Summary: Opportunistic Promptable Segmentation with SAM2CT

Problem Statement
The development of robust machine learning models for CT imaging is constrained by the scarcity of large-scale, high-quality 3D segmentation datasets. While clinical Picture Archiving and Communication Systems (PACS) contain vast repositories of CT images and reports, generating 3D segmentations typically requires extensive, manual contouring by radiologists, which is costly and time-consuming. Conversely, radiologists routinely generate sparse annotations during standard clinical reads—such as line measurements (e.g., lesion diameter) and arrows to highlight findings—which are stored as Grayscale Softcopy Presentation State (GSPS) DICOM objects. Currently, these rich, existing annotations are underutilized for training 3D segmentation models because no existing promptable segmentation models can directly interpret these specific GSPS formats (arrows and lines) to generate 3D masks.

Methodology
The authors propose a new paradigm termed Opportunistic Promptable Segmentation, which leverages existing GSPS annotations as prompts to automatically generate 3D segmentations. To enable this, they introduce SAM2CT, the first promptable segmentation model designed to convert radiologist annotations (arrows and lines) into 3D CT segmentations.

• Architecture: SAM2CT is built upon the Segment Anything Model 2 (SAM2) architecture. It retains the core components of SAM2 (image encoder, prompt encoder, memory attention, memory encoder, memory bank, and mask decoder) but introduces two critical modifications:
  1. Enhanced Prompt Encoder: The prompt encoder is extended with three learnable parameters to interpret line endpoints and arrow start/end points, allowing the model to process GSPS-style inputs directly.
  2. Memory-Conditioned Memories (MCM): In standard SAM2, memory tokens are generated using unconditioned image embeddings. SAM2CT modifies this by feeding conditioned image embeddings (those that have undergone memory attention) into the memory encoder. This creates memory tokens that are more specific to the target lesion, improving segmentation consistency across 3D volumes.
• Training Strategy: The model was fine-tuned end-to-end on a curated dataset of five public CT lesion segmentation datasets (covering 8 tumor types). To simulate the opportunistic use of GSPS data, the authors synthesized arrow and line prompts from ground-truth masks during training. The training utilized 8-slice sub-volumes to manage computational costs.
• Comparators: The study compares SAM2CT against a fine-tuned SAM2 variant (SAM2 FT) that supports arrow/line prompts but lacks MCM, as well as standard UNet-based architectures (DynUNet and Swin UNETR) where prompts are encoded as binary mask channels.

Key Contributions

1. Novel Paradigm: The paper defines and demonstrates "Opportunistic Promptable Segmentation," a method to mine historical PACS GSPS data for 3D dataset generation without additional physician effort.
2. SAM2CT Model: The development of the first model capable of interpreting clinical line and arrow annotations to produce 3D segmentations.
3. Architectural Innovation: The introduction of Memory-Conditioned Memories (MCM), a strategy shown to improve 3D volumetric segmentation by encoding lesion-specific features into the memory bank. The authors demonstrate that MCM improves performance not only in SAM2CT but also when applied as a test-time adaptation to other SAM2-based models without further fine-tuning.
4. Synthetic Prompt Generation: A methodology for synthesizing realistic arrow and line prompts from ground-truth masks to train models on data that originally lacked such annotations.

Results

• Benchmark Performance: On public hold-out test sets, SAM2CT achieved state-of-the-art performance among promptable models. It attained a Dice Similarity Coefficient (DSC) of 0.649 for arrow prompts and 0.757 for line prompts, outperforming SAM2 (FT), MedSAM2, DynUNet, and Swin UNETR across various lesion types (kidney, liver, pancreas, lymph nodes, etc.).
• Clinical Utility (Oncology): Applied to 60 real-world oncology cases from clinical PACS, radiologists rated SAM2CT-generated masks as clinically acceptable (requiring no modification) or requiring only minor adjustments in 87% of cases.
• Zero-Shot Generalization: The model demonstrated strong zero-shot performance on Emergency Department (ED) findings not present in the training data. It achieved a DSC of 0.610 for abscesses and 0.725 for gallstones using original GSPS prompts.
• RECIST Accuracy: SAM2CT achieved the lowest average percent error in RECIST 1.1 measurements (11.1%) compared to other models, indicating high clinical relevance for tumor burden quantification.
• Test-Time Adaptation: Applying the MCM strategy to other models (SAM2, MedSAM2) at test time yielded significant performance gains without additional training, validating MCM as a robust architectural improvement for 3D medical segmentation.

Significance and Claims
The authors claim that SAM2CT represents a scalable solution for generating large-scale 3D CT segmentation datasets by unlocking the "wealth of prompts" already stored in PACS systems. By converting routine, sparse radiologist annotations into dense 3D masks, this approach eliminates the need for costly manual contouring for dataset creation. The paper posits that this method facilitates the construction of multimodal, grounded datasets (linking 3D masks with free-text reports), which are essential for developing advanced vision-language models in radiology. The study concludes that opportunistic promptable segmentation is a viable strategy to accelerate medical AI development by leveraging existing clinical workflows.