Technical Summary: Human and AI Collaboration for Pulmonary Nodule Segmentation (Hi-Seg)

Problem Statement

The study addresses the critical bottleneck in training robust artificial intelligence (AI) models for medical imaging: the scarcity of high-quality segmentation masks and the limited availability of expert medical annotators. While deep learning models and foundation models like the Segment Anything Model (SAM) have accelerated segmentation, fully automated approaches struggle with generalization in real-world clinical settings. Specifically, fully automated deep learning models often fail to distinguish lesions with textures similar to healthy tissue (e.g., ground-glass opacities), and foundation models like SAM are highly sensitive to prompt quality. In clinical practice, the ideal prompts (derived from ground-truth masks) required for high SAM performance are unavailable, leading to significant performance degradation (e.g., Dice scores dropping to ~42% for lung nodules without ideal prompts). Furthermore, blind reliance on AI can be misleading, necessitating expert review, which is time-consuming and inefficient. The central challenge is how to rapidly generate high-quality lesion segmentation masks that support robust AI training while accommodating the scarcity of expert annotators.

Methodology

The authors propose Hi-Seg, a human-in-the-loop segmentation framework built upon the Segment Anything Model (SAM) designed specifically for pulmonary nodule segmentation.

Framework Design

Hi-Seg integrates human semantic reasoning and iterative trial-and-error learning with SAM's rapid boundary delineation capabilities. The workflow operates as follows:

1. Initialization: Annotators (ranging from senior radiologists to non-medical personnel) view chest CT scans in axial, coronal, and sagittal planes.
2. Prompting: The annotator provides an initial prompt (a left-click inside the nodule) to generate a candidate mask.
3. Iterative Refinement: If the mask is unsatisfactory, the annotator refines it through additional prompts:
   • Left-click: Adds regions to the mask (if the nodule is under-segmented).
   • Right-click: Removes regions from the mask (if the mask spills into the background).
4. Feedback Loop: SAM updates the mask based on the new prompts and previous predictions. Annotators observe the feedback, learn which prompt placements yield high-quality masks, and progressively refine their strategies to converge on accurate segmentation with minimal interactions.
5. Technical Optimization: To ensure low-latency interaction, Hi-Seg pre-computes and reuses image embeddings generated by SAM's Vision Transformer (ViT) encoder, avoiding redundant computation during the iterative process.

Experimental Setup

• Datasets: The study utilized 1,179 patients across 12 centers.
  • Training/Internal Validation: The public LIDC-IDRI dataset (1,010 patients).
  • External Testing: A single-center cohort (904 patients) and a multi-center cohort (275 patients from 12 institutions).
• Ground Truth: Masks were generated by experienced radiologists following standardized workflows, including consensus voting and manual refinement.
• Participants: Five annotators with varying expertise were recruited: one senior radiologist (12 years experience), one junior medical student, and three non-medical annotators (who received 1.5 hours of radiology training).
• Comparators: Hi-Seg was compared against:
  • Five state-of-the-art deep learning models (U-Net, U-Net++, Attention U-Net, TransU-Net, nnU-Net).
  • 13 variants of SAM-based models (including SAM, HQ-SAM, MedSAM, MedSAM2) using pseudo-human prompts (automatically generated from ground-truth masks to simulate human interaction).
  • Prompt-free SAM (automatic segmentation without prompts).
  • Manual delineation by the same participants.

Key Contributions

1. Human-in-the-Loop Framework: The introduction of Hi-Seg, which leverages real human interactive prompts and iterative feedback rather than relying on static, pseudo-human prompts derived from ground truth.
2. Trial-and-Error Learning Mechanism: The demonstration that annotators can rapidly learn effective prompting strategies through iterative interaction, allowing non-experts to achieve expert-level performance.
3. Large-Scale External Validation: The first large-scale external validation of collaborative human-SAM segmentation across diverse clinical datasets and annotator backgrounds.
4. Ablation of Prompting Strategies: A systematic disentanglement of the contributions of human prompting, iterative interaction, and feedback-based refinement to segmentation performance.

Results

• Performance Accuracy: Across all annotator groups, Hi-Seg achieved a mean Dice score of approximately 85% (84.47% on single-center and 87.00% on multi-center datasets).
  • This outperformed the five deep learning models by 10–22%.
  • It outperformed the 13 SAM variants using pseudo-human prompts by 1–29%.
  • The best-performing pseudo-human model (MedSAM2 with prompt #4) achieved ~83% Dice, still inferior to Hi-Seg.
• Expertise Independence: Hi-Seg significantly narrowed the performance gap between experts and non-experts.
  • Non-medical annotators using Hi-Seg achieved accuracy comparable to the junior medical student's manual performance.
  • The junior medical student using Hi-Seg achieved accuracy comparable to the senior radiologist's manual performance.
  • Even the senior radiologist saw a statistically significant improvement (+1.43% Dice) when using Hi-Seg compared to manual annotation.
• Efficiency:
  • Hi-Seg reduced annotation time by approximately 30% for the junior medical student and non-medical annotators.
  • For the senior radiologist, annotation time increased slightly (+3.46%), attributed to the cognitive shift from direct delineation to prompt refinement and ceiling effects in expert manual performance.
• Subgroup Analysis: Hi-Seg demonstrated robust performance across various nodule characteristics (size, density, spiculation, etc.). However, ground-glass nodules remained the most challenging subgroup, where fully automated models and non-medical annotators had higher failure rates, indicating a need for expert oversight in complex cases.
• Ablation Findings: The study confirmed that performance gains arise not merely from human prompting but from the iterative, feedback-driven workflow. Prompt-free SAM achieved only ~45% Dice, while one-shot human prompting reached ~65%. The full Hi-Seg framework (iterative + feedback) achieved ~85%, highlighting the value of continuous error correction and knowledge accumulation.

Significance and Claims

The paper claims that human-in-the-loop segmentation represents a transformative approach for integrating foundation models into routine clinical practice. By enabling junior trainees and briefly trained non-medical personnel to collaborate effectively with AI, Hi-Seg offers a pathway to:

• Reduce Clinician Workload: By accelerating annotation and improving accuracy, even for experts.
• Enable Scalable Crowdsourcing: Facilitating large-scale medical image annotation by non-experts under a structured human-AI workflow.
• Enhance Safety and Reliability: By combining the speed of AI with the semantic reasoning and contextual understanding of humans, particularly for complex cases where fully automated models fail.

The authors conclude that while blind reliance on AI is misleading and expert annotators are scarce, a collaborative framework like Hi-Seg can facilitate the safe and efficient deployment of foundation models, potentially transforming clinical workflows and improving healthcare accessibility. The study acknowledges limitations, including the focus on 2D slices, the relatively small number of human participants, and the short-term nature of the learning interaction, suggesting future work should explore 3D interactions and long-term learning effects.