Interactive Medical-SAM2 GUI: A Napari-based semi-automatic annotation tool for medical images

This paper introduces Interactive Medical-SAM2 GUI, an open-source, Napari-based desktop application that streamlines semi-automatic 3D medical image annotation by integrating SAM2-style mask propagation with interactive box and point prompting to enable efficient, cohort-oriented workflows for research.

The Gist

Imagine you are a doctor or a researcher trying to map out a tumor inside a patient's body. The medical scan (like an MRI or CT) isn't just a single picture; it's a stack of hundreds of thin slices, like a loaf of bread. To train an AI to recognize diseases, humans have to draw a line around the tumor on every single slice.

Doing this manually is like trying to paint a mural by hand, one tiny square inch at a time. It's slow, boring, and expensive.

This paper introduces a new tool called Interactive Medical-SAM2 GUI. Think of it as a "Smart Paintbrush" that lives inside a popular image-viewing program called Napari. Here is how it works, using some simple analogies:

1. The "Video" Trick (How it thinks)

Most AI models look at one slice at a time. But this tool treats the entire 3D scan like a short movie.

• The Analogy: Imagine you are drawing a character in an animation book. Instead of drawing the character from scratch on every single page, you draw them clearly on the first page and the last page. The tool then uses "Magic" (AI) to fill in all the pages in between, guessing how the character moves and changes shape.
• In the tool: You draw a box around the tumor on the first slice where it appears and the last slice where it appears. The AI (Medical-SAM2) automatically fills in the hundreds of slices in the middle.

2. The "Assembly Line" Workflow

Usually, researchers have to open one patient's file, label it, close it, find the next file, and repeat.

• The Analogy: This tool is like a conveyor belt in a factory. You load a whole box of patient scans (a "cohort") onto the belt. You work on Patient A, hit "Next," and Patient B slides right into place. You don't have to hunt for files; the tool keeps the line moving so you can focus on labeling.

3. The "Refinement" Step

The AI is great, but it's not perfect. Sometimes it might draw a line a little too wide or miss a tiny spot.

• The Analogy: Think of the AI as a draftsman who sketches the outline quickly. You are the art director. You look at the sketch, and if a line is wrong, you just tap a few dots (point prompts) to say, "Fix this spot here." The tool instantly corrects the drawing.
• The Process: You get the AI to do 90% of the heavy lifting, you do the final 10% of "touch-ups," and then you hit save.

4. The "Instant Report"

Once you are done, you don't just get a picture; you get data.

• The Analogy: It's like a 3D printer that not only prints the object but also weighs it and measures its volume instantly.
• The Result: As soon as you save the labels, the tool tells you exactly how big the tumor is (in cubic millimeters) and lets you spin a 3D model of it on your screen to check the shape.

Why is this a big deal?

• It's Local: You don't have to upload sensitive patient data to the cloud (which is often against hospital privacy rules). Everything happens on your own computer.
• It's Open: It's free software that anyone can use and improve.
• It's Fast: It turns a job that used to take hours into something that takes minutes.

In short: This tool takes the boring, repetitive work of drawing medical maps and turns it into a fast, interactive game where the AI does the heavy lifting, and the human just guides the ship. It helps researchers build better medical AI faster, which eventually leads to better care for patients.

Technical Summary

Based on the provided paper, here is a detailed technical summary of Interactive Medical-SAM2 GUI:

1. Problem Statement

While voxel-level annotation is critical for developing and validating medical imaging algorithms, the process remains a bottleneck due to its high cost and time consumption, particularly for 3D scans containing hundreds of slices.

• Limitations of Existing Tools: Traditional platforms (e.g., ITK-SNAP, 3D Slicer) offer robust visualization but rely heavily on manual work for cohort-scale consistency.
• Limitations of Current AI Integrations: Existing AI-assisted tools (e.g., MedSAMSlicer, napari-sam) often focus on per-slice interaction. They lack a unified, local-first workflow that seamlessly combines navigation across a patient cohort, sparse-prompt propagation, interactive correction, and quantitative export in a single pipeline.
• Clinical Constraints: Web-based labeling solutions face barriers regarding data governance, privacy, and institutional de-identification requirements, necessitating local-first workflows for routine clinical research.

2. Methodology

The authors developed Interactive Medical-SAM2 GUI, an open-source desktop application built on the Napari multi-dimensional viewer and PyTorch.

• Core Architecture:

  • Model: Integrates Medical-SAM2 (a medical adaptation of SAM2) which treats 3D medical volumes as a sequence of 2D slices (similar to video frames). This enables memory-based propagation of masks from sparse prompts across the entire volume.
  • Data Handling: Uses SimpleITK for robust I/O, ensuring the preservation of critical image geometry (spacing, origin, direction) for DICOM series and NIfTI volumes.
  • Preprocessing: Includes optional N4 bias-field correction for MRI data.

• Workflow Design (The "Cohort-Oriented" Pipeline):

  1. Cohort Navigation: Users select a single root folder containing multiple patient studies. The tool allows sequential annotation with explicit "proceed" or "skip" actions, eliminating the need to manually browse individual files.
  2. Box-First Prompting: The primary interaction for initialization is the bounding box.
     • Single-Object: Users place boxes on the first and last slices where an object appears; the system propagates the mask through intermediate slices.
     • Multi-Object: Supports multiple objects within a volume by allowing prompts on relevant slices for each object.
  3. Refinement: Users can add point prompts to correct or refine predictions on specific slices (e.g., adding small missed areas).
  4. Correction & Finalization: A "prompt-first" workflow encourages obtaining the best possible segmentation via AI propagation, followed by a final manual correction step to "lock in" the label before saving.
  5. Export & Visualization: Upon saving, the tool automatically computes per-object volumetry (e.g., tumor volume) and generates 3D volume renderings for rapid visual inspection.

3. Key Contributions

• Unified Local Pipeline: Bridges the gap between AI inference and clinical annotation workflows by providing a single, local-first environment for navigation, propagation, correction, and export.
• 3D Propagation Strategy: Effectively adapts the SAM2 video segmentation paradigm to medical 3D volumes, allowing users to annotate entire 3D structures by interacting only with the first and last slices (or sparse slices) rather than every slice.
• Clinician-Centric UX: Designed specifically for research annotation with features like batch processing of patient cohorts, explicit skip/proceed logic, and automated volumetric reporting, reducing the cognitive load on annotators.
• Open-Source Accessibility: The tool is released as open-source software (GitHub: SKKU-IBE/Medical-SAM2GUI), built on standard, widely adopted libraries (Napari, PyTorch, SimpleITK).

4. Results

• Functional Validation: The paper demonstrates the tool's capability to handle standard DICOM and NIfTI inputs, successfully propagating masks across 3D volumes based on sparse prompts.
• Quantitative Output: The system successfully generates per-object volumetry and 3D renderings, facilitating tasks like tumor burden tracking.
• Geometry Preservation: The use of SimpleITK ensures that the spatial integrity of the original medical images is maintained in the saved masks, a critical requirement for downstream algorithmic validation.

5. Significance

This work addresses a critical practical limitation in medical AI research: the lack of efficient, scalable, and privacy-compliant tools for 3D annotation.

• Efficiency: By leveraging prompt-based propagation, it significantly reduces the time and effort required for manual 3D labeling compared to slice-by-slice methods.
• Reproducibility: The standardized "box-first" and "prompt-first correction" workflow promotes consistent interaction patterns across different users and studies.
• Research Enablement: It lowers the barrier for researchers to generate high-quality, voxel-level ground truth datasets for training and validating medical imaging algorithms without relying on complex web infrastructures or expensive commercial software.
• Scope: The authors explicitly position the tool for research annotation workflows, clarifying that it is not intended for clinical decision support, thereby aligning with regulatory and ethical standards for medical software development.