💻 computer science

SAMannot: A Memory-Efficient, Local, Open-source Framework for Interactive Video Instance Segmentation based on SAM2

SAMannot is a privacy-preserving, local, and open-source framework that integrates an optimized version of SAM2 with a human-in-the-loop workflow to enable efficient, interactive video instance segmentation for research-grade dataset creation.

Original authors: Gergely Dinya, András Gelencsér, Krisztina Kupán, Clemens Küpper, Kristóf Karacs, Anna Gelencsér-Horváth

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gergely Dinya, András Gelencsér, Krisztina Kupán, Clemens Küpper, Kristóf Karacs, Anna Gelencsér-Horváth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a film editor trying to cut out a specific actor from a 2-hour movie scene by scene. In the old days, you'd have to manually trace the actor's outline on every single frame with a digital pen. That would take weeks, your hand would cramp, and if you made a mistake, you'd have to start over.

Now, imagine a magical assistant that can instantly guess the outline of the actor just because you tapped a point on their shoulder. That's what SAM2 (Segment Anything Model 2) does. It's a super-smart AI that can find objects in images.

However, there's a catch: running this magical assistant on a whole movie is like trying to power a spaceship with a AA battery. It eats up so much computer memory that it crashes, and most of the time, you have to upload your private movie to a big cloud server to get it done, which is risky if your movie contains sensitive data (like medical records or private animal research).

Enter SAMannot.

Think of SAMannot as a smart, local workshop built specifically to make that magical assistant work efficiently on your own computer, without needing the cloud or a supercomputer. Here is how it works, using some everyday analogies:

1. The "Chunking" Strategy (Memory Management)

Imagine trying to read a 1,000-page book all at once; your brain would get overwhelmed. Instead, you read it one chapter at a time.

The Problem: Standard AI tries to "read" the whole video at once, crashing your computer's memory.
The SAMannot Solution: It breaks the video into small "chunks" (like chapters). It processes one chunk, saves the work, clears its memory, and moves to the next. This keeps your computer running cool and fast, even on a standard laptop or desktop.

2. The "Lock and Refine" Workflow

Imagine you are painting a mural. You don't want to accidentally smudge the part you just finished while working on the next section.

The Feature: SAMannot lets you set "barrier frames" (checkpoints). Once you are happy with a section, you "lock" it. The AI won't touch those frames again, even if you ask it to process the whole video. This gives you total control to fix mistakes in one area without ruining the rest.

3. The "Auto-Prompt" Bridge

Imagine you are walking a dog on a long leash. If you stop to tie your shoe, the dog might wander off. You need a way to keep the dog on track between your stops.

The Feature: When you finish one "chunk" of the video and move to the next, the AI usually gets confused about where the object is. SAMannot has a clever trick: it looks at the shape of the object in the last frame, finds the "skeleton" (the key points), and automatically places a "guide point" for the next chunk. It's like the AI leaves a breadcrumb trail for itself so it doesn't get lost when you switch sections.

4. Privacy and Ownership (Local & Open)

Many tools today are like renting a luxury apartment: you can live there, but you have to pay rent, and the landlord can see what you're doing.

The SAMannot Difference: This tool is open-source (the blueprints are free for anyone to see) and local (it runs entirely on your computer). It's like building your own house. Your data never leaves your machine, making it perfect for sensitive research (like tracking endangered animals or analyzing patient data) where privacy is non-negotiable.

5. The "Human-in-the-Loop" Team

SAMannot isn't trying to replace you; it's trying to be your best assistant.

How it works: You tell the AI, "That's a bird," by clicking on it. The AI does the heavy lifting of drawing the outline for the next 100 frames. If the bird flies behind a tree and the AI gets confused, you just click again to correct it. It turns a task that used to take weeks into something that takes hours.

Why Does This Matter?

The researchers tested this on videos of animals, cars, and people. They found that:

It's fast: It runs smoothly on standard graphics cards.
It's accurate: It gets the outlines right almost as well as a human expert.
It's cheap: No monthly subscription fees.
It's safe: Your data stays on your hard drive.

In summary: SAMannot takes a powerful, expensive, and memory-hungry AI tool and wraps it in a user-friendly, memory-efficient package that lets regular researchers do professional-grade video editing right on their own desks, keeping their secrets safe and their wallets happy.

1. Problem Statement

The field of video instance segmentation (VIS) faces significant bottlenecks in research and application workflows:

Privacy and Compliance: Many domains (e.g., medical, surveillance, proprietary industrial data) cannot utilize cloud-based annotation platforms due to data privacy laws or security constraints.
Cost and Accessibility: Commercial platforms (e.g., Encord, Labelerr) offering AI-assisted annotation are often paywalled, while open-source alternatives (e.g., CVAT free version) lack integrated, efficient instance tracking and propagation features.
Resource Constraints: Foundation models like Segment Anything Model 2 (SAM2) are computationally expensive. Running them on long video sequences typically leads to high memory (VRAM) overhead, making interactive, local annotation on commodity hardware difficult.
Workflow Gaps: Existing local tools often rely on purely manual frame-by-frame annotation, which is slow and labor-intensive, lacking the "human-in-the-loop" efficiency provided by modern AI models.

2. Methodology and Architecture

SAMannot is a local, open-source framework designed to integrate SAM2 into a human-in-the-loop workflow while optimizing for memory efficiency and usability.

System Architecture

Frontend: Built with Tkinter, featuring a control panel, a canvas for visualization, and a frame slider. It supports session management, label creation, and real-time visualization of prompts and masks.
Backend: Manages annotation logic, memory, and communication with the SAM2 wrapper. It is divided into four core components:
1. Memory Management: Splits input videos into user-defined blocks (sequences of $N$ frames). Only the current block is loaded into memory, and the SAM2 inference state is reinitialized for each block to prevent VRAM overflow.
2. Prompt & Feature Management: Handles user inputs (points/boxes) and assigns persistent Instance IDs and semantic labels. It supports global feature tracking (e.g., state changes) across the entire video.
3. Propagation: Uses a specialized wrapper around SAM2 to propagate masks temporally. It employs a sliding-window strategy, discarding latent data older than 16 frames to minimize memory usage.
4. Export & Logging: Generates research-ready datasets in YOLO and PNG formats, along with structured interaction logs.

Key Technical Innovations

Block-Based Processing: To handle long videos, the system processes data in chunks. This allows the model to run on mid-range GPUs (e.g., 24GB VRAM) without crashing, a common issue with standard SAM2 inference on long sequences.
Auto-Prompting Mechanism: To bridge gaps between blocks, the system automatically generates point prompts for the start of a new block. It extracts the topological skeleton of the mask from the previous block's end frame and places prompts at endpoints and junctions. It also uses a one-frame temporal lookahead to ensure prompts align with the object's position in the new block.
"Lock-and-Refine" Workflow: Users can set barrier frames (checkpoints). Propagation cannot cross these barriers, allowing users to validate and "lock" specific segments while iteratively refining others without overwriting stable data.
Persistent Identity Management: Unlike raw SAM2 which treats objects as transient slots, SAMannot maps internal object slots to a persistent data structure, ensuring consistent Instance IDs and semantic labels across fragmented annotation sessions.

3. Key Contributions

Local, Privacy-Preserving Framework: A fully open-source (MIT License) solution that runs entirely locally, eliminating the need for cloud uploads and ensuring data sovereignty.
Memory Optimization: Successfully adapted SAM2 for interactive use on commodity hardware by implementing block-based processing and a sliding-window attention mechanism, reducing VRAM requirements significantly compared to baseline SAM2.
Enhanced Workflow Efficiency: Introduced an automated "lock-and-refine" workflow and an auto-prompting mechanism that reduces manual effort in long-video annotation, specifically addressing the challenges of occlusion and object re-appearance.
Standardized Output: Provides seamless export to standard formats (YOLO, PNG masks) and structured logs, facilitating immediate integration into downstream machine learning pipelines.

4. Results and Evaluation

The tool was evaluated on the DAVIS 2017 (TrainVal subset) and LVOS datasets, focusing on segmentation accuracy and resource utilization.

Segmentation Accuracy:
- DAVIS 2017: Achieved a mean IoU of 0.9185 and a mean Dice coefficient of 0.9512.
- LVOS (Harder dataset with occlusions/motion blur): Achieved a mean IoU of 0.8645 and a mean Dice of 0.9044.
- Note: In some cases, SAMannot's masks were more anatomically complete than the ground truth (e.g., including bird feet), leading to lower IoU scores despite higher qualitative accuracy.
Resource Efficiency:
- Tested on an NVIDIA RTX 4090 (24GB VRAM).
- The block-based approach maintained a stable VRAM footprint (min ~1.5GB, max ~2.9GB) regardless of video length, preventing the overflow typical of full-sequence inference.
Performance:
- Annotation duration varies by scene complexity but demonstrates a scalable workflow. The system supports real-time interaction with low latency.

5. Significance and Impact

Democratization of Advanced Tools: SAMannot lowers the barrier to entry for high-fidelity video annotation, making state-of-the-art foundation models accessible to researchers without access to expensive commercial platforms or massive GPU clusters.
Domain Versatility: The framework is domain-agnostic, with demonstrated potential in biological research (animal behavior tracking), robotics (edge-case data generation), and surveillance.
Future-Proofing: The modular architecture separates the GUI from the model wrapper, allowing for easy integration of future foundation models (e.g., SAM3) or alternative backends without rewriting the core application.
Community Engagement: As an open-source project with active maintenance via GitHub, it encourages community contributions and adaptation for specific scientific needs.

In conclusion, SAMannot effectively bridges the gap between powerful AI models and practical, privacy-compliant research workflows, offering a scalable, cost-effective, and high-performance alternative for video instance segmentation.