Technical Summary: SAM 2++: Tracking Anything at Any Granularity

1. Problem Statement

Video tracking has historically fragmented into independent sub-tasks based on target granularity: Single Object Tracking (SOT) using bounding boxes, Video Object Segmentation (VOS) using pixel-level masks, and Point Tracking (PT) using fine-grained keypoints. While these tasks share a core objective—estimating target states over time—they have led to specialized models with redundant designs and limited generalization. Existing unified vision models often share backbones but retain task-specific interfaces or ignore specific granularities (like point tracking), failing to establish a truly unified representation of tracking states. Furthermore, the lack of a dataset providing simultaneous annotations for masks, boxes, and points hinders the training and evaluation of unified tracking frameworks.

2. Methodology: SAM 2++

The authors propose SAM 2++, a unified framework extending the Segment Anything Model 2 (SAM 2) to track targets at any granularity (masks, boxes, points) through an integrated design of prompt encoding, output decoding, and memory representation.

2.1 Unified Task Input and Output

• Task-Specific Prompts: To handle inconsistent input granularities, the framework introduces specific prompts while preserving the original SAM 2 prompt encoder.
  • Mask: Uses binary masks ($I_{mask}$).
  • Box: Uses bounding box coordinates ($I_{box}$).
  • Point: Uses point coordinates ($I_{point}$) combined with a Gaussian mask ($G_{point}$). This Gaussian mask highlights the point in a form consistent with the decoder's output and the memory encoder's input, replacing naive binary masks for better representation. The radius and sigma of this Gaussian are gradually decreased during training to facilitate convergence.
• Unified Decoder: Extending the SAM 2 Mask Decoder, this component produces a unified mask-form pre-output for all tasks.
  • SOT: A Corner-based Head (CornHead) is added to explicitly optimize bounding box position and scale from the mask output, avoiding the inaccuracies of simple outer-box derivation.
  • PT: Point coordinates are derived from the mask output using a soft-argmax operation during training (to ensure gradient flow) and a standard argmax during inference. This aligns point prediction with the Gaussian prompt and the memory source.

2.2 Task-Adaptive Memory Mechanism

The core innovation addresses the observation that different tracking granularities have distinct memory requirements. Full parameter sharing in memory modules leads to performance degradation because a single memory representation cannot satisfy the precise segmentation needs of masks, the coarse localization needs of boxes, and the specific Gaussian nature of points simultaneously.

• Decoupling: The framework decouples memory components while maintaining a shared tracking pipeline.
  • Memory Encoder: Each task (mask, box, point) has its own independent copy of the convolutional Memory Encoder.
  • Memory Attention: Each task is equipped with an independent LoRA (Low-Rank Adaptation) module for the transformer-based Memory Attention.
• Result: This design allows the model to maintain a shared memory representation structure while adapting to the specific state semantics of each granularity, preventing interference and enabling mutual promotion among tasks.

2.3 Training Strategy

The model is trained jointly on three tasks using an alternating sampling strategy (probabilities 1:4:5 for mask:box:point). The training process incorporates:

• Interactive Prompts: For mask and box tasks, the model supports both ground-truth and noisy/interactive inputs (e.g., noisy boxes, positive clicks).
• Corrective Frames: For interactive scenarios, the model samples corrective points from false positive/negative regions to refine predictions iteratively.
• Loss Functions: A combination of Focal and Dice losses for masks, IoU and L1 losses for boxes (via the corner head), and L1 distance loss for points. Occlusion prediction is supervised via cross-entropy regardless of object visibility.

3. Key Contributions

3.1 The SAM 2++ Framework

The paper proposes the first unified framework capable of tracking targets at any granularity (masks, boxes, points) within a single architecture. It achieves this through:

• Task-specific prompts that map diverse inputs to general embeddings.
• A Unified Decoder that standardizes outputs.
• A Task-Adaptive Memory mechanism that balances parameter sharing with task-specific state semantics.

3.2 Tracking-Any-Granularity (TAG) Dataset

To support unified training and evaluation, the authors constructed TAG, a large-scale video tracking dataset containing 6,000 videos with 2.2 million frames.

• Annotations: It provides high-quality annotations for all three granularities (segmentation masks, bounding boxes, and key points) for the same video sequences.
• Data Engine: The dataset was created using a phased data engine combining manual annotation and model-assisted completion.
  • Phase 1: Manual annotation of every frame (1,000 videos).
  • Phase 2 & 3: Manual annotation of sparse frames (every 10th and 20th) with model-in-the-loop completion for intermediate frames, utilizing backward tracking and fusion strategies to ensure quality.
• Scope: The dataset covers diverse scenes, resolutions, and challenging attributes like motion blur, occlusion, and deformation.

4. Experimental Results

Comprehensive experiments across multiple benchmarks demonstrate that SAM 2++ sets a new state-of-the-art (SOTA) across all three tasks:

• Video Object Segmentation (VOS): Outperforms specialized models (e.g., XMem, DEVA, Cutie) on benchmarks including YouTubeVOS, MOSE, LVOS-v2, and BURST.
• Single Object Tracking (SOT): Surpasses dedicated trackers (e.g., OSTrack, ARTrack, HIPTrack) on TrackingNet, GOT-10k, and VastTrack. Notably, it achieves this as an end-to-end unified tracker without the pre-cropping strategies common in SOT.
• Point Tracking: Outperforms both offline and modified online trackers (e.g., CoTracker3, TAPTR) on BADJA, Perception Test, and TAP-Vid. It demonstrates strong generalization to arbitrary point tracking despite being trained on keypoint data.
• Unified Performance: The model consistently outperforms other unified vision models (e.g., Unicorn, UNINEXT, MITS) and the previous unified tracking attempt (TAG), validating the effectiveness of the task-adaptive memory and the unified representation.

5. Significance and Claims

The paper claims that SAM 2++ establishes a unified and robust tracking framework that bridges the gap between disparate tracking paradigms. Its significance lies in:

1. Unification: It demonstrates that tracking tasks with different granularities can be unified under a single memory-matching paradigm without sacrificing performance, challenging the necessity of task-specific designs.
2. Generalization: By training on the diverse TAG dataset and utilizing the task-adaptive memory, the model learns robust representations that generalize across tasks, often surpassing specialized models trained on task-specific data.
3. Resource Provision: The introduction of the TAG dataset provides a critical resource for future research in unified tracking, offering the first large-scale, multi-granularity benchmark.
4. Practical Utility: As a foundation model, SAM 2++ serves as a versatile tool for both tracking applications and automated annotation, reducing the cost of switching between different tracking granularities.

The authors acknowledge limitations, noting that the current version does not support language or audio references and that the decoupled memory parameters are supervised by single tasks, limiting multi-task learning benefits for those specific components. However, the framework represents a significant step toward a general-purpose video tracking foundation model.