Technical Summary: X2SAM: Any Segmentation in Images and Videos

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in global visual understanding and reasoning (e.g., image captioning, VQA), their capability to generate dense, pixel-level outputs for precise spatial and temporal comprehension remains limited. Conversely, foundation segmentation models like SAM and SAM2 produce high-quality masks but rely on low-level visual prompts (points, boxes) and lack the ability to natively interpret complex conversational text instructions.

Existing segmentation MLLMs attempt to bridge this gap but suffer from structural fragmentation:

• Image-only models (e.g., LISA) process text but lack support for visual prompts and video inputs.
• Video-only models (e.g., VISA, VideoLISA) support temporal text-to-mask generation but often lack unified architectures for static images and visual prompting.
• Current limitations: No single framework natively supports both textual and visual prompts across both static images and dynamic videos while maintaining temporal consistency.

2. Methodology: X2SAM Framework

X2SAM is a unified segmentation-oriented MLLM designed to extend "any-segmentation" capabilities from images to videos. It integrates an MLLM with a specialized Mask Memory module to handle diverse tasks under a single instruction-following paradigm.

2.1 Unified Formulation

The framework treats objects of interest as conditional states and language instructions as contextual inputs. It utilizes special tokens <p> and </p> to demarcate object conditions and a dedicated <SEG> token to indicate the segmentation mask. The hidden representation of the <SEG> token serves as a semantic directive guiding the mask decoder. This formulation supports 14 distinct tasks across images and videos, including:

• Generic, Open-Vocabulary, Referring, and Reasoning Segmentation.
• Grounded Conversation Generation (GCG).
• Interactive and Visual Grounded Segmentation (VGD).

2.2 Architecture

The model employs a dual-branch visual extraction and a unified generation pipeline:

1. Visual Encoders:
   • Vision Encoder: Based on Qwen3-VL, it extracts global visual representations ($Z_v$) from images or video frames (augmented with timestamps).
   • Mask Encoder: Based on SAM2, it captures fine-grained, high-resolution features ($Z_m$) essential for dense mask prediction.
   • Region Sampler: A parameter-free module that extracts localized visual prompt embeddings from $Z_m$ using point-sampling and adaptive pooling, enabling the injection of visual cues (points/boxes) into the LLM.
2. Large Language Model (LLM):
   • Processes tokenized text instructions and visual embeddings.
   • Auto-regressively generates the language response ($Y_q$) and a dedicated SEG latent embedding. This embedding acts as a semantic bridge between language understanding and mask prediction.
3. Mask Decoder:
   • Redesigned from X-SAM, it integrates Token-to-Image Attention and Query-to-Image Attention.
   • It receives the projected SEG token embedding ($Z_p$), learnable mask queries ($Q_m$), and temporally refined visual features ($Z_w$).
   • Zero-initialization is used for Token-to-Image attention parameters to ensure stable training.
4. Mask Memory Module:
   • Crucial for video tasks, this module maintains a First-In-First-Out (FIFO) cache of guided vision features from preceding frames.
   • Memory Attention: Attends to previous frames' guided features to produce temporally refined features for the current frame.
   • Memory Encoder: Encodes downsampled vision features and current mask logits into new guided features for storage.
   • This mechanism replaces independent frame-by-frame decoding, ensuring mask consistency across video sequences.

2.3 Training Strategy

X2SAM utilizes a Unified Joint Training strategy over heterogeneous image and video datasets:

• Phase 1 (Agnostic Segmentor): The mask encoder is frozen; the mask decoder is trained on SA-1B (mask-only) to learn class-independent shape and boundary priors.
• Phase 2 (Joint Training): The model is co-trained on 14 segmentation tasks and chat datasets.
  • Dimension-Shifting Pipeline: Video tensors are transposed and split into frame-level tensors to process them through the image-level interface, with temporal dependencies handled by the Mask Memory.
  • Modality-Aware Batching: To manage memory constraints, video batches are kept small (batch size 1 per device) while image batches are expanded (multiplier of 4), allowing efficient use of GPU parallelism.
  • Loss Function: A joint objective combines auto-regressive loss (language), mask loss (BCE + Dice), and focal loss (classification).

3. Key Contributions

1. Unified Framework: X2SAM is the first model to natively support seven segmentation tasks (generic, open-vocabulary, referring, reasoning, grounded conversation, object-centric, and visual grounded) across both images and videos within a single instruction-following interface.
2. Mask Memory Module: Introduces a novel module that couples semantic grounding with temporal propagation, enabling temporally consistent mask generation for videos without relying on cascaded propagation or frame-wise independence.
3. Video Visual Grounded (V-VGD) Benchmark: Proposes a new benchmark evaluating the ability to segment object tracks in videos using interactive visual prompts (points/boxes), filling a gap in existing evaluation protocols.
4. Efficient Joint Training: Demonstrates a unified training strategy that reduces training costs (3.3K GPU hours vs. 5.2K for simple joint training) while maintaining balanced performance across modalities.

4. Experimental Results

X2SAM was evaluated on 14 tasks across image and video benchmarks, showing strong performance in both domains:

• Image Segmentation: X2SAM remains competitive with image-specialized generalists (e.g., X-SAM). It achieves state-of-the-art or near-state-of-the-art results on referring segmentation (RefCOCOg) and reasoning segmentation, while significantly improving open-vocabulary segmentation (I-OV) compared to prior MLLM-based methods.
• Video Segmentation: X2SAM outperforms existing MLLM-based video generalists (e.g., VISA, VideoLISA, UniPixel) on most benchmarks.
  • Generic Segmentation: Achieves 65.1 mIoU on VSPW and 69.9/88.4 AP/AP50 on YT-VIS19.
  • Referring & Reasoning: Sets new state-of-the-art on video referring (Ref-YT21, Ref-DV17) and reasoning (ReVOS) tasks, surpassing specialized non-MLLM models like ReferFormer-B in reasoning tasks.
  • Grounded Conversation: Improves V-GCG mIoU by +21.5 over VideoGLaMM.
• Visual Grounded (V-VGD): On the new V-VGD benchmark, X2SAM substantially outperforms SAM2-H (e.g., 74.4 AP vs. 54.0 AP on YT-VIS19 with box prompts), validating the effectiveness of the region sampler and mask memory in propagating visual prompts.
• Chat Capabilities: Despite the focus on segmentation, X2SAM preserves strong general image and video chat abilities, outperforming many chat-specialized models on benchmarks like MMBench and VideoMME.

5. Significance and Claims

The paper claims that X2SAM successfully unifies the "any-segmentation" paradigm from static images to dynamic videos. By integrating an MLLM with a Mask Memory module, it overcomes the limitations of prior fragmented approaches, offering a single interface for complex multi-modal instructions (text and visual) across diverse tasks.

The authors emphasize that X2SAM establishes a practical baseline for unified pixel-level spatio-temporal understanding. While it acknowledges that specialized models may still outperform it on narrowly focused tasks (e.g., optimized video object segmentation without language), X2SAM demonstrates that a generalist model can achieve a strong balance between task coverage, accuracy, and the preservation of general conversational abilities. The introduction of the V-VGD benchmark and the unified joint training strategy are highlighted as key steps toward more robust and versatile vision-language systems.