ActiveSAM: Image-Conditional Class Pruning for Fast and Accurate Open-Vocabulary Segmentation

Technical Summary: ActiveSAM

Problem Statement
The paper addresses the inefficiency of applying the Segment Anything Model 3 (SAM 3) to Open-Vocabulary Semantic Segmentation (OVSS). While SAM 3 provides a strong frozen backbone for concept-prompted segmentation, directly applying it to OVSS is computationally expensive. Standard pipelines typically run full-resolution decoding over the entire dataset vocabulary for every image, despite each image containing only a small active subset of classes. This creates a systems bottleneck where the computational cost scales with the vocabulary size rather than the actual content of the image. Furthermore, existing training-free methods often struggle with spatial precision or rely on CLIP-based features that degrade under real-world distribution shifts.

Methodology
ActiveSAM is a training-free, zero-shot inference framework that adapts the frozen SAM 3 model into an "active-vocabulary segmenter." It decouples computation into four main stages:

1. Contextual Prompt Expansion (CPE): To improve alignment robustness, raw class names are canonicalized (e.g., fixing label artifacts like "windowpane" to "window pane") and expanded with contextual information. This includes retrieving the $M_s$ nearest vocabulary neighbors in the frozen text-embedding space and incorporating up to $M_t$ WordNet hypernyms. These tokens are concatenated with the canonicalized class tokens to form enriched prompts, which are pre-computed and cached.
2. Preview-Driven Class Selection: ActiveSAM utilizes SAM 3’s presence head at a low resolution ($r_p = 672$) to estimate class presence scores for the entire vocabulary. This stage skips segmentation-head computation to minimize latency. An image-adaptive quantile threshold determines an active set $A(I)$ of likely present classes. A gating threshold ($V_{gate} = 40$) ensures that datasets with small vocabularies bypass this pruning step to prevent accuracy loss, while large-vocabulary datasets retain only the top-scoring classes.
3. Bucketed Full-Resolution Decoding: The full-resolution image encoder ($r_f = 1008$) is run once. The active prompts are grouped into buckets (size $K=32$) and processed through the frozen grounding decoder. This multiplexing reduces the number of decoder calls from the full vocabulary size to $\lceil |A(I)|/K \rceil$. Instance and semantic scores are fused via pixelwise maximization.
4. Margin-Aware Background Calibration (MABC): For benchmarks with explicit background labels, ActiveSAM refines predictions by considering the margin between the top two class scores. A calibrated confidence score is computed based on the top score and its margin over the second-best score. Pixels with calibrated confidence below a threshold are assigned to the background class, improving rejection of low-confidence predictions without introducing dataset-specific thresholds.

Key Contributions

• Active-Vocabulary Pruning: The introduction of a training-free framework where SAM 3’s low-resolution presence head determines per-image active classes, replacing the inefficient practice of decoding the entire vocabulary.
• Complementary Modules: The integration of Contextual Prompt Expansion (CPE) for richer semantic prompts and Margin-Aware Background Calibration (MABC) for principled background rejection, both without updating SAM 3 weights.
• Performance and Efficiency: ActiveSAM achieves state-of-the-art mean mIoU on eight standard OVSS benchmarks while significantly reducing inference time. It outperforms the current state-of-the-art, SegEarth-OV3, by approximately +1.4 mIoU on average and runs up to 5.5× faster on large-vocabulary datasets.
• Robustness: The framework demonstrates strong robustness against image corruptions (noise, blur, compression, fog), maintaining high performance where CLIP-based baselines suffer severe degradation.

Results

• Accuracy: ActiveSAM achieves a mean mIoU of 65.3% across eight benchmarks (Pascal VOC, Pascal Context, COCO, Cityscapes, ADE20K), outperforming SegEarth-OV3 (63.9%) and SAM 3 (57.5%). The largest improvements over SAM 3 are observed on large-vocabulary benchmarks like Pascal Context-59 (+10.1 mIoU) and COCO-Stuff (+11.1 mIoU).
• Speed: On large-vocabulary datasets, ActiveSAM is approximately 4.8× to 5.5× faster than SegEarth-OV3. For example, on ADE20K, it achieves 1.92 FPS compared to SegEarth-OV3’s 0.35 FPS.
• Robustness: Under ImageNet-C corruptions (Gaussian noise, motion blur, JPEG compression, fog), ActiveSAM maintains the largest lead over competitors. For instance, under Gaussian noise, ActiveSAM retains a mean mIoU of 46.48%, significantly higher than CorrCLIP (16.02%) and RF-CLIP (18.33%).
• Ablation: Removing CPE reduces mean mIoU by 1.0%, while removing MABC reduces it by 0.2% (primarily affecting background-aware datasets). Removing the class-pruning step leaves accuracy unchanged but eliminates the speedup, confirming that pruning provides efficiency without accuracy cost.

Significance
The paper claims that ActiveSAM resolves the computational bottleneck of full-vocabulary decoding in promptable foundation models by leveraging early presence signals. It demonstrates that training-free OVSS can be both highly accurate and efficient, making it suitable for deployment in dynamic, open-world environments such as autonomous driving and embodied AI, where vocabulary changes frequently and input conditions are noisy. The framework requires no target-dataset training, no weight updates, and no oracle class-presence labels, offering a universal solution for zero-shot segmentation.