Technical Summary: Boosting SAM for Cross-Domain Few-Shot Segmentation via Conditional Point Sparsification

1. Problem Statement

The Segment Anything Model (SAM) has demonstrated unprecedented segmentation capabilities on in-domain datasets (e.g., COCO) through promptable segmentation. Recent training-free approaches have leveraged SAM for Few-Shot Segmentation (FSS) by using dense point matching between reference and target images to generate prompts. However, these methods suffer significant performance degradation when applied to Cross-Domain Few-Shot Segmentation (CD-FSS), particularly in medical (e.g., skin lesions) and satellite (e.g., water bodies) domains.

The authors identify that this failure stems from large domain shifts that disrupt the point–image interactions learned by SAM during its pre-training. Specifically:

• Feature Homogeneity: Objects in cross-domain datasets (e.g., skin nevi, water) exhibit more uniform and homogeneous semantics compared to the diverse parts of objects in in-domain datasets (e.g., cats with heads, legs, tails).
• Prompt Density Mismatch: SAM was trained with dense points to handle complex shapes. In cross-domain scenarios, these dense points introduce ambiguity and noise because the homogeneous semantics of the target objects require fewer, more informative points.
• Inconsistency: Existing methods perform robustly on in-domain data but fail to generalize when dense prompts are applied to cross-domain targets, even when the points correctly fall within the target regions.

2. Methodology: Conditional Point Sparsification (CPS)

The paper proposes Conditional Point Sparsification (CPS), a training-free framework designed to adaptively guide SAM interactions for cross-domain images. The pipeline consists of four key stages:

A. Dense Point Matching and Boundary Point Pruning

1. Dense Matching: Using the Positive-Negative Alignment (PNA) module with a DINOv2 backbone, the method establishes dense correspondences between the reference image ($I_r$) and the target image ($I_t$). This yields an initial set of candidate prompt points ($P_t$) on the target image.
2. Boundary Point Pruning: To mitigate spatial misalignment caused by resolution discrepancies between feature maps and the original image, the method computes the convex hull of the matched points. Points lying on the convex hull boundary are removed, retaining only interior points ($\tilde{P}_t$). This step filters out potentially noisy prompts near object boundaries.

B. Adaptive Point Sparsification

Instead of using a fixed density, CPS sparsifies the dense point set $\tilde{P}_t$ based on a density inferred from the reference image.

1. Grid Partitioning: The target image is divided into a regular grid.
2. Centroid Selection: For each grid cell containing points, the point closest to the global centroid of the target object is retained. This preserves structural cues while reducing density.

C. Conditional Density Lookup

The core innovation is determining the optimal sparsification density ($D_{ref}$) dynamically:

1. Reference Evaluation: A set of candidate densities is tested on the reference image ($I_r$) using its ground-truth mask ($M_r$).
2. Optimization: For each candidate density, prompt points are sampled, fed into SAM, and the resulting mask is evaluated against $M_r$ using Intersection over Union (IoU).
3. Selection: The density yielding the highest IoU is selected as $D_{ref}$. This conditional density is then applied to sparsify the target image's points ($\hat{P}_t$).

D. Post-hoc Mask Refinement

To address discontinuities in the activation heatmap (e.g., holes or abrupt boundary changes) caused by sparse prompts, the initial predicted mask ($\hat{M}_t$) undergoes morphological opening and closing operations. This enforces spatial continuity and smooths boundaries to produce the final mask ($\tilde{M}_t$).

3. Key Contributions

1. Diagnosis of Domain Shift: The authors reveal that performance degradation in existing SAM-based FSS methods on cross-domain images is caused by domain shifts that break learned point–image interactions, specifically noting that dense points are suboptimal for homogeneous cross-domain objects.
2. Conditional Point Sparsification (CPS): A novel, training-free strategy that infers appropriate prompt density from few-shot reference exemplars (using ground-truth masks) and applies this density to sparsify matched points in target images.
3. Comprehensive Validation: The method introduces specific modules (Boundary Point Pruning, Adaptive Sparsification, Post-hoc Refinement) that collectively improve performance without requiring additional training or fine-tuning.

4. Experimental Results

The authors evaluated CPS on four cross-domain benchmarks: ISIC (skin lesions), Chest X-Ray (tuberculosis), DeepGlobe (satellite/terrain), and SUIM (underwater scenes).

• Performance: CPS consistently outperforms existing training-free SAM-based methods (e.g., Matcher, GF-SAM) and traditional training-dependent CD-FSS methods.
  • On ISIC, CPS achieved 51.3% (1-shot) and 56.6% (5-shot) mIoU, surpassing the previous best training-free method (GF-SAM) by 2.6% and 1.4% respectively.
  • On Chest X-Ray, CPS reached 70.0% (1-shot) and 70.2% (5-shot), outperforming Matcher by 3.2% and 3.9%.
  • On DeepGlobe and SUIM, CPS also set new state-of-the-art results among training-free approaches.
• Ablation Studies:
  • Adaptive Sparsification: Provided the largest gain (+7.9% on ISIC), confirming that fixed densities are suboptimal.
  • Boundary Point Pruning: Improved point accuracy by filtering boundary noise.
  • Post-hoc Refinement: Further improved mIoU by 3.6% on ISIC by fixing spatial discontinuities.
• Generalization: CPS was also tested on in-domain natural image datasets (COCO-20i and FSS-1000), where it achieved competitive performance, demonstrating that the sparsification strategy does not negatively impact performance on complex, in-domain objects.

5. Significance and Claims

The paper claims that Conditional Point Sparsification offers a practical, training-free solution to extend SAM's capabilities to challenging cross-domain segmentation tasks. By leveraging the ground-truth masks of reference exemplars to adaptively tune prompt density, CPS aligns the point-image interaction with the specific characteristics of the target domain (e.g., homogeneity in medical/satellite images).

The authors emphasize that their approach avoids the computational overhead and privacy concerns associated with collecting large-scale labeled data for adaptation or the heavy computational cost of training-based pipelines. Instead, it provides a lightweight, inference-stage modification that significantly boosts the generalization of foundation models in scenarios where domain shifts are severe.