TF-SSD: A Strong Pipeline via Synergic Mask Filter for Training-free Co-salient Object Detection

1. Problem Definition

Co-salient Object Detection (CoSOD) aims to segment objects that consistently appear across a group of related images.

• Current Limitations: State-of-the-art (SOTA) methods are predominantly training-based, relying on closed-set datasets. This leads to poor generalization when encountering unseen object categories or distributions (overfitting to biased training data).
• The Gap: While Vision Foundation Models (VFMs) like SAM (Segment Anything Model) and DINO (Self-Distillation with No Labels) show strong generalization, few studies have leveraged them for CoSOD.
  • SAM excels at generating precise segmentation masks but lacks semantic understanding to distinguish co-salient objects from irrelevant ones across an image group.
  • DINO possesses strong semantic understanding and attention mechanisms for salient objects but lacks precise boundary delineation.
• Core Challenge: How to combine SAM's exhaustive mask proposals with DINO's semantic guidance to identify accurate co-salient masks without any training (zero-shot).

2. Methodology: TF-SSD

The authors propose TF-SSD, a training-free framework that synergizes SAM and DINO through a progressive three-stage pipeline.

Stage 1: Quality Mask Generator (QMG)

• Input: SAM generates a raw, exhaustive set of mask proposals ($M_{raw}$) for each image, containing many redundant, trivial (too small), or overlapping masks.
• Process: QMG filters these masks in three steps:
  1. Initial Filtering: Removes masks with extremely small area ratios ($r_{area}$).
  2. Overlap Filtering: Eliminates redundant overlapping segments by calculating overlap ratios, preserving larger masks.
  3. Quality Assessment: Computes a balanced quality score ($S_{ba}$) combining SAM's predicted IoU score and an area-based penalty function. This ensures masks are of moderate size and high confidence.
• Output: A refined set of candidate masks ($M_{refine}$).

Stage 2: Intra-image Saliency Filter (ISF)

• Goal: Identify which masks within a single image correspond to visually salient objects, as QMG lacks semantic awareness.
• Mechanism:
  • Utilizes DINO's attention maps (from the [CLS] token of the ViT encoder), which naturally highlight salient regions.
  • Calculates a saliency score ($s_{sal}$) for each refined mask by measuring the spatial alignment (element-wise multiplication) between the mask region and the DINO attention map.
• Output: A set of salient masks ($M_{salient}$) for each image, filtered by an adaptive threshold.

Stage 3: Inter-image Prototype Selector (IPS)

• Goal: Determine which salient masks are co-salient (common) across the entire image group.
• Mechanism:
  1. Prototype Extraction: For each salient mask, extracts a semantic prototype vector using DINO's global features (masked image regions).
  2. Similarity Computation: Constructs a pairwise cosine similarity matrix between prototypes of all masks across all images in the group.
  3. Selection Strategy: For each image, it identifies the mask whose prototype has the maximum similarity score with prototypes from other images in the group (ignoring self-similarity).
• Output: The mask with the highest cumulative co-saliency score is selected as the final prediction for each image.

3. Key Contributions

1. Novel Training-Free Framework: TF-SSD is the first method to effectively leverage the synergy between SAM (segmentation) and DINO (semantics) for CoSOD without fine-tuning or training.
2. Progressive Filtering Pipeline:
   • QMG: Systematically reduces noise and redundancy in SAM's raw outputs.
   • ISF: Leverages DINO's attention to filter for visual saliency within single images.
   • IPS: Models inter-image semantic consistency to select the true co-salient objects.
3. Superior Performance: The method achieves SOTA results, outperforming both fully supervised and other training-free approaches.

4. Experimental Results

Experiments were conducted on three standard benchmarks: CoCA, CoSal2015, and CoSOD3k.

• Quantitative Performance:
  • TF-SSD outperforms the recent training-free method ZS-CoSOD by a significant margin (+13.7% in $F_{max}$ on CoCA).
  • It surpasses many fully supervised methods (e.g., GCoNet+, VCP) and all unsupervised methods across metrics like $F_{max}$, $S_{measure}$, and MAE.
  • On CoCA, it achieves an $F_{max}$ of 0.686, compared to 0.549 for ZS-CoSOD.
• Qualitative Performance:
  • Visual comparisons show TF-SSD produces smoother boundaries and more accurate segmentation for complex objects (e.g., "Chopsticks," "Axe") where other methods fail to distinguish co-salient targets from background noise.
• Ablation Studies:
  • Removing any component (QMG, ISF, or IPS) leads to significant performance drops.
  • The Maximum Similarity strategy in IPS (selecting the highest score rather than averaging) proved superior, indicating strong semantic correlation for co-salient objects.

5. Significance

• Paradigm Shift: TF-SSD demonstrates that training-free approaches can rival or exceed supervised methods in CoSOD by effectively utilizing pre-trained foundation models.
• Generalization: By avoiding closed-set training, the method offers robust generalization to unseen object categories and diverse real-world scenarios.
• Efficiency: The framework requires no model training, making it immediately deployable and computationally efficient for new datasets.
• Future Baseline: The paper establishes a strong baseline for future research into leveraging VFMs for complex vision tasks requiring both precise segmentation and high-level semantic reasoning.