SSeg: Active Sparse Point-Label Augmentation for Semantic Segmentation

1. Problem Statement

Semantic segmentation is critical for environmental monitoring (e.g., underwater coral reefs, aerial landscapes), but training state-of-the-art models requires dense, pixel-level annotations. In fields like ecology, obtaining these dense labels is prohibitively expensive and time-consuming, often requiring expert marine biologists.

• Current State: Experts typically provide only sparse point-labels (e.g., 50–200 points per image) rather than full masks.
• The Challenge: Existing methods to propagate these sparse points into dense masks face two main issues:
  1. Inefficient Sampling: Random or grid-based sampling often misses small/thin objects or lands on ambiguous boundaries, leading to poor propagation.
  2. Propagation Limitations:
     • Superpixel methods (e.g., PLAS) offer good coverage but often miss fine object details and boundaries.
     • Foundation models (e.g., SAM2, DINOv2) offer precise boundaries but struggle with spatial coherence and dense coverage, often leaving gaps or failing on background regions.

2. Methodology: The SSeg Framework

The authors propose SSeg, a workflow that combines Active Point Selection with a Hybrid Label Augmentation strategy to generate high-quality dense masks from minimal expert clicks.

A. Active Point Selection (DynamicPoints)

Instead of random or grid sampling, SSeg employs an active strategy to select the most informative $n$ points to query. The selection is guided by an acquisition function $A(p)$ that balances three criteria:

1. Proximity to Object Centroids: Uses SAM2 to generate candidate masks and calculates centroids. Points near the centers of large objects are prioritized to ensure robust object supervision.
2. Exploration Coverage: Penalizes points too close to already selected labels to ensure spatial distribution across the image.
3. Background Inclusion: Explicitly selects points from regions not covered by SAM2 masks (background/low-contrast zones) to prevent the model from failing on non-object areas.

Acquisition Function:
$$A(p) = \lambda O(p) + (1 - \lambda) E(p)$$
Where $O(p)$ is the proximity score to object centroids, $E(p)$ is the distance to existing labels, and $\lambda$ balances exploitation vs. exploration. The remaining half of the budget is filled with random points from uncovered background regions.

B. Hybrid Label Augmentation

Once points are selected, SSeg propagates them into a full-image mask using a two-step hybrid approach:

1. SAM2 Expansion (Precision): Each point is expanded into a binary mask using SAM2.
   • Conflict Resolution: If masks overlap, the system resolves conflicts by calculating a support score based on the proximity of the original point-labels to the overlap region's centroid. The label with the highest support score wins.
   • Result: A partial segmentation ($I^{SAM2}_L$) with high-quality boundaries but potential gaps.
2. PLAS Filling (Coverage): PLAS (a superpixel-based method) is run in parallel to generate a dense segmentation ($I^{PLAS}_L$) covering the entire image.
   • Fusion: The final mask $I_L$ takes the SAM2 label where available (for precision) and falls back to the PLAS label for any pixels not covered by SAM2 (for completeness).

3. Key Contributions

1. Novel Active Sampling Strategy: A dynamic selection method that optimizes point placement by balancing object-centric supervision with necessary background coverage, outperforming static random/grid sampling.
2. Hybrid Propagation Architecture: A robust fusion of foundation models (SAM2) for boundary precision and superpixel methods (PLAS) for full coverage, resolving the trade-off between detail and completeness.
3. Conflict Resolution Mechanism: A specific algorithm to handle overlapping masks based on point proximity, significantly improving accuracy in dense object clusters.
4. Interactive Tool: The release of a practical annotation tool allowing ecology researchers to generate dense masks from sparse clicks, leveraging foundation models without manual tuning.

4. Experimental Results

The method was evaluated on two diverse datasets: UCSD Mosaics (underwater coral reefs) and SkyScapes (aerial imagery).

• Performance vs. State-of-the-Art:
  • SSeg outperformed baselines (PLAS, D+NN, CoralSCOP) by +3% to +8% mIoU (masked mean Intersection over Union).
  • Unlike D+NN, which relies on "Oracle-based" initialization (using ground truth to place initial points), SSeg achieves superior performance without any ground-truth guidance for point placement.
• Ablation Studies:
  • Hybrid Approach: The full SSeg pipeline consistently outperformed "SSeg-SAM2Only," proving the necessity of the PLAS filling module for unassigned regions.
  • Overlap Resolution: The proposed conflict resolution strategy improved mIoU by over 7% compared to naive "First-come" or "Last-come" heuristics.
  • Sampling Strategy: The active "DynamicPoints" strategy significantly outperformed grid and random sampling, particularly in the aerial domain where grid sampling failed.
• Downstream Utility:
  • Models trained on SSeg-generated masks (using only 25 points) achieved 90% of the performance of models trained on full dense ground truth.
  • This represents a 99.99% reduction in annotation effort (25 points vs. ~262k pixels) while maintaining near-optimal model performance.

5. Significance

This work addresses a critical bottleneck in ecological monitoring: the scarcity of labeled data. By automating the selection of annotation points and intelligently propagating them, SSeg enables researchers to:

• Drastically reduce the time and cost of data annotation (by four orders of magnitude).
• Generate high-quality training data for deep learning models in complex, unstructured environments (underwater and aerial).
• Leverage the power of foundation models (SAM2) without requiring massive datasets or manual prompt engineering.

The framework provides a scalable, practical solution for transforming sparse expert knowledge into dense, actionable data for environmental analysis.