H-SPAM: Hierarchical Superpixel Anything Model

1. Problem Statement

Superpixels are widely used in computer vision to group pixels into coherent, compact regions, simplifying tasks like semantic segmentation and interactive annotation. However, existing methods face three critical limitations:

• Accuracy vs. Regularity Trade-off: Traditional methods (e.g., SLIC) rely on low-level features and struggle with semantic boundaries. Deep learning-based methods achieve better boundary alignment but often sacrifice regularity, producing noisy, irregular, or fragmented shapes that are hard to interpret.
• Lack of Hierarchy: Most state-of-the-art (SOTA) methods generate a single fixed-scale partition. This limits their utility in vision pipelines requiring multi-scale reasoning or progressive refinement (e.g., interactive annotation).
• Hierarchy vs. Quality: Existing hierarchical methods attempt to create nested partitions but typically degrade in accuracy and regularity to enforce the "nestedness" constraint (where finer regions must be fully contained within coarser ones). They often produce noisy shapes to capture object boundaries, failing to maintain a clean hierarchy.

2. Methodology: H-SPAM

The authors propose H-SPAM (Hierarchical Superpixel Anything Model), a unified framework that generates accurate, regular, and perfectly nested hierarchical superpixels by leveraging deep features and external object priors (e.g., from the Segment Anything Model - SAM).

The framework operates in three main stages:

A. Input and Feature Extraction

• Initial Partition: The process starts with a fine-grained superpixel map generated by a deep learning model (based on SPAM [29]).
• Feature Representation: Each superpixel is represented by a feature vector $F$ combining:
  • Color ($F_C$): LAB color space.
  • Spatial ($F_S$): 2D pixel coordinates.
  • Deep Features ($F_D$): High-level semantic features extracted via a CNN.
• Object Priors: A high-level object map (from SAM or FastSAM) assigns each superpixel to a specific object instance.

B. Two-Phase Hierarchical Merging

The core innovation is a Region Adjacency Graph (RAG) based merging process divided into two distinct phases to balance object consistency with hierarchical structure:

1. Phase 1: Intra-Object Merging

   • Constraint: Merges are strictly forbidden between superpixels belonging to different objects. This is enforced by assigning an infinite cost to inter-object edges.
   • Cost Function: Merges occur only within the same object ($\Theta(u) = \Theta(v)$). The cost combines appearance similarity and spatial distance:
     $$cost_{phase1} = \|\mu_{u}^{FC,FD} - \mu_{v}^{FC,FD}\|^2 + w(s) \|\mu_{u}^{FS} - \mu_{v}^{FS}\|^2$$
   • Goal: To refine the internal structure of objects while strictly preserving object boundaries.

2. Phase 2: Inter-Object Merging

   • Constraint: Once each object is reduced to a single region, the algorithm allows merging between different objects.
   • Cost Function: Inspired by Ward's method, the cost incorporates region sizes to prioritize merging smaller objects first, then larger ones:
     $$cost_{phase2} = \frac{s_u \cdot s_v}{s_u + s_v} \|\mu_{u}^{FC,FD} - \mu_{v}^{FC,FD}\|^2$$
   • Goal: To build a complete hierarchy from the initial fine partition down to a single global region, spanning the full range of scales.

C. Attention-Based Modulation (Optional)

To support specific downstream tasks (like focusing on salient regions), H-SPAM integrates visual attention maps (e.g., from DINO) or user clicks:

• Mechanism: The fusion cost is increased for regions with high attention scores.
• Effect: Salient or user-selected regions are delayed in the merging process, preserving higher resolution (more superpixels) in those areas while coarser regions merge earlier.
• Refinement: The authors propose averaging attention scores within prior objects to prevent "leakage" artifacts and ensure coherent guidance.

3. Key Contributions

1. First Object-Based Hierarchical Framework: H-SPAM is the first deep learning framework for hierarchical superpixels that integrates prior object masks to enforce boundary alignment while maintaining perfect nestedness.
2. Two-Phase Merging Strategy: By separating intra-object and inter-object merging, the method achieves a new benchmark in balancing segmentation accuracy (adherence to objects) and regularity (shape consistency).
3. Adaptive Modes: Introduction of visual attention and user-interaction modes that allow dynamic control over the density of superpixels in regions of interest without breaking the hierarchical structure.
4. Efficiency: The method is computationally efficient (e.g., generating a 1250-level hierarchy for a 481x321 image in ~0.44s).

4. Experimental Results

The authors evaluated H-SPAM on standard benchmarks (BSD, NYUv2, SBD) using metrics for Accuracy (ASA), Regularity (SRC), Boundary Recall (BR), and Contour Density (CD).

• vs. Hierarchical Methods: H-SPAM significantly outperforms all existing hierarchical methods (e.g., HHTS, SH, RISF) in both accuracy and regularity. It is the only hierarchical method that achieves performance close to non-hierarchical SOTA methods.
• vs. Non-Hierarchical Methods: While constrained by the hierarchy, H-SPAM performs on par with the best non-hierarchical deep learning methods (specifically SPAM). It ranks second in Accuracy (ASA) and Regularity (SRC) behind only the non-hierarchical SPAM, demonstrating that the hierarchical constraint does not severely degrade quality.
• Ablation Studies:
  • Object Priors: Using object priors (SAM/FastSAM) drastically improves accuracy, especially at low superpixel counts ($K$).
  • Spatial Parameter ($w_{pos}$): Tuning the spatial weight allows control over regularity; a default value of $w_{pos}=5$ offers the best trade-off.
  • Attention: Attention maps successfully preserve fine details in salient regions without introducing artifacts when averaged over object priors.

5. Significance

H-SPAM addresses a long-standing gap in computer vision by providing a multi-scale representation that does not compromise on shape quality or boundary accuracy.

• Interpretability: Unlike previous hierarchical methods that produce noisy shapes, H-SPAM generates "easily interpretable" regions that align with semantic objects.
• Downstream Utility: The perfectly nested hierarchy is ideal for tasks requiring progressive refinement, such as interactive image annotation, video tracking, and multi-scale reasoning in transformer-based models.
• Foundation Model Integration: It effectively bridges the gap between high-level foundation models (like SAM) and low-level superpixel segmentation, using the former to guide the latter for superior performance.

In conclusion, H-SPAM sets a new standard for hierarchical segmentation, proving that it is possible to achieve the accuracy of deep learning methods and the regularity of traditional methods simultaneously within a fully hierarchical framework.