Technical Summary: S3POT

Problem Statement

Existing face parsing methods frequently misclassify occlusions (e.g., hands, objects, accessories) as facial components. This occurs because "occlusion" is a high-level spatial concept rather than a concrete object category, making it nearly impossible to construct a real-world dataset covering all possible occlusion types. Furthermore, accurate manual annotation of occlusion masks is labor-intensive. While synthetic data generation has been attempted, it often suffers from limited generalizability to real-world scenarios. Additionally, visual foundation models like the Segment Anything Model (SAM) offer precise segmentation capabilities but require accurate prompts, which are difficult to generate without ground-truth masks.

Methodology: S3POT

The authors propose S3POT (Segmentation via Self-Supervised Prompting Occlusion with conTrast), a contrast-driven framework that synergizes face generation with self-supervised spatial prompting. The method operates without occlusion ground-truth masks during training and consists of three primary modules:

1. Reference Generation (RG)

To create a basis for contrast, the framework generates an occlusion-free "reference" face ($I_r$) from the original occluded image ($I$).

• Mechanism: It utilizes Regional GAN Inversion (RGI) from E4S, conditioned on the original image for texture and a predicted parsing mask ($M_r$) for structure.
• Goal: To preserve facial geometry while realistically reconstructing the occluded regions, creating a clean version of the face to serve as a spatial guide.

2. Feature Enhancement (FE)

This module addresses the limitation that raw feature representations from SAM's image encoder are insufficient to distinguish between face and occlusion via simple contrast.

• Initial Prompting: The system computes cosine similarity between the image tokens of the raw image ($Z$) and the reference image ($Z_r$). The pixel location with the highest similarity is selected as an initial prompt ($p_i$), under the assumption that highly similar regions represent the shared facial structure, while dissimilar regions likely contain occlusion or noise.
• Cross-Attention Fusion: The initial prompt is processed through a feature adjustment component inspired by SAM's mask decoder. This involves:
  • Prompt-to-Image Cross-Attention ($p_iZCA$): Allows the prompt to attend to image features.
  • Image-to-Prompt Cross-Attention ($Zp_iCA$): Allows image features to attend back to the prompt.
  • MLP: Performs non-linear transformations.
• Outcome: This bidirectional fusion produces enhanced image tokens ($Z_e$ and $Z_{re}$) that create a sharper contrast between facial regions and occlusions in the feature space.

3. Prompt Selection (PS)

This module constructs a set of positive (occlusion) and negative (face) prompts for the SAM mask decoder using a Global-Local-Global (GLG) strategy.

• Facial Region Extraction: A facial region mask ($M_f$) is derived from the parsed mask to isolate facial tokens ($Z_{re}^{M_f}$ and $Z_e^{M_f}$).
• Greedy Matching: A greedy matching algorithm (Algorithm 1) pairs tokens between the reference and original facial regions based on similarity.
  • Non-occlusion Prompts ($P_N$): Tokens that successfully match between the reference and original images.
  • Occlusion Prompts ($P_O$): Unpaired tokens in the original image, identified as occlusion candidates.
• Self-Attention Screening: A self-attention network re-weights the combined prompt set ($P$) to filter out redundancy and noise, as excessive prompts can degrade SAM's performance.
• Training Objective: The network is trained using three novel, complementary objective functions without ground-truth occlusion masks:
  1. Occlusion Prompt Recall (OPR): Maximizes the predicted probability for occlusion prompts.
  2. Face Prompt Recall (FPR): Minimizes the predicted probability for face prompts (complementary to OPR).
  3. Face Prompt Penalty (FPP): Specifically penalizes face prompts with anomalously high probabilities to prevent false positives.

Key Contributions

1. Contrast-Driven Framework: A novel approach to face occlusion segmentation that leverages the generative capability of modern face models to create reference images, enabling self-supervised learning without occlusion ground truth.
2. Two-Stage Prompt Strategy: A method combining feature enhancement via cross-attention and a greedy matching algorithm to automatically generate and screen positive/negative prompts for foundation models.
3. Novel Objective Functions: The introduction of OPR, FPR, and FPP to regulate self-attention learning and guide the model toward accurate segmentation in the absence of labeled occlusion data.
4. Dedicated Dataset: The collection of a specialized dataset (1,389 CelebA and 1,104 FFHQ images) with manually annotated occlusion masks for evaluation, addressing the lack of such data in the field.

Experimental Results

The authors evaluated S3POT on their collected dataset against state-of-the-art methods in face parsing, visual reference segmentation, and text-guided reasoning segmentation.

• Performance: S3POT achieved an IoU of 76.2 in standard training mode and 80.4 in per-image overfitting mode.
• Comparisons:
  • Face Parsing: Outperformed adapted models like SegFace* (73.1 IoU) and DML-CSR* (72.4 IoU). Existing parsing models often failed to segment hands or fingers correctly, misclassifying them as skin.
  • Visual Reference Segmentation: Significantly outperformed models like PerSAM, Matcher, and VRP-SAM (which ranged from 21.3 to 51.2 IoU). These models struggled to distinguish occlusions from facial features due to similar skin tones or boundary issues.
  • Reasoning Segmentation: Outperformed LISA and Sesame (approx. 6.5 IoU for occlusion segmentation), which the authors attribute to the models' difficulty in comprehending the linguistic concept of "occlusion" without specific training.
• Ablation Studies:
  • Removing Feature Enhancement (FE) dropped performance significantly (from 76.2 to 56.9), confirming the necessity of feature space discrimination.
  • Removing Prompt Selection (PS) or using fixed thresholds resulted in severe performance degradation, validating the greedy matching and self-attention screening mechanisms.
  • All three objective functions were required for optimal results; using only recall terms led to under-segmentation or over-segmentation.

Significance and Claims

The paper claims that S3POT resolves the ambiguity of occlusion segmentation by shifting the paradigm from direct classification to contrast-driven prompt generation. By leveraging the structural consistency of generated reference faces and the adaptive feature modification of foundation models, the method achieves high-precision segmentation without the prohibitive cost of annotating diverse occlusion categories.

The authors emphasize that their approach is robust to variations in reference image quality, relying on high-level structural guidance rather than pixel-perfect reconstruction. While the method handles common occlusions effectively, the authors modestly note limitations with extremely fine-grain occlusions (e.g., facial hair) or ambiguous boundaries (e.g., translucent objects), suggesting these as areas for future work involving multi-scale feature fusion or uncertainty-aware masking.