Technical Summary: Weakly Supervised Segmentation as Semantic-Based Regularization

Problem Statement

Semantic segmentation requires dense, pixel-level annotations, which are expensive and time-consuming to produce, creating a bottleneck for deploying models in new domains. Weakly Supervised Semantic Segmentation (WSSS) attempts to mitigate this by training on cheaper, indirect cues like image-level tags, bounding boxes, or scribbles. However, these weak labels often introduce ambiguity regarding object boundaries and shapes, making it difficult for models to produce coherent segmentations without structural guidance.

While foundation models like the Segment Anything Model (SAM) have reshaped WSSS by generating high-quality pseudo-labels from simple prompts, they face significant limitations. Standard SAM relies on fixed, pre-trained weights that struggle with domain shifts (e.g., medical imaging) and lacks a principled mechanism to incorporate domain-specific knowledge or heterogeneous weak labels. Current approaches often depend on heuristic prompt designs rather than a general reasoning mechanism for spatial and semantic structure.

Methodology

The authors propose a neurosymbolic framework that integrates differentiable fuzzy logic with deep segmentation models to address these gaps. The method operates in a two-stage pipeline:

1. Logic-Guided Fine-Tuning of Foundation Models

In the first stage, the Segment Anything Model (SAM) is fine-tuned to satisfy a set of differentiable fuzzy logic constraints derived from weak annotations and structural priors.

• Neurosymbolic Formulation: Weak annotations (bounding boxes, scribbles) and domain priors (smoothness, superpixel boundaries, geometric shapes) are unified as continuous logical constraints.
• Differentiable Logic: Instead of treating logical formulas as strict True/False statements, the method uses product t-norm semantics to relax them into continuous functions. This allows the logical satisfaction probability to be optimized end-to-end using standard gradient-based optimizers.
• Constraint Types:
  • Weak Supervision: Scribbles (pixel-level consistency), Bounding Boxes (tightness constraints ensuring every row/column contains the target class), and Background exclusion.
  • Unsupervised Structural Priors: Smoothness (neighbor consistency, region filling), Superpixel boundaries (penalizing label changes within superpixels), and Domain-specific geometric priors (e.g., "corner constraints" for circular medical structures like optic discs).
• Training: Only the mask decoder of SAM is fine-tuned while the image encoder remains frozen. The model minimizes the semantic loss (negative log-likelihood of the formula being satisfied) to generate refined, structurally consistent pseudo-masks.

2. Pseudo-Label Supervised Training

In the second stage, the refined pseudo-labels generated by the logic-guided SAM are treated as ground truth. A standard, prompt-free segmentation network (e.g., DeepLabV2 or Mask2Former) is trained from scratch on these pseudo-labels using conventional cross-entropy loss. This yields a final model capable of deployment without human prompts.

Key Contributions

1. Neurosymbolic Formulation: The paper introduces a formulation of WSSS using differentiable fuzzy logic, unifying heterogeneous supervision signals (boxes, scribbles) and structural priors into a single optimization objective.
2. General Training Scheme: A method to fine-tune foundation models (SAM) to integrate multiple weak annotation types and priors jointly, moving beyond heuristic prompt engineering.
3. Empirical Validation: Demonstration that reasoning-guided learning produces higher-quality pseudo-labels and achieves state-of-the-art (SOTA) segmentation accuracy, often exceeding densely supervised baselines.

Experimental Results

The framework was evaluated on Pascal VOC 2012 (natural images) and REFUGE2 (retinal fundus images for optic disc/cup segmentation).

• Pascal VOC 2012:

  • Pseudo-Label Quality: The logic-guided fine-tuning achieved 94.5% mIoU on pseudo-labels, significantly outperforming prompt-based baselines (e.g., 91.5% for box-prompted SAM).
  • Final Segmentation: The second-stage Mask2Former model achieved 89.1% mIoU on the test set, surpassing the fully supervised Mask2Former baseline (86.1%) and all other WSSS methods.
  • Ablation: Removing scribble or border constraints caused the most significant performance drops, highlighting their critical role.

• REFUGE2 (Medical Imaging):

  • Adaptability: Zero-shot MedSAM performed poorly (47.7% mean Dice). Logic-guided fine-tuning using only weak labels (boxes and points) improved performance to 87.5% mean Dice, closely approaching the fully supervised MedSAM upper bound (88.7%).
  • Final Performance: The second-stage network trained on these pseudo-labels achieved 87.1% mean Dice, outperforming fully supervised ResUNet (86.5%) and showing distinct advantages in segmenting smaller structures (optic cups).

Significance and Claims

The paper claims that integrating differentiable fuzzy logic with foundation models bridges the gap between the flexibility of deep networks and the interpretability of symbolic reasoning. By grounding logical constraints on weak cues, the model learns not only to fit annotations but also to satisfy high-level spatial and relational priors.

The authors assert that this approach:

• Produces pseudo-labels of higher quality than those generated by zero-shot foundation models or standard weak supervision pipelines.
• Enables foundation models to adapt to out-of-distribution domains (like medical imaging) using only weak supervision, eliminating the need for pixel-perfect ground truth during the adaptation phase.
• Achieves SOTA performance on standard benchmarks, often exceeding the performance of models trained with dense supervision.

The paper acknowledges limitations, noting that the framework's best performance relies on domain-specific logical constraints which may require expert knowledge. It suggests that future work could explore using Large Language Models (LLMs) to generate these constraints automatically or extending the approach to even weaker supervision signals like image-level tags.