LoGoSeg: Integrating Local and Global Features for Open-Vocabulary Semantic Segmentation

1. Problem Statement

Open-Vocabulary Semantic Segmentation (OVSS) aims to assign semantic labels to every pixel in an image for arbitrary categories defined by text prompts, extending beyond the fixed label sets of traditional segmentation.

While Vision-Language Models (VLMs) like CLIP provide zero-shot recognition capabilities, applying them directly to pixel-level segmentation faces three critical challenges:

1. Imprecise Spatial Alignment: VLMs are pre-trained with image-level supervision, lacking explicit constraints to align text concepts with specific visual regions. This leads to blurred boundaries and uncertain predictions, especially in cluttered scenes.
2. Object Hallucination: The absence of strong object priors and region-level constraints often causes models to predict objects that do not exist (false positives) or miss existing objects (false negatives).
3. Feature Integration Imbalance: Existing methods often struggle to effectively combine fine-grained local structural information with high-level global semantic context.

Current solutions fall into two categories:

• Two-stage methods: Rely on external mask proposal generators (e.g., SAM) followed by classification. These are inefficient and dependent on mask quality.
• Single-stage methods: More efficient but often suffer from the alignment and hallucination issues mentioned above.

2. Methodology: LoGoSeg

The authors propose LoGoSeg, a unified, efficient single-stage framework that eliminates the need for external mask proposals, additional backbones, or extra datasets. The architecture consists of four main components:

A. Prior-Guided Regional Alignment

This module addresses object hallucination and improves category discriminability by integrating global priors with local region alignment.

• Object Existence Prior: The model computes a global image-text similarity score for each category. This score acts as a dynamic weight (semantic prior) to reweight the text embeddings. Categories unlikely to exist in the image are suppressed, reducing false positives.
• Region-Level Alignment: The image feature map is partitioned into non-overlapping regions. The model calculates similarity between these regional visual features and the weighted text embeddings.
• Guidance Integration: A lightweight MLP generates spatial attention logits for regions. These are fused with the text guidance to create a Region-Aware Guidance Tensor, which provides precise, category-specific constraints for each spatial region.

B. Contextual Cross-modal Fusion

To overcome the limitations of global features in spatial precision, LoGoSeg employs a Dual-Stream Fusion Mechanism that jointly encodes local structures and global semantics:

• Directional Local Stream: Uses rectangular self-attention (separate horizontal and vertical branches) to capture fine-grained spatial dependencies and directional details.
• Global Context Stream: Utilizes a 2D State-Space Model (SS2D) to capture long-range global contextual dependencies.
• Adaptive Fusion: The outputs of both streams are concatenated and fused with language cues (learnable query embeddings and pooled text embeddings) via linear attention. This creates a unified representation space that balances local detail with global semantics.

C. Decoder

A transformer-based decoder uses learnable queries to interact with the fused feature map. It employs a hierarchical decoding process with learnable upsampling and guidance-modulated convolutions to reconstruct fine-grained segmentation masks at the original resolution.

D. Loss Function

The model is trained using a pixel-wise multi-label binary cross-entropy loss, treating segmentation as a dense prediction task where each pixel is assigned probabilities for all text-defined categories.

3. Key Contributions

1. Region-Aware Alignment with Object Priors: A novel strategy that dynamically weights categories based on global image-text similarity and enforces precise region-level visual-textual correspondences. This significantly reduces classification hallucinations.
2. Dual-Stream Feature Fusion: A mechanism that effectively combines detailed local spatial context (via directional attention) with comprehensive global semantics (via state-space modeling), overcoming the limitations of single-stream approaches.
3. Unified Single-Stage Framework: LoGoSeg achieves state-of-the-art performance without relying on external mask proposals (like SAM), auxiliary backbones, or additional training datasets, ensuring high computational efficiency.

4. Experimental Results

The authors evaluated LoGoSeg on six standard benchmarks: A-847, PC-459, A-150, PC-59, PAS-20, and PAS-20b.

• Performance: LoGoSeg achieved State-of-the-Art (SOTA) results across all benchmarks.
  • Using a ViT-B/16 backbone, it outperformed the best one-stage baseline (SED) by 1.9 mIoU on PC-459 and the best two-stage competitor (MAFT+) by 4.2 mIoU on PC-59.
  • With a larger ViT-L/14 backbone, it achieved 97.4 mIoU on PAS-20 and 83.3 mIoU on PAS-20b.
• Scalability: The method demonstrated near-linear performance improvements when scaling from medium (ViT-B) to large (ViT-L) backbones, confirming the robustness of its design.
• Generalization: Even when trained on limited subsets of data (e.g., only PC-59 or A-150 categories), LoGoSeg maintained strong generalization to unseen categories, outperforming competitors like ZegFormer and CAT-Seg.
• Ablation Studies: Experiments confirmed that removing any of the three core components (Object Prior, Cross-modal Fusion, or Region Alignment) led to significant performance drops, validating the necessity of each module.

5. Significance

LoGoSeg represents a significant advancement in open-vocabulary segmentation by solving the "hallucination" and "alignment" problems inherent in VLM-based segmentation.

• Efficiency: By removing the two-stage dependency on external mask generators, it offers a more streamlined and computationally efficient pipeline.
• Robustness: The integration of object priors and dual-stream fusion makes the model highly robust in complex, cluttered, and ambiguous scenes where previous methods fail.
• Practicality: Its ability to achieve SOTA results without requiring massive external datasets or auxiliary modules makes it a highly practical solution for real-world deployment in open-vocabulary scenarios.

In conclusion, LoGoSeg successfully bridges the gap between global semantic understanding and local pixel-level precision, setting a new benchmark for single-stage open-vocabulary semantic segmentation.