Technical Summary: Toward Semantic-Agnostic and Shape-Aware Vision-Language Segmentation Models

1. Problem Statement

Recent advancements in Vision-Language Models (VLMs) have enabled segmentation tasks guided by natural language instructions. However, current state-of-the-art models (e.g., LISA, GSVA) rely heavily on high-level semantic object categories (e.g., "stop sign," "dog") found in standard datasets like COCO or LVIS. This semantic dependence creates a significant limitation: these models struggle to reason about intrinsic visual properties such as shape, geometry, texture, or color when explicit semantic labels are absent.

Consequently, existing VLMs fail to handle non-semantic queries effectively. For instance, a model trained primarily on semantic categories may fail to segment a "red octagonal object" if it cannot align the description with a specific class label like "stop sign." This limitation restricts the generalization and controllability of segmentation models in real-world applications where precise control over visual properties (e.g., anatomical constraints, specific geometric shapes) is required without predefined semantic classes.

2. Methodology

The authors introduce SANSA (Semantic-Agnostic aNd Shape-Aware) segmentation, a new paradigm requiring models to segment objects based solely on perceptual descriptions (shape, color, texture, spatial properties) rather than semantic categories. The methodology consists of two primary stages:

A. SANSA Segmentation Prompt Generation

To address the lack of semantic-agnostic datasets, the authors propose an automated pipeline to generate segmentation prompts using a description VLM (InternVL 2.5). To prevent the VLM from leaking semantic information (e.g., naming the object), two complementary strategies are employed:

1. DISP (Dictionary-based Segmentation Prompts):

   • Constraint: The description VLM is forced to generate tokens exclusively from a predefined dictionary of semantic-agnostic attributes (colors, textures, shapes, patterns, lighting, and connectors).
   • Post-processing: An external LLM (Mistral 7B) reformulates these dictionary-constrained outputs into fluent, natural segmentation prompts while strictly preserving the original perceptual constraints.
   • Input Preprocessing: Target objects are cropped using their binary masks before being fed to the VLM to minimize contextual semantic leakage.

2. EXSP (Example-based Segmentation Prompts):

   • Guidance: Instead of a fixed dictionary, the VLM is guided by positive examples (semantic-agnostic descriptions) and negative examples (descriptions containing semantic words) within the instruction prompt.
   • LLM-as-a-Judge (LLMJ) Filtering: Since the EXSP strategy is unconstrained, a filtering step is applied where a separate LLM classifies generated descriptions as "semantic" or "semantic-agnostic" based on text alone. Descriptions identified as containing semantic information are removed from the dataset.

B. Model Finetuning

The authors finetune a state-of-the-art segmentation VLM, LISA 7B, using the generated SANSA prompts.

• Architecture: They employ LoRA (Low-Rank Adaptation) to finetune the multimodal model (LLaVA) responsible for generating segmentation tokens and the vision decoder.
• Loss Function: The training utilizes a weighted combination of pixelwise binary cross-entropy (BCE) and Dice loss.
• Data: The models are trained on a subset of COCO (10k images) converted into SANSA prompts, with evaluation performed on held-out test sets and a manually curated "Human Prompt" (HP) set.

3. Key Contributions

• SANSA Segmentation Paradigm: The introduction of a segmentation task that relies exclusively on visual properties (shape, geometry, color, texture) without semantic object categories.
• Automated Prompt Generation Strategies: The proposal of two scalable strategies (DISP and EXSP) to generate semantic-agnostic segmentation prompts, overcoming the difficulty of manual annotation for non-semantic descriptions.
• Performance Improvement: Demonstration that finetuning on SANSA prompts significantly improves segmentation accuracy on non-semantic tasks compared to pretrained models, while maintaining performance on standard semantic tasks.

4. Experimental Results

Experiments were conducted on a subset of the COCO dataset (2k test images) and a Human Prompt (HP) set (160 images).

• Quantitative Performance:
  • Finetuned SANSA models achieved up to a 20% improvement in mIoU on the SANSA task compared to pretrained state-of-the-art models (LISA, GSVA, PolyFormer).
  • The EXSP strategy yielded the highest average mIoU (37.46% on automatic test sets, 42.98% on HP), suggesting it generates prompts that generalize well to human-written semantic-agnostic descriptions.
  • The DISP strategy showed robust performance, particularly on the filtered EXSP+LLMJ test set, confirming the effectiveness of the dictionary constraint.
• Qualitative Performance:
  • Pretrained models (LISA, GSVA) failed to interpret semantic-agnostic prompts (e.g., "Segment the red octagonal object"), often failing to segment the target or segmenting incorrect objects.
  • Finetuned SANSA models accurately segmented objects based on shape and color descriptions alone.
• Robustness and Generalization:
  • Semantic Preservation: Finetuning on SANSA prompts did not degrade performance on the original semantic segmentation task; mIoU on standard semantic prompts remained comparable to the pretrained LISA baseline (less than 1% drop).
  • Filtering Efficacy: The LLMJ filter achieved 82.5% accuracy and 94% precision in detecting semantic leakage in EXSP prompts.

5. Significance

The paper argues that the current reliance on semantic alignment in VLMs limits their ability to reason about intrinsic visual properties. By shifting the focus to low- and mid-level visual reasoning, SANSA segmentation demonstrates that models can be made more generalizable and controllable.

The authors highlight that this approach is particularly valuable for domains where semantic labels are unavailable, insufficient, or where precise control over segmentation properties (such as the number of connected components or specific geometric constraints) is necessary. The work establishes a foundation for extending VLMs beyond semantic-only understanding, enabling them to operate effectively on purely perceptual descriptions.