Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation

Technical Summary: Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation

Problem Statement
Open-vocabulary segmentation (OVS) models, while capable of generalizing to unseen category names via vision-language pretraining, struggle to generalize to unseen combinations of object categories and attributes (e.g., distinguishing a "flat roof industrial building" from a "flat roof residential building" when the specific combination was not seen during training). Existing approaches typically encode the entire textual prompt as a single holistic sequence. This holistic encoding entangles category and attribute semantics within a shared representation, preventing the model from decomposing and recombining semantic units at inference time. Consequently, models fail to enforce the logical "AND" constraints required for fine-grained compositional reasoning, leading to degraded performance on unseen attribute–category compositions.

Methodology
The authors propose a Decomposed Vision-Language Alignment framework designed to explicitly factorize textual prompts and enforce compositional constraints at both the feature and scoring levels. The method is model-agnostic but instantiated here within the SAM3 architecture.

1. Explicit Prompt Decomposition:
   Instead of concatenating text into a single sentence, the framework encodes the category and each attribute independently. To resolve geometric ambiguity (e.g., "flat roof" applying to any flat surface), attributes are conditioned on the category during encoding (e.g., encoding "flat roof" as "flat roof building"). This yields a structured representation consisting of a category token sequence ($t_c$), a set of attribute token sequences ($t_{a_k}$), and an attribute presence mask ($m$).

2. Feature-Gated Cross-Attention:
   The framework performs decomposed cross-attention between visual queries and each semantic unit.

   • Category Interaction: Standard cross-attention is used to align queries with the category token.
   • Attribute Gates: For each attribute, a separate cross-attention branch generates a gating map ($G_{a_k}$) via a sigmoid activation. These gates act as "necessary condition filters."
   • AND Composition: The final feature output is obtained by element-wise multiplication of the category-aligned features and all active attribute gates ($F_{out} = Q_c \odot \prod G_{a_k}$). This multiplicative aggregation enforces that a visual feature is preserved only if it satisfies all attribute constraints simultaneously.

3. Log-Space AND Compositional Scoring:
   To avoid numerical instability and gradient vanishing associated with multiplying probabilities, the scoring mechanism operates in log-space.

   • Matching logits for the category and each attribute are computed independently.
   • These logits are converted to probabilities via sigmoid, then mapped to log-probabilities.
   • The final compositional score is the sum of these log-probabilities, normalized by the number of active components. This formulation mathematically approximates the logical AND operation ($\log(\prod \pi) = \sum \log(\pi)$) while maintaining stable gradients during optimization.

Key Contributions

• Decomposed Vision-Language Alignment: A framework that integrates seamlessly into transformer-based segmentation architectures, explicitly decoupling category and attribute interactions to improve fine-grained OVS.
• Feature-Gated Cross-Attention: A novel module where attributes function as spatial filters, decoupling attribute-specific feature learning and enforcing compositional constraints via multiplicative gating.
• Log-Space AND Compositional Scoring: A scoring strategy that explicitly models logical AND relationships among semantic units, offering numerical stability and improved optimization behavior compared to direct probability multiplication.
• Compositional Generalization Evaluation Protocol: The introduction of a rigorous evaluation protocol that splits data into seen and unseen attribute–category compositions, utilizing metrics such as unseen AP, Seen–Unseen Gap ($R_d$), and AND-Efficiency to measure compositional reasoning capabilities.

Experimental Results
The method was evaluated on the UBC building dataset and the PACO-LVIS dataset under a compositional generalization setting where specific attribute–category combinations were held out from training.

• Performance Gains: On the UBC dataset, the proposed method increased the unseen AP from 4.0 (SAM3 baseline) to 7.1, while the AND-Efficiency metric improved significantly from 0.533 to 0.959. On PACO-LVIS, unseen AP rose from 37.9 to 39.2, and AND-Efficiency from 0.874 to 0.907.
• Ablation Studies: Experiments confirmed that holistic encoding limits generalization. Replacing holistic encoding with independent encoding and mean aggregation improved results, but the full Log-Space AND mechanism provided the most significant gains. Furthermore, the method outperformed adapted Compositional Zero-Shot Learning (CZSL) methods (e.g., AttrAsOp, CGE), demonstrating that pixel-level spatial gating is superior to global composition vectors for dense prediction tasks.
• Qualitative Analysis: Visualizations of the feature gates showed that the model successfully learned to suppress instances that matched the category but violated specific attribute constraints (e.g., suppressing commercial buildings when the prompt required "residential"), confirming the effectiveness of the multiplicative gating mechanism.

Significance
The paper claims that its primary significance lies in addressing the systematic bias of vision-language models in fine-grained compositional reasoning. By transferring the decomposition insight from image-level Compositional Zero-Shot Learning to the pixel-level domain of segmentation, the work demonstrates that explicit factorization of semantic units and the enforcement of logical constraints at the feature level are critical for generalizing to unseen compositions. The authors posit that this framework provides a robust foundation for developing more reliable open-world perception systems capable of handling complex, structured semantic descriptions.