Decouple and Rectify: Semantics-Preserving Structural Enhancement for Open-Vocabulary Remote Sensing Segmentation

1. Problem Statement

Open-Vocabulary Semantic Segmentation (OVS) in remote sensing (RS) aims to segment objects using text prompts for categories unseen during training. While Vision-Language Models like CLIP provide robust semantic generalization, they suffer from two critical limitations when applied to RS imagery:

1. Lack of Structural Fidelity: CLIP features, trained on image-level alignment, often fail to capture the fine-grained spatial details and precise boundaries required for dense RS segmentation (e.g., building edges, thin roads).
2. Ineffective Integration of Structural Priors: Existing methods attempt to compensate by fusing DINO features (which capture strong structural details) with CLIP features. However, current approaches treat CLIP features as a monolithic, uniform semantic space. They inject structural cues indiscriminately, which often disrupts the language-aligned semantics of CLIP, leading to "diffused" activation maps and coarse boundaries.

Core Insight: The authors observe that CLIP feature channels are functionally heterogeneous. Some channels are critical for semantic alignment ("positive channels"), while others contribute less to semantics and may even blur spatial responses ("negative channels"). Existing methods fail to distinguish between these, causing semantic degradation when structural priors are added.

2. Methodology: DR-Seg Framework

The proposed DR-Seg framework introduces a "Decouple-and-Rectify" strategy to achieve structural enhancement without compromising semantic integrity. The pipeline consists of three main stages:

A. Semantic-Preserving Subspace Decoupling (SPSD)

Instead of treating CLIP features as a single block, DR-Seg explicitly partitions them into two subspaces based on channel-wise functional heterogeneity:

• Mechanism: The method calculates a semantic score for each channel using two metrics derived from class prototypes:
  1. Cross-Class Entropy: Measures how selectively a channel responds to specific classes (lower entropy = higher selectivity).
  2. Inter-Class Similarity: Measures how well a channel distinguishes between different classes.
• Action: Channels are ranked by these scores. The top-ranked channels form the Semantics-Dominated Subspace ($F_{sem}$), which is preserved unchanged to maintain language alignment. The remaining channels form the Structure-Dominated Subspace ($F_{str}$), which is routed for enhancement.

B. Prior-Driven Graph Rectification (PDGR)

This module selectively enhances the structure-dominated subspace using DINO features as structural priors.

• Graph Construction: A sparse affinity graph is built over the spatial locations of $F_{str}$. The affinity between nodes is determined by DINO feature similarity and spatial proximity.
• Rectification: A learnable graph neural network propagates information along the edges of this graph. This allows the network to adapt DINO's structural priors to the specific geometric layouts of RS images (e.g., dense objects, arbitrary orientations) without using standard convolutions that might blur boundaries.
• Recomposition: The rectified structural features ($\hat{F}_{str}$) are recombined with the preserved semantic features ($F_{sem}$) to form a Refined Branch.

C. Uncertainty-Guided Adaptive Fusion (UGAF)

The final prediction combines the Original CLIP Branch (robust semantics) and the Refined Branch (enhanced structure).

• Adaptive Strategy: A uniform fusion is suboptimal because some regions are semantically clear but structurally vague, while others are ambiguous.
• Mechanism: An uncertainty map is generated from the original CLIP predictions (using entropy).
  • Low Uncertainty (Confident): The original CLIP branch dominates.
  • High Uncertainty (Ambiguous): The refined branch (with structural corrections) is weighted more heavily.
• Multi-Rotation Handling: To handle RS object orientations, the CLIP branch processes multi-rotated inputs, while the DINO-guided rectification is applied only to the canonical (0°) branch to save computational cost, with the results fused adaptively.

3. Key Contributions

1. Discovery of Functional Heterogeneity: The paper identifies and validates that CLIP channels are not uniform; they contain distinct "semantic" and "structural" components. This challenges the assumption that structural priors can be injected globally.
2. DR-Seg Framework: A novel architecture that:
   • Decouples CLIP features into semantic and structural subspaces.
   • Rectifies only the structural subspace using DINO-guided graph learning.
   • Fuses the branches adaptively based on pixel-wise uncertainty.
3. State-of-the-Art Performance: The method establishes new benchmarks across eight diverse remote sensing datasets, demonstrating superior generalization and boundary precision compared to existing OVS methods.

4. Experimental Results

The method was evaluated on 8 benchmarks (DLRSD, iSAID, Potsdam, Vaihingen, LoveDA, UAVid, UDD5, VDD) using CLIP ViT-B and ViT-L backbones.

• Quantitative Performance:
  • DLRSD Training: DR-Seg achieved +2.83% mIoU improvement over the previous best (RSKT-Seg) with ViT-L.
  • iSAID Training: DR-Seg achieved +3.86% mIoU improvement over the previous best with ViT-L.
  • Cross-Dataset Generalization: Significant gains were observed on unseen datasets (e.g., UDD5, VDD), proving the method's robustness to domain shifts.
• Qualitative Analysis: Visualizations show that DR-Seg produces sharper boundaries, fewer category confusions, and better recovery of small/thin structures (e.g., cars, roads) compared to GSNet and RSKT-Seg.
• Ablation Studies:
  • Removing SPSD (decoupling) caused a significant performance drop, confirming the necessity of selective enhancement.
  • Removing UGAF (adaptive fusion) reduced performance, highlighting the importance of balancing semantics and structure based on uncertainty.
  • The optimal semantic ratio ($\rho$) was found to be 0.5, indicating a balanced partition is most effective.

5. Significance

This work provides a paradigm shift in Open-Vocabulary Remote Sensing Segmentation. By moving away from "holistic fusion" to "selective, semantics-preserving enhancement," DR-Seg solves the fundamental conflict between CLIP's semantic strength and its structural weakness. It demonstrates that understanding the internal functional heterogeneity of foundation models is crucial for adapting them to specialized domains like remote sensing, where precise spatial delineation is as important as semantic recognition. The proposed "Decouple-and-Rectify" strategy offers a generalizable blueprint for integrating structural priors into vision-language models without sacrificing their core semantic capabilities.