🤖 AI

Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation

The paper proposes "Seeking Consensus" (SeeCo), a plug-and-play, training-free framework that enhances open-vocabulary remote sensing semantic segmentation by dynamically recalibrating models on-the-fly through geometric and semantic consensus learning to resolve scene-specific ambiguities and improve foreground activation.

Original authors: Guanchun Wang, Chenxiao Wu, Xiangrong Zhang, Zelin Peng, Jianxun Lai, Tianyang Zhang, Xu Tang

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guanchun Wang, Chenxiao Wu, Xiangrong Zhang, Zelin Peng, Jianxun Lai, Tianyang Zhang, Xu Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to identify objects in a photo taken from a drone high above a city. In a normal photo, a car always looks like a car. But in a drone photo, a car could be facing north, south, east, or west, and it might look completely different depending on the angle. Similarly, a "building" could be a tiny shed or a massive skyscraper, and the words we use to describe them might not match what the computer sees.

This paper introduces a new tool called SeeCo (Seeking Consensus) to help computers get better at this task, specifically for "Open-Vocabulary Semantic Segmentation." That's a fancy way of saying: "Helping a computer label every pixel in a satellite image with a name, even if it hasn't been trained on that specific name before."

Here is how SeeCo works, using simple analogies:

The Problem: The "Static" Observer

Most current computer vision models are like a static security guard standing in one spot. They look at an image once, from one angle, with one description in mind.

The Rotation Problem: If a car is parked diagonally, the guard might miss it because they only learned to recognize cars facing straight ahead.
The Description Problem: If the guard is told to look for a "building," they might only recognize a standard house. They might miss a skyscraper or a warehouse because the description was too narrow.
The Result: The computer gets confused, misses parts of objects, or labels things incorrectly.

The Solution: The "Dynamic Team" (SeeCo)

SeeCo is like hiring a team of experts who work together while looking at the image, rather than just relying on a single, pre-trained memory. It doesn't require retraining the whole team (which is expensive and slow); instead, it adjusts their strategy on the fly for every single new image.

The team uses two main strategies to reach a "Consensus" (an agreement on what is actually there):

1. Geometric Consensus (The "Spin-Doctor" Strategy)

The Analogy: Imagine you are trying to identify a weirdly shaped rock. Instead of looking at it once, you spin it around in your hands, looking at it from the top, the side, and the back. You then combine all those views to get a complete picture.
How SeeCo does it:

It takes the satellite image and creates several "rotated" versions of it (like spinning the photo).
It asks the computer to label the object in every single rotated version.
It then averages these answers to create a "Geometric Consensus."
The Benefit: If the computer missed a building because it was rotated, the rotated views will catch it. This ensures the computer doesn't miss objects just because they are facing a different direction.

2. Semantic Consensus (The "Word-Game" Strategy)

The Analogy: Imagine you are trying to describe a "vehicle" to a friend who has never seen a truck. If you just say "vehicle," they might think of a bicycle. But if you say "vehicle, like a truck, a bus, or a heavy transport," they get a much clearer picture.
How SeeCo does it:

The computer usually relies on a simple label like "building."
SeeCo uses a Large Language Model (like a super-smart AI chatbot) to generate a rich list of synonyms and descriptions for that category (e.g., "residential unit," "skyscraper," "structure").
It feeds these richer descriptions into the computer's brain to help it understand the variety of things that fit that label.
The Benefit: This helps the computer recognize that a "building" can look very different from a "road" or "grass," reducing confusion.

The "Online Consensus Injector" (The Manager)

Once the team has gathered all these different views and rich descriptions, they need to combine them into a final decision.

Think of this as a manager who takes the input from the "Spin-Doctor" (geometric views) and the "Word-Game" (rich descriptions) and blends them together.
This manager makes tiny, instant adjustments to the computer's brain while it is looking at the image. It's like a GPS recalculating your route in real-time because of traffic, rather than following a static map.

The Results

The authors tested this "dynamic team" approach on eight different datasets of satellite and drone images.

No Extra Training: They didn't have to retrain the computer from scratch. They just plugged SeeCo in like a new app.
Better Accuracy: The computer made fewer mistakes, especially with objects that were rotated or had unusual appearances.
Universal: It worked well with different existing computer models, proving it's a flexible tool.

In short: SeeCo stops the computer from being a rigid, single-perspective observer. Instead, it turns the computer into a flexible thinker that looks at an image from multiple angles and uses a richer vocabulary to understand exactly what it's seeing, all in real-time.

1. Problem Statement

Open-Vocabulary Semantic Segmentation (OVSS) in remote sensing aims to identify land cover categories using textual descriptions without requiring pixel-level annotations for every new category. While Vision-Language Models (VLMs) like CLIP have enabled this, existing methods face critical limitations in the remote sensing domain:

Static Inference Paradigm: Current methods apply a fixed inference process to all images, ignoring the unique distribution and characteristics of each specific scene.
Geometric Ambiguity: Unlike natural images (object-centric, fixed orientation), remote sensing images (RSIs) are captured from a bird's-eye view with arbitrary orientations and complex layouts. Static models fail to maintain rotation invariance, leading to incomplete foreground activation (under-activation) when objects are rotated.
Semantic Bias: Land cover categories (e.g., "building") exhibit high intra-class heterogeneity (e.g., dense housing vs. skyscrapers). Static textual descriptions (e.g., just the category name) are insufficient to capture this diversity, causing a mismatch between the text encoder (pre-trained on natural scenes) and the visual features of RSIs.

2. Methodology: The SeeCo Framework

The authors propose SeeCo, a plug-and-play, training-free framework that enhances existing OVSS models during inference (on-the-fly) by seeking dual consensus: Geometric and Semantic. It does not require retraining the base model but recalibrates it for each specific scene.

Core Components:

Geometric Consensus Learning (GCL):
- Goal: To enforce rotation-invariant representation and alleviate under-activation.
- Mechanism: The input image is rotated $K$ times to simulate multi-view observations. The model generates segmentation maps for each view.
- Consensus Target: These maps are inverse-transformed and aggregated to create a robust Geometric Consensus Target ( $Y_{GCL}$ ). This serves as a pseudo-mask to supervise the model, ensuring consistent object representation regardless of orientation.
Semantic Consensus Learning (SCL):
- Goal: To alleviate semantic bias and enrich textual descriptions without tuning the frozen text encoder.
- Mechanism: A Multi-modal Collaborative Prompting strategy is used. A Large Language Model (LLM) is guided by visual priors (sample images) and textual knowledge to generate rich, diverse synonyms and descriptions for each land cover category (e.g., expanding "building" to "residential unit," "skyscraper," etc.).
- Adaptive Recalibration: An adaptive context mechanism fuses the original text embeddings with these enriched embeddings to create a Semantic Consensus Target ( $Y_{SCL}$ ), better matching the scene's specific visual diversity.
Online Consensus Injector (OCI):
- Function: A lightweight module that injects the consensus knowledge into the base OVSS model during inference.
- Structure:
  - Text Side: Uses trainable embeddings (scene-adaptive contexts) to fuse enriched text.
  - Vision Side: Uses a Low-Rank Adaptation (LoRA) branch to fine-tune the last few Transformer blocks of the vision encoder.
- Optimization: The OCI is optimized online via a self-supervised loss function that minimizes the difference between the model's current predictions and the constructed Geometric ( $Y_{GCL}$ ) and Semantic ( $Y_{SCL}$ ) consensus targets.
- Final Output: The final segmentation map is a weighted combination of the geometric and semantic consensus predictions.

3. Key Contributions

Novel Framework (SeeCo): The first plug-and-play framework for remote sensing OVSS that performs on-the-fly recalibration during inference, breaking the static paradigm.
Dual Consensus Mechanism: Introduces GCL (via multi-view rotation consistency) and SCL (via LLM-driven textual enrichment) to jointly address geometric and semantic mismatches.
Training-Free & Universal: Requires no offline training or manual annotations. It can be applied to any existing training-free OVSS model to boost performance.
Lightweight Design: Uses an Online Consensus Injector with low-rank layers, adding minimal parameters and computational overhead.

4. Experimental Results

The method was evaluated on eight diverse remote sensing benchmarks (including OpenEarthMap, LoveDA, iSAID, Potsdam, Vaihingen, UAVid, UDD, and VDD).

Performance Gains:
- When integrated with ProxyCLIP, SeeCo achieved a new state-of-the-art (SOTA) average mIoU of 42.4%, outperforming the previous best by 1.7%.
- It showed significant improvements on challenging datasets like Vaihingen (up to 11.9% gain), effectively narrowing the domain gap between natural scene pre-training and remote sensing data.
- Consistent gains were observed across all eight datasets when applied to different base models (ClearCLIP, ProxyCLIP, SegEarth-OV).
Ablation Studies:
- Both GCL and SCL contributed positively. GCL improved rotation robustness, while SCL significantly boosted performance on datasets with high intra-class heterogeneity.
- The combination of both mechanisms yielded the best results, demonstrating their complementary nature.
Efficiency:
- While inference time increases slightly due to the on-the-fly optimization (e.g., from 0.023s to 0.518s per image on ClearCLIP), the parameter increase is negligible (e.g., +0.25M parameters).
- The method effectively reduces feature confusion and improves the integrity of segmented regions (visualized via t-SNE and qualitative maps).

5. Significance

Paradigm Shift: Moves OVSS from a static, one-size-fits-all inference approach to a dynamic, scene-adaptive approach, which is crucial for the variable nature of remote sensing imagery.
Domain Adaptation: Successfully bridges the gap between VLMs pre-trained on natural images and the specific challenges of remote sensing (arbitrary orientation, intra-class diversity) without expensive retraining.
Practical Deployment: As a plug-and-play solution, it allows existing models to be immediately upgraded for better performance in real-world applications like urban planning, disaster assessment, and precision agriculture.