SATGround: A Spatially-Aware Approach for Visual Grounding in Remote Sensing

1. Problem Statement

Vision-Language Models (VLMs) have shown promise in remote sensing for tasks like image captioning and visual question answering (VQA). However, a critical bottleneck remains in Visual Grounding: the ability to precisely localize objects in satellite imagery based on natural language queries.

• Limitation of Current Approaches: Existing methods typically treat bounding box coordinates as text tokens (e.g., converting numbers like [43, 37, 85, 57] into a sequence of tokens).
• The Core Issue: This text-based approach lacks explicit geometric structure. Large Language Models (LLMs) struggle with numerical inference and spatial reasoning due to tokenization limitations and the lack of a native spatial metric. Consequently, these models often fail to understand proximity, scale, or rotation, leading to poor localization robustness, especially in complex satellite scenes with arbitrary object orientations.

2. Methodology: SATGround

The authors propose SATGround, a framework that integrates multi-modal learning with a specialized, structured localization mechanism. Instead of forcing the language model to "spell out" coordinates, SATGround decouples text generation from spatial regression.

Key Architectural Components:

1. Dual-Modality Output:
   The model is designed to output both a textual response ($a$) and a set of bounding box coordinates ($\ell$) simultaneously.
   $$\mathcal{M}_{loc}: (\mathbf{x}, \mathbf{q}) \mapsto (\mathbf{a}, \mathbf{\ell})$$
   where $\mathbf{x}$ is the satellite image and $\mathbf{q}$ is the text query.

2. Special Control Tokens:
   The vocabulary is extended with two dedicated tokens:

   • <bb> (Bounding Box): Indicates the start of a localization prediction in the text stream.
   • <loc> (Location): A placeholder token that follows <bb>.
   • Mechanism: When the model generates <bb>, it deterministically appends <loc>. The latent embedding vector corresponding to the <loc> token is not used for next-token prediction. Instead, it is extracted and passed to a dedicated Grounding Module.

3. Grounding Module ($\phi_{grounding}$):

   • A lightweight, trainable Multi-Layer Perceptron (MLP) that takes the latent embeddings of <loc> tokens.
   • It regresses the final Oriented Bounding Box (OBB) coordinates directly in continuous space.
   • OBB Representation: Unlike standard axis-aligned boxes, SATGround predicts 5-tuples $(x_1, y_1, x_2, y_2, \theta)$ to handle the arbitrary rotation of objects common in satellite imagery. Coordinates are normalized to $[0, 100]$ and angles to $[0, 180^\circ)$.

4. Training Strategy:

   • Loss Function: The model is optimized using a combined loss:
     $$\mathcal{L} = \lambda_{text}\mathcal{L}_{CE} + \lambda_{bb}\mathcal{L}_{ground}$$
     where $\mathcal{L}_{CE}$ is the standard cross-entropy loss for text, and $\mathcal{L}_{ground}$ is the regression loss for bounding boxes.
   • Bipartite Matching (Hungarian Algorithm): To handle multiple objects in a single query, the authors use a grouped bipartite matching strategy. Predicted boxes are matched to ground truth within specific "referring expression groups" (consecutive <bb> tokens) to prevent semantic mismatches (e.g., matching a "tennis court" prediction to a "basketball court" ground truth).

5. Inference:
   During inference, the model generates text greedily. Upon generating a <loc> token, the system pauses text generation, extracts the embedding, queries the grounding module to get the coordinates, and then resumes.

3. Key Contributions

1. Novel Localization Mechanism: Introduction of a structured grounding interface using <bb> and <loc> tokens that separates linguistic reasoning from geometric regression, avoiding the pitfalls of text-based coordinate prediction.
2. Generalist Framework: Adaptation of a pretrained generalist VLM (using LoRA for efficient fine-tuning) to handle both instruction-following and precise object detection in remote sensing.
3. State-of-the-Art Performance: Significant improvements across multiple benchmarks, particularly in visual grounding accuracy.
4. Orientation Awareness: Explicit modeling of Oriented Bounding Boxes (OBBs), which is crucial for remote sensing where objects (ships, planes, buildings) are rarely axis-aligned.

4. Experimental Results

The framework was evaluated on several remote sensing benchmarks, including GeoChat, EarthDial, NWPU VHR-10, and Swimming Pool.

• Visual Grounding (Referring Expression Detection):

  • On the GeoChat benchmark, SATGround achieved 24.4% accuracy, outperforming the strong baseline EarthDial (18.6%) by 5.8%.
  • When fine-tuned on EarthDial, it achieved 33.3% accuracy, a 33.2% relative improvement over the previous state-of-the-art.
  • Zero-shot performance on NWPU VHR-10 and Swimming Pool datasets also showed consistent superiority over baselines like InternVL and GeoChat.

• Grounding Description:

  • In zero-shot settings, SATGround outperformed baselines on text quality metrics (ROUGE-1, ROUGE-L) and localization accuracy (Acc@0.5) across four diverse datasets (HIT-UAV, NWPU, Swimming Pool, UCAS-AOD).
  • Notably, on the UCAS-AOD dataset, it achieved a 33.2% ROUGE-1 score, a +12.0% absolute gain over EarthDial.

• Visual Question Answering (VQA):

  • The method maintained competitive performance on HRBEN and LRBEN VQA benchmarks, proving that adding explicit geometric modeling does not degrade the model's general reasoning capabilities.

• Ablation Studies:

  • Token Design: Using separate <bb> and <loc> tokens was proven superior to merging them into a single token.
  • Matching: Grouped Hungarian matching significantly outperformed global matching, reducing semantic errors.
  • Orientation: Using OBBs provided a +5.0% overall improvement over axis-aligned boxes, with a massive +17.9% gain for large objects.

5. Significance

SATGround addresses a fundamental limitation in applying Large Language Models to remote sensing: the inability to reason about spatial geometry when coordinates are treated as text. By introducing a structured, spatially-aware grounding module, the paper demonstrates that:

1. Precision: Explicit geometric modeling is essential for accurate object localization in satellite imagery.
2. Robustness: Separating text generation from coordinate regression improves robustness to perturbations and out-of-distribution scenarios.
3. Scalability: The approach is lightweight (only ~16.8M trainable parameters via LoRA) and can be adapted to various remote sensing tasks, paving the way for more reliable, instruction-based satellite data analysis tools.