GroundSet: A Cadastral-Grounded Dataset for Spatial Understanding with Vector Data

1. Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated remarkable zero-shot capabilities in general domains but suffer from critical deficiencies in fine-grained spatial understanding, particularly in Remote Sensing (RS). Current RS-specialized models struggle with precise object detection, segmentation, and spatial reasoning (e.g., sub-meter precision), which are prerequisites for high-stakes applications like urban planning and disaster management.

The primary bottleneck is the scarcity of high-quality, dense spatial annotations. Existing datasets rely on:

• Legacy datasets (e.g., DOTA, DIOR) with limited semantic scope (generic classes) and closed-set detection formats.
• Crowd-sourced data (e.g., OpenStreetMap) which often lacks geometric consistency and taxonomic rigor.
• Synthetic Question-Answer generation from older data, which fails to provide novel domain knowledge or scalability.

Consequently, current models exhibit poor zero-shot generalization and often underperform compared to their base architectures when applied to new RS tasks.

2. Methodology

A. Data Curation: The Cadastral Approach

The authors introduce GroundSet, a large-scale dataset derived from verifiable cadastral vector data maintained by national agencies (specifically France's IGN). This approach prioritizes annotation fidelity over immediate global diversity.

• Source: High-resolution (20cm) aerial orthophotos aligned with legally verified vector maps containing stationary objects (buildings, roads, vegetation, water bodies).
• Scale: 510,483 high-resolution image patches (672×672 pixels) covering 85,864 km² across 20 administrative departments.
• Annotation Density: 3.8 million annotated objects with a median of 8 objects per image (significantly higher than benchmarks like VRSBench).
• Semantic Granularity: 135 highly specific semantic categories (e.g., distinguishing Catholic vs. Orthodox churches, specific crop types, heritage sites), far exceeding the few dozen generic classes in existing datasets.

B. Processing Pipeline

1. Preprocessing:
   • Filtering: Removal of non-visual entities (tunnels, underground structures) to prevent hallucinations.
   • Resampling: Mitigation of semantic redundancy in rural areas by capping unique scene compositions (e.g., {Forest, Path}) at 64 samples, reducing the dataset size by 85% while balancing distribution.
   • Geometric Merging: Merging contiguous polygons of the same class to align legal partitions with visual footprints.
   • Coordinate Normalization: Converting geographical coordinates to relative pixel coordinates [0, 1].
2. Annotation Generation:
   • Augmentation of vector data (polygons, lines, points) with Horizontal Bounding Boxes (HBB), Oriented Bounding Boxes (OBB), and Segmentation Masks.
   • Visual-Semantic Discrepancy Handling: An automated profiling pipeline (using Gemini-2.5 Flash) flags occluded or invisible instances rather than discarding them, allowing for "clean" vs. "hard" training splits.
3. Instruction Generation:
   • Automated generation of instruction-tuning data using rigid geometric rules and templates to ensure captions are grounded in facts, preventing VLMs from hallucinating attributes.
   • Tasks: The dataset supports 7 spatial reasoning tasks: Captioning, Localized Classification, Object Detection, Multi-Class Detection, Referring Expression Comprehension (REC), Segmentation, and Visual Question Answering (VQA).

C. Experimental Setup

• Baseline Model: A standard LLaVA-1.6 (7B) architecture was fine-tuned using LoRA (Low-Rank Adaptation) for a single epoch.
• Training Strategy: The model was trained on 1.8 million instructions. Crucially, no complex architectural modifications or auxiliary spatial modules were added; the focus was strictly on the impact of the data.
• Ablation Studies: Tested the impact of auxiliary geometric tasks (text-only polygon reasoning, exhaustive segmentation queries) and filtering occluded objects.

3. Key Contributions

1. The Data (GroundSet): A novel, large-scale EO dataset with verified cadastral origins, offering unprecedented semantic richness (135 classes) and density (3.8M objects). It provides a rigorous, legally verified ground truth that solves the scalability-quality trade-off.
2. The Benchmark: A comprehensive instruction-tuning suite comprising 7 spatial understanding tasks derived directly from vector data, designed to test the limits of modern VLMs in fine-grained spatial grounding.
3. The Baseline: A demonstration that standard VLM architectures (LLaVA), when trained on high-fidelity vector data, can master complex spatial grounding tasks without architectural changes, outperforming specialized RS models and commercial giants.

4. Results

The authors evaluated state-of-the-art models (Open-source, RS-specialized, Spatial-reasoning specialists, and Commercial models like Gemini-2.5) in a zero-shot setting against the GroundSet test set, and compared them to their own fine-tuned baseline.

• Zero-Shot Performance: Existing RS-specialized models (e.g., GeoChat, SkySenseGPT) and commercial models (Gemini) exhibited severe degradation in zero-shot settings, often underperforming their base architectures. They struggled with the novel semantic categories and fine-grained geometry.
• Fine-Tuned Baseline Performance: The LLaVA-1.6 model fine-tuned on GroundSet achieved state-of-the-art results across all tasks:
  • Classification: 94.18% accuracy (vs. ~49% for Gemini, ~29% for LLaVA-1.6 zero-shot).
  • Detection (F1@0.5): 49.47% (vs. ~3-18% for others).
  • Segmentation (F1@0.5): 44.65% (vs. <20% for others).
  • Captioning & VQA: Significant improvements over all baselines.
• Cross-Dataset Generalization: When evaluated on the VRSBench benchmark (which focuses on mobile objects like vehicles/planes, absent from GroundSet's stationary training data), the GroundSet-trained model outperformed RS-specialists in core grounding tasks (Detection and REC). This proves that the spatial grounding capabilities learned from cadastral data are transferable and robust to domain shifts, even if the semantic vocabulary is not.

Key Finding: The performance gap is attributed to the data quality (semantic richness and geometric precision) rather than architectural superiority. Standard models can master fine-grained spatial understanding if grounded in high-fidelity vector data.

5. Significance

• Bridging the Knowledge Gap: The paper identifies a critical "knowledge gap" in current web-scale pre-training, where models lack the specific, granular domain knowledge found in cadastral records. GroundSet fills this gap.
• Paradigm Shift: It challenges the notion that complex architectural modifications are necessary for spatial reasoning. Instead, it argues that high-fidelity supervision is the primary driver of performance.
• Operational Viability: By providing legally verified, sub-meter precise data, GroundSet enables the deployment of MLLMs in high-stakes, real-world applications (urban planning, disaster management) where previous models were too inaccurate.
• Future Research: The release of native vector primitives (rather than just raster masks) encourages the community to explore non-autoregressive architectures (e.g., GNNs, Diffusion LLMs) that can better leverage geometric structures.

In conclusion, GroundSet demonstrates that the path to robust spatial AI in Remote Sensing lies not in scaling model parameters alone, but in integrating verified, high-density vector ground truth into the training pipeline.