Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative Evaluation

Technical Summary: Do Open-Vocabulary Detectors Transfer to Aerial Imagery?

Problem Statement
Open-Vocabulary Object Detection (OVD) has demonstrated success in recognizing novel categories in natural, ground-level imagery by leveraging vision-language models (VLMs). However, the transferability of these models to the aerial domain remains unexplored. Aerial imagery presents distinct challenges, including extreme scale variation, strict nadir (top-down) viewpoints, and dense object clustering. Furthermore, semantic categories in aerial datasets often rely on fine-grained attributes or domain-specific terminology that may not align with the natural language priors encoded in VLMs trained on ground-level data. This paper addresses the critical question of whether general-purpose, zero-shot OVD models can serve as a viable baseline for Unmanned Aerial Vehicle (UAV) applications without specialized fine-tuning.

Methodology
The authors present the first systematic benchmark evaluating five state-of-the-art OVD models on the LAE-80C dataset, a rigorous stress test aggregating validation splits from four established aerial datasets (DOTA-v2.0, DIOR, FAIR1M, and xView). The dataset comprises 3,592 images and 86,558 annotated instances across 80 categories, featuring significant taxonomic complexities such as hierarchical overlaps (e.g., "vehicle" vs. "passenger car") and attribute-dependent classes (e.g., "working" vs. "unworking" chimney).

The evaluation involves five models pretrained exclusively on natural, ground-level imagery:

1. Grounding DINO (Transformer-based, Swin-L backbone)
2. OWLv2 (Dense image-text matching, ViT-L/14 backbone)
3. YOLO-World (Real-time, RepVL-PAN architecture)
4. YOLOE (Real-time, YOLOv11-based)
5. LLMDet (Integrates a frozen LLM for semantic reasoning)

The experimental protocol employs a strict zero-shot setting (no fine-tuning on aerial data) and isolates performance factors through three inference modes:

• Global Inference: Prompting with all 80 categories simultaneously to test semantic confusion.
• Oracle Inference: Prompting only with ground-truth categories present in the image to isolate visual localization capabilities.
• Single-Category Oracle: Iterative inference per class to test attention degradation.

Additionally, the study evaluates prompt engineering strategies, including domain-specific prefixing ("Aerial view of {category}") and synonym expansion, using Intersection over Area (IoA) as the primary metric to account for scale variance in satellite imagery.

Key Results
The study reveals a severe domain transfer failure across all evaluated models:

• Overall Performance: The best-performing model, OWLv2, achieves only a 27.6% F1-score with a 69% false positive rate. Other models perform significantly worse, with YOLO variants achieving F1-scores below 4%.
• Semantic Confusion as the Primary Bottleneck: Reducing the vocabulary size from 80 classes (Global mode) to an average of 3.2 classes (Oracle mode) yields a 15× improvement in F1-score for Grounding DINO (from 0.5% to 7.4%). This indicates that the models can extract relevant visual features but fail to map them to the correct category due to overlapping taxonomies and semantic confusion, rather than a total failure of visual localization.
• Failure of Prompt Engineering: Strategies such as adding "Aerial view of" prefixes or expanding synonyms failed to improve performance. In some cases, they degraded results by over-constraining the model or introducing semantic drift, confirming that text-based adaptation cannot compensate for the visual domain gap (side-profile vs. nadir features).
• Dataset Brittleness: Performance varies drastically across sub-datasets. Models achieve moderate results on object-centric datasets like DIOR (F1 ~0.53 for LLMDet) but collapse when faced with the high density of FAIR1M (F1 ~0.12) or the extreme scale variations of xView.

Significance and Claims
The paper establishes that the "open vocabulary" promise of current VLM-based detectors does not transfer effectively from natural to aerial domains without fundamental architectural changes or domain-adaptive pretraining. The authors claim that:

1. The domain gap is architectural, not lexical: The mismatch lies in how visual encoders extract features (trained on side-profiles) rather than in the linguistic descriptions of objects.
2. Semantic confusion dominates: The primary barrier to performance is the model's inability to disambiguate fine-grained, hierarchical categories in a zero-shot setting, not the inability to see the objects.
3. Current models are inadequate for autonomous deployment: The observed precision-recall trade-offs are extreme; conservative models miss most objects, while aggressive models generate too many false alarms for autonomous use.

The work provides a baseline for future research, suggesting that future aerial OVD systems must focus on intelligent vocabulary filtering based on scene context, hybrid detection pipelines, or test-time adaptation mechanisms rather than relying on off-the-shelf natural-image models.