OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks

This paper introduces OmniEarth, a comprehensive benchmark comprising 9,275 images and 44,210 verified instructions that evaluates Vision-Language Models across 28 geospatial tasks with a focus on perception, reasoning, and robustness, revealing significant performance gaps in current models for remote sensing applications.

The Gist

Imagine you have a super-smart robot assistant that is amazing at looking at photos of cats, dogs, and sunsets, and chatting about them. You might think, "Great! Let's ask this robot to look at satellite photos of cities, forests, and oceans to help us plan our future."

But here's the problem: Satellite photos are weird. They look nothing like the photos we take with our phones. They are taken from high up, they use different "eyes" (like radar instead of light), and they show things we can't easily see, like how a city has grown over ten years.

The paper "OmniEarth" is like a giant, rigorous final exam designed specifically to test if these AI robots are actually ready for the real world of Earth observation, or if they are just guessing based on what they read in their training manuals.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Textbook vs. Reality" Gap

Currently, there are many AI models that can talk about images. But when researchers tried to test them on satellite data, they realized the existing tests were too easy or too narrow.

• The Analogy: Imagine teaching a student to drive using only a video game simulator with perfect weather and flat roads. Then, you hand them the keys to a real truck in a snowstorm. They might crash because the simulator didn't teach them about ice or heavy trucks.
• The Issue: Old benchmarks were like that simulator. They didn't test if the AI could handle bad weather, different camera types (like radar), or complex questions like "How much damage did this hurricane cause?"

2. The Solution: OmniEarth (The "Grand Master Exam")

The authors created OmniEarth, a massive new benchmark (a standardized test) to see how good these AI models really are. They didn't just throw a few pictures at the AI; they built a comprehensive curriculum.

They organized the test into three main subjects:

A. Perception (The "Eyes")

Can the AI actually see what's in the picture?

• The Analogy: This is like asking a student to identify a car in a photo.
• The Twist: In OmniEarth, they ask harder questions. "Is that a specific model of airplane?" "How many ships are in that harbor?" "Can you draw a line around the flooded area?"
• The Result: The AI is good at saying "That's a city," but terrible at counting specific cars or drawing precise lines around a building. It's like a student who knows what a "tree" is but can't tell you how many leaves are on it.

B. Reasoning (The "Brain")

Can the AI connect the dots?

• The Analogy: This is like asking, "If I see a new road being built here, and a school being built there, what will the neighborhood look like in 5 years?"
• The Twist: They test Time Travel (looking at photos from 10 years ago vs. today to see changes) and Geography (looking at a photo and guessing which city it is based on the layout).
• The Result: The AI struggles here. It can describe a picture, but it's bad at predicting the future or understanding cause-and-effect (e.g., "This river flooded because the dam broke").

C. Robustness (The "Stress Test")

Can the AI handle a messy world?

• The Analogy: Imagine taking a test, but someone puts a smudge on the paper, blurs the words, or changes the lighting. Can you still answer correctly?
• The Twist: They showed the AI photos that were blurry, covered in clouds, or taken by radar (which looks like static noise) instead of a normal camera.
• The Result: When the images got messy, the AI's performance tanked. It relied too much on "guessing" based on the text question rather than actually looking at the messy image.

3. The Big Surprise: The "Blind Test"

This is the most interesting part of the paper. The researchers did a "Blind Test."

• The Setup: They asked the AI a question about a satellite image, but they didn't show the image. They only gave the text question.
• The Metaphor: It's like asking a student, "What is the capital of France?" without showing them a map. If they get it right, it's because they memorized the answer, not because they looked at the map.
• The Shock: Many AI models got the answers right even without the image! This means they weren't actually "seeing" the Earth; they were just guessing based on the wording of the question. They were "cheating" by relying on language patterns instead of visual evidence.

4. The Data: A Global Library

To make this test fair, they didn't just use old, public photos.

• They gathered 9,275 images from all over the world (7 continents!).
• They used Jilin-1, a private satellite constellation, to get fresh, high-quality images that the AI hadn't seen before during its training.
• They created 44,210 questions that were checked by humans to ensure they were accurate.

5. The Conclusion: We Have a Long Way to Go

The paper concludes that while AI is getting smarter, it is not yet ready to be a reliable partner for Earth scientists.

• Current Status: The AI is like a brilliant student who has memorized the textbook but has never stepped outside. It knows the theory but fails when the real world gets messy, blurry, or complex.
• The Future: We need to build AI that actually looks at the data, understands the physics of the Earth, and doesn't just guess based on the question.

In short: OmniEarth is the "reality check" that tells us, "Hey, your AI is great at chatting, but it still needs to learn how to really see our planet."

Technical Summary

Here is a detailed technical summary of the paper "OmniEarth: A Benchmark for Evaluating Vision–Language Models in Geospatial Tasks."

1. Problem Statement

While Vision-Language Models (VLMs) have achieved significant success in general-domain tasks, their application to Remote Sensing (RS) and Earth observation faces unique challenges that existing benchmarks fail to address comprehensively.

• Domain Gap: Satellite imagery differs significantly from natural images in object scale, spatial resolution, illumination, and sensing modalities (e.g., SAR, multispectral).
• Lack of Systematic Evaluation: Existing benchmarks (e.g., VRSBench, GEOBench-VLM, CHOICE) suffer from:
  • Limited Granularity: They often focus on coarse categories or lack pixel-level analysis and implicit reasoning tasks.
  • Linguistic Bias: Many benchmarks use single question-answer formats where models can guess correct answers based on language priors rather than visual evidence.
  • Data Leakage: Many benchmarks reuse public datasets that may have been part of foundation models' pre-training, compromising zero-shot evaluation reliability.
  • Modality Gaps: Limited support for non-optical modalities (like SAR) and cross-modal consistency.

2. Methodology: The OmniEarth Benchmark

OmniEarth is a systematic benchmark designed to evaluate Remote Sensing Vision-Language Models (RSVLMs) under realistic Earth observation scenarios.

A. Data Construction

• Scale: Contains 9,275 high-quality images and 44,210 manually verified instructions.
• Sources:
  • Proprietary Data: Includes first-time public release of Jilin-1 (JL-1) satellite imagery.
  • Multi-Modal: Combines Optical (RGB), Synthetic Aperture Radar (SAR) from Capella Space, Multispectral, and Nighttime light data.
  • Geographic Coverage: Spans 7 continents and over 400 cities, ensuring diverse land-cover and urban/rural contexts.
  • Temporal Dynamics: Includes bi-temporal and multi-temporal data for change detection and trend analysis.
• Quality Control:
  • Blind Test Protocol: Uses a "text-only" vs. "image+text" comparison to detect reliance on linguistic priors.
  • Semantic Consistency: Each image is queried with five semantically equivalent questions to ensure logical consistency and reduce prompt bias.
  • Human Verification: A three-group, three-round cross-validation protocol ensures annotation accuracy.

B. Task Taxonomy

OmniEarth organizes 28 fine-grained tasks into three core capability dimensions:

1. Perception (12 Tasks):
   • Image-level: Scene Classification, Land-cover Classification, Image Modality Recognition, Image Captioning.
   • Instance-level: Visual Grounding, Referring Expression Comprehension, Object Counting, Fine-grained Category Classification, Attribute Recognition.
   • Pixel-level: Referring Expression Segmentation, Generalized Referring Expression Segmentation, Change Mask Segmentation.
2. Reasoning (12 Tasks):
   • Spatial: Spatial Relationship Reasoning, Geometric Measurement, Functional Region Localization.
   • Temporal: Change Description, Damage Assessment, Long-term Trend Reasoning, Seasonal Temporal Reasoning.
   • Geographic Application: Geo-localization, Disaster Cause Inference, Geo-Entity Understanding, City Recognition, Planning Suggestions.
3. Robustness (4 Tasks):
   • Environmental Resilience: Image Condition Assessment (clouds, haze), Degraded-condition VQA (noise, blur, occlusion).
   • Semantic Reliability: Hallucination Detection, Semantic Consistency (matching SAR to Optical).

C. Evaluation Metrics

• Classification/Counting: Top-1 Accuracy.
• Localization (Grounding/REC): Accuracy at IoU thresholds (0.25, 0.5, 0.75) and mIoU.
• Segmentation: Mean Intersection-over-Union (mIoU).
• Generation (Captioning/Reasoning): CIDEr, BLEU-4, ROUGE-L, METEOR, BERTScore.
• Visual Gain ($\Delta$): The performance difference between full multimodal input and text-only (blind) input to measure true visual grounding.

3. Key Contributions

1. OmniEarth Benchmark: The first comprehensive benchmark covering 28 fine-grained tasks across perception, reasoning, and robustness, specifically tailored for Earth observation.
2. Bias-Aware Protocols: Introduction of blind testing and quintuple semantic consistency to rigorously separate visual understanding from language priors.
3. Diverse Data Sources: Integration of proprietary JL-1 data and high-fidelity SAR-RGB pairs, ensuring low data overlap with pre-training sets and high modality diversity.
4. Comprehensive Evaluation: Evaluation of 19 state-of-the-art VLMs, including commercial models (GPT-4o, Gemini), open-source general models (Qwen, InternVL), and RS-specific models (GeoChat, VHM).

4. Experimental Results

The authors evaluated 19 models, revealing significant gaps between general VLMs and the demands of geospatial tasks:

• Perception:
  • General VLMs perform well on image-level tasks (Scene Classification ~65-90%) but struggle with fine-grained perception.
  • Pixel-level segmentation is a major bottleneck, with most models scoring below 15% mIoU, and many RS-specific models near zero.
• Reasoning:
  • Spatial Reasoning: General models handle qualitative spatial relationships well (>85%) but fail at quantitative tasks (Geometric Measurement <40%).
  • Temporal Reasoning: Performance is moderate for descriptive changes but poor for long-term trend inference and seasonal recognition.
  • Geographic Application: Models struggle with planning suggestions and disaster cause inference, indicating a lack of deep domain knowledge integration.
• Robustness:
  • General VLMs show better resilience to image degradation (blur, noise) than RS-specific models.
  • Cross-modal (RGB-SAR) matching remains a significant challenge for most models.
• Visual Grounding (Blind Test):
  • Critical Finding: Many RSVLMs show minimal performance gaps between blind (text-only) and full-input settings. This indicates that models often rely on linguistic shortcuts or option elimination rather than analyzing visual evidence.
  • For example, in temporal reasoning tasks, models often generate plausible answers based solely on text templates without processing the actual image changes.

5. Significance

• Diagnostic Tool: OmniEarth provides a rigorous framework to identify specific failure modes in current VLMs, such as poor localization, weak temporal reasoning, and over-reliance on language priors.
• Guidance for Future Research: The results suggest that simply fine-tuning general VLMs on RS data is insufficient. Future models require:
  • Architectural designs specifically for multi-scale and multi-modal RS data.
  • Training strategies that enforce visual grounding to prevent hallucination and linguistic bias.
  • Better integration of domain-specific geographic knowledge.
• Community Resource: By releasing the dataset and benchmark publicly, OmniEarth establishes a new standard for evaluating and advancing remote sensing AI, moving beyond simple classification to complex, reasoning-heavy geospatial tasks.

In conclusion, OmniEarth highlights that while VLMs have made progress, they are not yet "ready" for complex, real-world Earth observation tasks that require precise localization, robust reasoning under degradation, and genuine multimodal analysis.