💬 NLP

MarsRetrieval: Benchmarking Vision-Language Models for Planetary-Scale Geospatial Retrieval on Mars

This paper introduces MarsRetrieval, a comprehensive benchmark and unified protocol designed to evaluate and improve vision-language models for text-guided geospatial discovery on Mars, highlighting the critical need for domain-specific fine-tuning to overcome the limitations of current foundation models in capturing Martian geomorphic distinctions.

Original authors: Shuoyuan Wang, Yiran Wang, Hongxin Wei

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Shuoyuan Wang, Yiran Wang, Hongxin Wei

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery on Mars. You have a massive library containing millions of photos of the Red Planet's surface, taken by orbiters and rovers. But here's the catch: the library has no index, no labels, and no way to search by description.

If you wanted to find a picture of a "crater that looks like a pancake" or a "dune field shaped by ancient winds," you would have to flip through every single photo one by one. That would take a lifetime.

This is the problem MarsRetrieval is trying to solve. It's a new "test" (or benchmark) designed to see if Artificial Intelligence (AI) can act like a super-smart librarian who can instantly find the right Martian photo just by reading a sentence you type.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Black Box" Library

Right now, most AI models used for Mars are like students who have memorized a specific list of flashcards. If you show them a picture of a "rock," they can tell you it's a rock. But if you ask them, "Show me a picture of a landslide that happened near the equator," they get confused. They don't understand the story behind the image, only the pixels.

Scientists need AI that understands language (what they are asking) and vision (what the planet looks like) at the same time.

2. The Solution: The "MarsRetrieval" Test

The authors created a giant challenge called MarsRetrieval. Think of it as a "Driver's License Exam" for AI models. To pass, the AI has to prove it can do three specific jobs:

Job A: The Translator (Paired Image-Text Retrieval)
- The Analogy: Imagine you have a photo of a Martian canyon and a sentence that says, "A deep valley carved by water." The AI needs to match the sentence to the photo perfectly, and vice versa. It's like a game of "Memory" where the cards are pictures and words instead of matching pairs.
- The Challenge: The photos range from huge views of the whole planet down to tiny close-ups of a single rock. The AI has to understand the scale.
Job B: The Concept Detective (Landform Retrieval)
- The Analogy: Imagine you ask the AI, "Show me all the 'sand dunes' on Mars." The AI shouldn't just show you one dune; it needs to find every dune, even if they look slightly different (some are tall, some are flat, some are in the shade).
- The Challenge: Mars has 48 different types of landforms (like "volcanoes," "glaciers," or "cracks"). The AI needs to recognize the idea of a landform, not just a specific picture it memorized.
Job C: The Global GPS (Global Geo-Localization)
- The Analogy: This is the hardest level. Imagine you have a map of the entire Earth, but it's covered in static noise. You ask the AI, "Where are all the 'ice glaciers'?" The AI has to point to the exact coordinates on the map where glaciers exist, ignoring the millions of places where there are no glaciers.
- The Challenge: It's like finding a needle in a haystack, but the haystack is the size of a planet, and the needle is a specific type of rock formation.

3. The Results: The "School Report Card"

The researchers tested many famous AI models (like the ones that power chatbots or image generators) on this test. Here is what they found:

The "General" Students Struggled: The smartest AI models we have today (trained on general internet data) did okay at Job A, but they failed miserably at Jobs B and C. They couldn't tell the difference between a "volcano" and a "crater" when the lighting was different, or they couldn't find the right spots on the map.
- Why? They are like a tourist who knows what a "beach" looks like in Hawaii but doesn't know what a "desert dune" looks like on Mars. They lack specialized knowledge.
The "Specialist" Student Aced It: The researchers created a model called MarScope. This model was specifically trained on Mars data. It was like taking a general student and giving them a summer internship with a geologist.
- The Result: MarScope crushed the test. It could find the right landforms and map them accurately.

4. The Big Lesson

The paper teaches us two main things:

You can't just use a "general" AI for science. Just because an AI is good at writing poems or recognizing cats doesn't mean it can help us explore Mars. Science requires specialized training.
Language is the key. The best way to find things on Mars isn't just looking at pictures; it's using scientific words to describe what we are looking for. The AI needs to learn the "language of geology."

The Bottom Line

MarsRetrieval is a new tool that helps scientists build better AI explorers. It ensures that when we send AI to help us explore Mars in the future, it won't just be a camera that takes pictures—it will be a smart assistant that can listen to a scientist's question ("Find me the ancient riverbeds") and instantly point to the exact location on the planet.

It bridges the gap between human curiosity (asking questions) and machine vision (seeing the answers).

1. Problem Statement

While deep learning has advanced planetary science (e.g., terrain classification, object detection), existing benchmarks like Mars-Bench are limited to closed-set supervised visual tasks. They fail to address the needs of modern planetary research, where scientists frequently use language-guided discovery to align textual concepts (e.g., "alluvial fans," "glacier-like forms") with vast amounts of unstructured geospatial imagery.

Current Vision-Language Models (VLMs) show promise in fields like medicine and chemistry but lack standardized evaluation for planetary-scale geospatial retrieval. Specifically, there is no protocol to test if VLMs can:

Align fine-grained scientific descriptions with multi-scale Martian imagery.
Generalize geomorphic concepts to diverse visual instances (long-tailed distribution).
Localize scientific concepts across a global mosaic (1.4 million tiles) amidst massive background noise.

2. Methodology: The MarsRetrieval Benchmark

The authors introduce MarsRetrieval, a comprehensive retrieval-centric benchmark designed to evaluate VLMs on three complementary tasks covering different spatial scales and scientific challenges.

A. Evaluation Protocol

The benchmark adopts a unified retrieval-centric protocol. Instead of task-specific fine-tuning for classification, models are evaluated on their ability to rank matched items above mismatched candidates in a shared embedding space using cosine similarity.

B. Three Core Tasks

Paired Image–Text Retrieval (Task 1):
- Goal: Bidirectional retrieval (Text $\to$ Image, Image $\to$ Text) to test foundational vision-language alignment.
- Data: 2,287 curated pairs spanning macro (orbital), meso (landform), and micro (rover) scales.
- Construction: A multi-stage pipeline involving web-scale data collection, automatic filtering (CLIP score, LLM relevance, MLLM consistency), diversity sampling, and expert-validated caption refinement to remove noise and ensure scientific accuracy.
Landform Retrieval (Task 2):
- Goal: Text-to-image retrieval testing concept-to-instance generalization.
- Data: 1,185 image patches organized into 48 geomorphic categories (long-tailed distribution) across 7 major genetic classes (e.g., Aeolian, Volcanic, Fluvial).
- Challenge: High intra-class morphological diversity (different erosion stages, lighting, scales) and inter-class similarity.
Global Geo-Localization (Task 3):
- Goal: Retrieving all relevant geographic locations for a given concept from a global mosaic of 1.4 million CTX tiles.
- Data: Ground truth derived from published scientific catalogs for 5 landform types (Alluvial fans, Glacier-like forms, Landslides, Pitted cones, Yardangs).
- Challenge: Extreme class imbalance (sparse positives in massive background noise) and the need for planetary-scale distribution estimation rather than just top- $K$ ranking.

C. Baselines and Models

The authors benchmarked a wide spectrum of foundation models:

Dual-encoder VLMs: CLIP, SigLIP, PE-Core, BGE-VL.
MLLM-based Embedding Models: E5-V, GME, Qwen3-VL-Embedding, VLM2Vec.
Vision-Only Models: DINOv3, AIMV2.
Domain-Specific Model: MarScope (a CLIP variant fine-tuned specifically for planetary science).

3. Key Results

A. General Foundation Models Struggle

Performance Gap: Even the strongest general-purpose foundation models (e.g., PE-Core, Qwen3-VL) perform moderately on Task 1 and poorly on Tasks 2 and 3.
Domain Sensitivity: In Landform Retrieval (Task 2), the best general model (PE-Core) achieved an mAP of 20.93%, whereas the domain-specific MarScope achieved 71.89%. This indicates general models lack the expertise to distinguish subtle Martian geomorphic features.

B. Importance of Domain-Specific Fine-Tuning

MarScope Dominance: The domain-specialized model consistently outperformed all general models across all three tasks, particularly in Global Geo-Localization where it achieved high AUPRC scores (e.g., 71.31% for Pitted Cones).
Conclusion: Generic multimodal representations are insufficient for expert-level planetary discovery; domain-specific post-training is critical for generalizable geospatial discovery.

C. Impact of Data Quality

Caption Refinement: In Task 1, using refined captions (scientifically grounded, noise-free) significantly improved retrieval metrics (MRR) for all models. Raw web-scraped captions led to substantial performance degradation, highlighting that high-quality data curation is a prerequisite for planetary AI.

D. Text vs. Image Queries

Text Superiority: In Global Geo-Localization (Task 3), text-based queries generally outperformed image-based queries.
Reasoning: Martian landforms exhibit extreme visual diversity (morphology, lighting, scale). A small set of image exemplars cannot capture the full distribution of a landform type, whereas scientific text concepts provide a more robust semantic anchor for retrieval.

4. Key Contributions

First Retrieval-Centric Benchmark: Introduced MarsRetrieval, the first benchmark specifically designed to evaluate VLMs for language-driven geospatial discovery on Mars.
Multi-Scale Evaluation: Established a holistic framework covering fine-grained alignment, concept generalization, and planetary-scale localization.
Empirical Evidence for Domain Specialization: Demonstrated that while general VLMs provide a baseline, domain-specific fine-tuning is non-negotiable for reliable planetary science applications.
Data Curation Protocol: Provided a rigorous pipeline for constructing high-quality, expert-validated image-text pairs for planetary science, addressing the "noise" problem in web-scale data.

5. Significance and Future Impact

Bridging AI and Science: MarsRetrieval bridges the gap between advanced multimodal AI capabilities and the practical workflows of planetary scientists, enabling automated hypothesis verification and global cataloging.
Standardization: It provides a standardized, reproducible testbed for comparing future foundation models, moving beyond simple classification metrics to retrieval-based utility.
Deep Space Exploration: The evaluation paradigm is adaptable to other autonomous planetary explorations (e.g., Moon, Europa), facilitating the development of foundation models for deep-space discovery where human-in-the-loop analysis is limited.

In summary, the paper argues that for AI to truly assist in planetary science, models must move beyond generic visual understanding to domain-specific, language-grounded geospatial reasoning, a capability that current general foundation models lack without specialized training.