LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration

1. Problem Statement

Despite significant advances in general-purpose Multimodal Vision-Language Models (VLMs) like LLaVA, their application to planetary science remains limited. The primary bottlenecks are:

• Data Scarcity: There is a lack of large-scale, high-quality datasets pairing real planetary imagery with detailed scientific descriptions. Existing lunar datasets are often unimodal, small-scale, or rely on synthetic simulations.
• Domain Complexity: Planetary remote sensing differs fundamentally from natural image understanding. Interpreting lunar terrain requires reasoning across multiple physical modalities (e.g., optical imagery, gravity anomalies, and terrain slope) to infer subsurface structures and geological processes, rather than just identifying semantic objects.
• Reasoning Gap: Current models struggle to perform high-level geological interpretation or answer complex scientific queries based on real lunar observations.

2. Methodology

The authors propose LLaVA-LE, a specialized VLM trained on a newly curated dataset, LUCID, using a two-stage training curriculum.

A. Dataset: LUCID (LUnar Caption Image Dataset)

LUCID is the first large-scale, real multimodal lunar dataset designed for VLM training.

• Sources: Derived from NASA missions: Lunar Reconnaissance Orbiter Camera (LROC) for optical imagery, GRAIL for gravity anomaly maps, and LOLA for terrain slope maps.
• Composition:
  • 96,000 high-resolution panchromatic image patches (224x224 px) paired with detailed scientific captions.
  • 81,000 Question-Answer (QA) pairs derived from ~20,000 of the images.
• Generation Pipeline:
  1. Concept Alignment Data: GPT-5.1 generates detailed captions for image patches. Crucially, the generation is guided by co-registered gravity and slope data to ensure the captions reflect inferred subsurface characteristics (e.g., crustal density, mechanical coherence) alongside surface morphology.
  2. Instruction-Tuning Data: The captions are converted into multi-turn dialogues (QA pairs) covering categories like formation/evolution, geomorphology, spatial relations, and surface properties.

B. Model Architecture & Training

The model is based on LLaVA-v1.5-13B and utilizes a two-stage training strategy:

1. Stage 1: Concept Alignment
   • Goal: Align lunar visual features with domain-specific geological language.
   • Process: The model is trained on 76k image-caption pairs. The vision encoder (CLIP ViT-L) and the base LLM are frozen; only the projection layer and lightweight LoRA (Low-Rank Adaptation) modules are updated.
   • Objective: The model learns to associate visual patterns (texture, crater rims, slopes) with scientific terminology and geological context.
2. Stage 2: Instruction Tuning
   • Goal: Enable conversational interaction and complex reasoning.
   • Process: The model is fine-tuned on the 81k QA pairs.
   • Objective: The model learns to answer specific scientific questions, perform multi-step reasoning, and engage in dialogue based on the visual input, without access to the ground-truth caption during inference.

3. Key Contributions

• LLaVA-LE Model: A specialized VLM capable of multimodal geological reasoning over real lunar remote sensing data, bridging the gap between general AI and planetary science.
• LUCID Dataset: The release of the first large-scale (96k images, 81k QA pairs), real-world multimodal lunar dataset. It uniquely integrates optical imagery with geophysical data (gravity/slope) to ground descriptions in physical reality.
• Open Source: Public release of the dataset, codebase, and model checkpoints to accelerate AI-driven planetary exploration research.
• Evaluation Benchmark: A new benchmark spanning three complexity levels (Detailed, Conversation, Reasoning) specifically designed to test lunar terrain analysis capabilities.

4. Results

The model was evaluated against Base LLaVA (general domain) and LLaVA-LE Stage 1 (concept alignment only) using GPT-4 and Gemini as automated judges.

• Performance Gains:
  • Overall: LLaVA-LE (Stage 2) achieved a 3.3× performance improvement over Base LLaVA and a 2.1× improvement over Stage 1.
  • Reasoning: The model achieved a reasoning score of 1.070, which exceeded the reference score of the judges themselves. This indicates the model can generate geological interpretations more accurate or comprehensive than the baseline reference provided to the judges.
  • Category Breakdown:
    • Detailed: ~3.4× improvement over Base.
    • Conversation: ~2.7× improvement.
    • Reasoning: ~3.6× improvement.
• Qualitative Analysis:
  • Base LLaVA often produced generic answers or hallucinated features not present in the scene.
  • LLaVA-LE successfully inferred subsurface properties (e.g., "mechanically coherent substrate") based on surface textures and crater distributions, aligning with independent gravity and slope data from the GRAIL and LOLA missions.

5. Significance

This work demonstrates that domain-specific multimodal data is critical for advancing VLMs in scientific fields. By curating real planetary data and integrating complementary geophysical modalities, the authors proved that VLMs can move beyond simple image captioning to perform high-level scientific reasoning.

• Scientific Impact: Provides a tool for lunar exploration that can assist scientists in interpreting remote sensing data, hypothesizing about subsurface structures, and answering complex geological queries.
• Methodological Impact: Establishes a framework for training specialized VLMs in data-scarce scientific domains by leveraging structured prompting and multi-modal alignment.
• Future Outlook: The open release of LUCID and LLaVA-LE sets a new standard for planetary AI, encouraging further research into multimodal reasoning for other planetary bodies and remote sensing applications.