🤖 AI

Moonstone: A Multimodal Foundation Model and Benchmark for Lunar Remote Sensing

This paper introduces Moonstone, the first multimodal foundation model benchmark for lunar remote sensing, which features a comprehensive 28-channel global dataset and a novel modality-grouped masked autoencoder (MG-MAE) that significantly outperforms existing baselines across six downstream tasks.

Original authors: Ayush Prasad, Swarnalee Mazumder

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ayush Prasad, Swarnalee Mazumder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the Moon as a giant, dusty library that has been visited by five different explorers over the last few decades. Each explorer brought a different kind of camera or sensor: some took photos, some measured heat, some listened to radar echoes, and some weighed the Moon's gravity.

The problem? All these books are scattered in different rooms, written in different languages, and no one has ever tried to read them all together to understand the story of the Moon.

This paper introduces Moonstone, a new project that acts like a super-smart librarian and a universal translator for all this lunar data. Here is how it works, broken down into simple concepts:

1. The Big Collection (The Dataset)

The authors gathered 28 different "channels" of data from five different space missions. Think of this like assembling a 28-piece puzzle where each piece is a different type of information about the same spot on the Moon.

The Mix: It includes photos, heat maps, radar scans, gravity readings, and chemical compositions.
The Challenge: Some of these "pieces" are missing for certain areas. For example, the radar data only covers about 16% of the Moon, while the gravity data covers 100%. It's like trying to solve a puzzle where some pieces are missing entirely in certain spots.

2. The Smart Translator (The Model: MG-MAE)

To make sense of this messy library, they built a new AI model called MG-MAE. Instead of treating every piece of data as a separate, unrelated fact, this model groups them by "family."

Grouping: It treats all the heat sensors as one family, all the radar sensors as another, and so on. This helps the AI understand that heat sensors talk to each other, even if they are far apart.
The "Ghost" Trick: Since some data is missing in certain spots (like the radar), the model uses a clever trick. It puts in a "ghost token" (a placeholder) for the missing data and tells the AI, "Ignore this spot for now, but don't crash." This allows the model to learn from the data it does have without getting confused by the gaps.
The Physics Lesson: The model also has a built-in rulebook based on real physics. For example, it knows that the color of light reflected by moon rocks changes smoothly, not in jagged jumps. This helps it fill in missing information in a way that makes physical sense.

3. The Test Drive (The Benchmark)

To prove this new AI is actually smart, the authors created a "driver's license test" with six different challenges:

Geology: Identifying what kind of rock or terrain is in a picture (49 different types).
Age: Guessing how old a surface is (from ancient to young).
Chemistry: Predicting how much iron or titanium is in the soil.
Cross-Modal Magic: Predicting the heat of a spot just by looking at photos and gravity data (without seeing the heat data itself).
Mapping: Drawing lines around dark "seas" (maria) versus light highlands.
Crater Hunting: Finding and outlining large impact craters.

4. The Results: Why It Wins

When they tested this new AI, it crushed the competition:

vs. Random Guessing: It learned much faster and got much better results than an AI starting from scratch.
vs. Earth AI: They tried using AI models trained on Earth (which know about trees and oceans) on the Moon. It failed miserably. The Moon has no trees or oceans, so Earth-trained AI was like trying to use a map of Paris to navigate Tokyo. The Moonstone model, trained specifically on lunar data, was far superior.
vs. Old Methods: It beat the best previous methods for finding craters and guessing chemical makeup by a significant margin.

5. The "Few-Shot" Superpower

One of the coolest things about Moonstone is how well it works when it hasn't seen many examples. In lunar science, we often have very few labeled examples (like having only a handful of photos with the correct answers).

The Analogy: Imagine a student who has read a whole encyclopedia of lunar facts (pretraining). If you give them a test with only 5 examples, they can still pass with flying colors because they already understand the underlying rules. Other models, without that encyclopedia, would fail.

Summary

The paper presents Moonstone as the first-ever "foundation model" for the Moon. It takes fragmented, messy data from five different missions, groups them logically, and teaches an AI to understand the Moon's geology, age, and chemistry. It proves that to understand the Moon, you can't just use Earth-trained AI; you need a model built specifically for the Moon's unique, multi-sensory environment. The authors have released all their data and code so other scientists can use this new "library" to explore the Moon further.

Technical Summary: Moonstone – A Multimodal Foundation Model and Benchmark for Lunar Remote Sensing

1. Problem Statement

Decades of orbital missions (e.g., NASA's LRO, India's Chandrayaan-1, GRAIL) have generated terabytes of multi-modal remote sensing data for the Moon, spanning optical imagery, spectroscopy, thermal emission, radar, gravity, and elemental composition. However, these datasets remain fragmented across incompatible archives (NASA PDS, USGS, ISRO PRADAN) with varying spatial resolutions and formats. Crucially, no standardized benchmark exists to evaluate machine learning models on lunar data. Existing Earth Observation (EO) foundation models cannot be directly applied because their patch embeddings are hardcoded to Earth-specific sensors (e.g., Sentinel-2, C-band SAR) and lack correspondence with lunar modalities like GRAIL gravity or M3 hyperspectral reflectance. Furthermore, lunar data presents unique challenges: highly non-uniform spatial coverage (ranging from 16% to 100% depending on the instrument), the absence of atmosphere/vegetation, and extreme scarcity of labeled data for supervised tasks.

2. Methodology: The Moonstone Framework

2.1 The Dataset

The authors introduce Moonstone, a global pretraining dataset comprising 28 channels from seven instrument families across five missions.

Resolution: 128 pixels per degree (approximately 237 m/pixel).
Organization: Channels are grouped into seven physically meaningful modality groups: Surface (WAC/LOLA), Thermal (Diviner), Spectral (M3), Gravity (GRAIL), Radar (Mini-RF), Hapke (WAC), and Composition (Clementine/LP-GRS).
Coverage: Varies significantly, from 100% for Surface and Gravity to as low as 16–18% for Radar.
Preprocessing: Data is aligned to a common equirectangular grid, z-score normalized, and stored as memory-mapped arrays to allow for unlimited random 256×256 crop sampling during training.

2.2 The Model: MG-MAE (Modality-Grouped Masked Autoencoder)

The paper proposes MG-MAE, a masked autoencoder specifically designed to handle the heterogeneity of lunar data. Key architectural components include:

Grouped Tokenization: Instead of assigning a tokenizer to each of the 28 channels, the model uses a single multi-channel convolutional tokenizer per modality group (7 tokenizers total). This allows the model to learn intra-group correlations (e.g., across the continuous M3 reflectance spectrum) while reducing parameter count.
Shared Encoder with Missing-Modality Masking: A shared ViT-Base encoder processes tokens from all groups. To handle instruments with partial coverage (e.g., Radar covering only 16% of the surface), the model inserts placeholder tokens (zero vectors) for missing groups and applies key-dimension attention masking (setting attention scores to $-\infty$ ). This prevents unavailable modalities from corrupting the attention mechanism, a critical distinction from standard multi-modal models that assume all modalities are present.
Coverage-Adaptive Masking: Recognizing that coverage varies by an order of magnitude, the masking ratio is adjusted per group based on global coverage ( $m_g = 0.75 + \alpha(c_g - \bar{c})$ ). This preserves more tokens from rare modalities (e.g., Radar) while masking more of fully covered groups.
Cross-Modal Decoder: A shared lightweight decoder uses cross-attention to reconstruct masked patches. It leverages information from all available groups to infer missing data in a specific group (e.g., inferring thermal patterns from surface observations).
Loss Function: The objective combines:
1. Proportionally Weighted MSE: Weights ( $w_g$ ) are dynamically adjusted based on the current reconstruction loss of each group, automatically upweighting harder groups (like Radar) and downweighting easier ones (like Gravity).
2. InfoNCE Contrastive Loss: Aligns mean-pooled features across different modality groups to encourage modality-invariant representations.
3. Spectral Continuity Regularization ( $\mathcal{L}_{SCR}$ ): An L2 penalty on second-order finite differences across the 8 reconstructed M3 spectral channels. This enforces physically plausible, smooth spectral reconstructions, a constraint specific to planetary spectroscopy not found in Earth EO models.

2.3 Training Protocol

The model is trained for 100 epochs on 256 patches per batch using AdamW. A complementary masking strategy is employed where, with 50% probability, 1–2 anchor groups retain 100% of their tokens while others are masked at 90%, forcing the model to rely heavily on cross-modal inference.

3. Benchmark and Evaluation

The authors define a benchmark suite of six downstream tasks with fixed train/val/test splits:

Geology Classification: 49-class classification of USGS geologic units.
Age Classification: 5-class classification of surface age periods.
Composition Regression: Predicting FeO and TiO2 weight fractions.
Cross-Modal Regression: Predicting thermal values from non-thermal inputs.
Mare Segmentation: Binary segmentation of mare basalts vs. highlands.
Crater Segmentation: Delineation of impact craters (>10 km).

Evaluation metrics include Accuracy/F1, $R^2$ , and mIoU.

4. Key Results

Superior Performance: MG-MAE pretrained features outperform "scratch" baselines, ImageNet-pretrained models, and vanilla MAE baselines on all six tasks.
- Geology: MG-MAE achieves 56.3% accuracy (finetuned), a +16.2% gain over scratch and a massive lead over ImageNet (33.2%).
- Crater Segmentation: MG-MAE achieves 0.768 mIoU, a +14.7 point gain over scratch.
- Composition: Linear probing on MG-MAE features yields an $R^2$ of 0.924, surpassing both scratch (0.802) and the finetuned model (0.908), indicating the encoder effectively captures geochemical trends.
Few-Shot Learning: In low-data regimes (5-shot and 10-shot), MG-MAE significantly outperforms scratch baselines (e.g., +9.3% accuracy on 5-shot geology), demonstrating utility for resource prospecting where ground truth is scarce.
Generalization: Under geographic splits (training on equatorial regions, testing on high latitudes), MG-MAE maintains consistent relative gains (+14.4% over scratch), confirming the representations generalize across different lunar regions.
Domain Specificity: Transfer learning from Earth Observation foundation models (SatMAE, Prithvi-EO-2.0, TerraMind) provides only marginal gains (+1.9% at best) over scratch, confirming that Earth-specific features (vegetation, atmosphere) do not transfer to the Moon.
Ablation Insights:
- Missing-modality masking is the most critical component; removing it drops geology accuracy by 4.6%.
- Grouped tokenization improves performance over per-channel tokenizers by capturing intra-group correlations.
- Spectral continuity regularization improves composition regression by 1.3 $R^2$ points.

5. Significance and Claims

The paper claims Moonstone is the first multi-modal foundation model benchmark for lunar remote sensing. Its significance lies in:

Addressing Data Fragmentation: It unifies disparate lunar datasets into a single, standardized, memory-mapped format.
Enabling AI-Assisted Lunar Science: By providing a benchmark, it allows for fair comparison of models, a capability previously missing for lunar data.
Validating Domain-Specific Pretraining: The results demonstrate that while Earth foundation models fail to transfer, pretraining on lunar data with specific architectural adaptations (handling missing modalities, spectral continuity) yields substantial performance gains.
Scalability: The architecture is designed to accommodate future data from upcoming missions (e.g., Chandrayaan-3, Lunar Trailblazer) as new modality groups without requiring architectural changes.

The authors conclude that multi-modal pretraining is essential for lunar science, particularly for tasks requiring the joint interpretation of morphology, mineralogy, and gravity, and that their released dataset, code, and benchmark suite provide a common protocol for future research in planetary foundation models.