EarthScape: A Multimodal Dataset for Surficial Geologic Mapping and Earth Surface Analysis

The paper introduces EarthScape, a multimodal dataset and reproducible pipeline designed to automate surficial geologic mapping by integrating diverse geospatial data sources, demonstrating that terrain features provide the most robust predictive signal while highlighting the dataset's utility for benchmarking multimodal fusion and domain adaptation.

The Gist

Imagine you are trying to understand the history of a neighborhood just by looking at a map. You want to know: Is this ground made of old river mud? Is it a pile of rocks that slid down a hill? Or is it just dirt that has been sitting there for thousands of years?

In the world of geology, this is called Surficial Geologic Mapping. It's crucial for building safe roads, finding minerals, and predicting landslides. But right now, making these maps is like trying to solve a massive jigsaw puzzle by hand, one tiny piece at a time. It takes geologists years of fieldwork, hiking, and staring at satellite photos to draw these lines. It's slow, expensive, and hard to scale.

Enter EarthScape.

Think of EarthScape as a "Gym for Artificial Intelligence" designed specifically to teach computers how to read the Earth's surface. It's a new, massive dataset that acts as a training ground for AI models to become expert geologists.

Here is the breakdown of what makes EarthScape special, using some everyday analogies:

1. The "Multi-Sensory" Detective

Most AI models for looking at the Earth are like detectives who only have one sense: sight. They look at a photo (RGB imagery) and guess what's there.

• The Problem: A photo of a muddy riverbank might look identical to a photo of a muddy construction site. The AI gets confused.
• The EarthScape Solution: EarthScape gives the AI a full sensory toolkit. It doesn't just give the AI a photo. It gives it:
  • The "Eyes": High-resolution aerial photos (what it looks like).
  • The "Height Sense": Digital Elevation Models (how high the ground is).
  • The "Shape Sense": Calculated terrain features (how steep, how curvy, how rough the ground is).
  • The "Context Clues": Where the rivers and roads are.

It's like giving a detective not just a photo of a crime scene, but also a 3D model of the room, a map of the sewer lines, and a list of who lives nearby. With all this info, the AI can finally tell the difference between natural mud and human-made fill.

2. The "Training Camp" with Two Towns

To make sure the AI is actually smart and not just memorizing the answers, the researchers set up a unique training camp.

• Town A (Warren County): The AI trains here. It learns to identify the geology of this specific area.
• Town B (Hardin County): The AI is tested here. It has never seen this town before.

This is like teaching a student to drive in a small, quiet town and then immediately testing them on a busy highway in a different city. If the student can drive well in the new city, they truly understand the rules of the road, not just the specific turns of the first town.

• The Result: The paper found that AI models that relied only on photos failed miserably when moved to the new town. But models that learned the shape and slope of the land (the "terrain features") were able to generalize and work well in the new location. It turns out, the shape of the land is a more universal language than the color of the land.

3. The "Long Tail" Problem

In this dataset, some types of ground are super common (like "Residuum," which is just weathered bedrock), while others are rare (like "Alluvial Fans," which are specific fan-shaped piles of rocks).

• The Analogy: Imagine a classroom where 90% of the students are wearing red shirts, and only 1% are wearing blue. If you ask a teacher to find the blue-shirted students, they might just guess "red" every time because it's statistically safer.
• The Challenge: EarthScape is designed to force the AI to pay attention to those rare, "blue-shirt" geological features, which are often the most dangerous or important ones (like landslide-prone areas).

4. Why This Matters

Before EarthScape, if you wanted to map the geology of a new area, you had to hire a team of geologists to spend months walking the land.

• With EarthScape: We are building the foundation for an AI that can look at a satellite image, understand the shape of the terrain, and instantly generate a geologic map.
• The Impact: This could help us build safer cities, find critical minerals faster, and predict natural disasters more accurately, all without needing to send humans into dangerous terrain first.

The Bottom Line

EarthScape is a "living" dataset. It's not just a static file; it's a growing library of Earth's surface data. The researchers are saying, "We built the gym, we provided the weights, and we showed you how to train. Now, let's build AI that can read the Earth's story as well as a human expert, but at the speed of light."

It's a step toward a future where our computers can "see" the geology beneath our feet, helping us live more safely and sustainably on this planet.

Technical Summary

1. Problem Statement

Surficial Geologic (SG) mapping involves identifying the spatial distribution of unconsolidated materials (e.g., alluvium, colluvium, artificial fill) on the Earth's surface. These maps are critical for critical mineral resource exploration, geologic hazard mitigation, and infrastructure planning. However, current workflows face significant bottlenecks:

• Labor Intensity: Traditional mapping relies on manual fieldwork and expert visual interpretation of remote sensing (RS) imagery, making it slow and expensive (estimated at $123k per standard 1:24k-scale map).
• Scalability Issues: Existing automated approaches using deep learning are often limited to specific geologic hazards (e.g., landslides) or narrow geographic areas. They lack standardized benchmarks, struggle with domain shift (generalizing to new regions), and fail to effectively integrate diverse data modalities.
• Data Gaps: While general RS datasets exist (e.g., for urban land cover), there is a lack of large-scale, multimodal datasets specifically designed for continuous surficial geologic materials, which require understanding complex surface morphology and multi-scale terrain features.

2. Methodology

A. The EarthScape Dataset

The authors introduce EarthScape, the first multimodal, multi-scale benchmark dataset designed specifically for SG mapping.

• Composition: The dataset comprises 31,018 georeferenced patches (256×256 pixels, ~390m²) from two distinct regions in Kentucky, USA (Warren and Hardin Counties), separated by ~75 km.
• Modalities (38 Co-registered Channels per Patch):
  1. Targets: Rasterized SG maps (7 mutually exclusive classes: Artificial fill, Alluvium, Alluvial fans, Terrace deposits, Colluvium, Colluvial aprons, Residuum).
  2. Imagery: High-resolution Aerial RGB and Near-Infrared (NIR) imagery (0.15m GSD).
  3. Elevation: LiDAR-derived Digital Elevation Models (DEM) (1.52m GSD).
  4. Terrain Derivatives: Five terrain features (Slope, Profile Curvature, Planform Curvature, Elevation Percentile, Standard Deviation of Slope) computed at six spatial scales (from 1.52m to 60.96m GSD) to capture hierarchical surface structures.
  5. Vector Data: Rasterized hydrography (NHD) and infrastructure (OSM) layers.
• Characteristics: The dataset exhibits severe class imbalance (Residuum dominates ~94% of patches) and long-tail distribution, mirroring real-world geologic conditions. It is explicitly designed to test cross-region generalization due to the geographic separation of the training (Warren) and test (Hardin) sets.

B. Model Architecture: SGMap-Net

The authors propose SGMap-Net, a lightweight, interpretable baseline model designed to integrate multimodal inputs without architectural over-complication.

• Standardization: Input modalities are standardized to a 3-channel representation via 1×1 convolutions.
• Feature Extraction: A shared encoder (ResNeXt-50 or ViT-B/16) processes the standardized inputs.
• Fusion Strategy: The core innovation is an attention-weighted aggregation mechanism. Modality features are projected into a latent space, and a Multi-Head Attention (MHA) mechanism computes attention weights based on the query (current modality) and keys/values (other modalities). This allows the model to adaptively emphasize the most informative modalities for each sample.
• Baselines: The study compares SGMap-Net against:
  • Early channel stacking (concatenating all inputs).
  • Mid-level feature concatenation.
  • Existing foundation models (SatMAE, DOFA, Panopticon-FM) adapted for terrain data.

C. Experimental Setup

• Task: Multi-label classification (predicting the presence/absence of each of the 7 SG units per patch).
• Splits:
  • In-Domain (ID): Training/Validation/Test from Warren County.
  • Cross-Domain (CD): Test set from Hardin County to evaluate geographic generalization.
• Metrics: Macro-averaged F1, AUC, Precision, Recall, and Mean Average Precision (mAP).

3. Key Contributions

1. EarthScape Dataset: A unique, AI-ready resource integrating imagery, elevation, multi-scale terrain derivatives, and vector data, addressing the lack of standardized benchmarks for continuous SG mapping.
2. Unified Framework: A reproducible pipeline for co-registering heterogeneous data sources (raster and vector) into a single multimodal framework.
3. Comprehensive Benchmarks: Systematic evaluation of single-modality, multi-scale, and multimodal configurations, establishing baselines for fusion strategies and backbone architectures.
4. Insight into Geologic AI: Demonstrating that terrain-derived shape features are more robust and transferable across regions than raw spectral (RGB) or absolute elevation data.

4. Key Results

A. Modality Performance

• Terrain Features Dominate: Terrain derivatives (specifically Slope and Elevation Percentile) provided the strongest predictive signals.
  • In-Domain F1: Slope (S) and Elevation Percentile (EP) achieved ~0.65, outperforming RGB (0.599) and raw DEM (0.632).
  • Cross-Domain Robustness: Terrain features showed significantly less performance degradation when moving from Warren to Hardin County compared to RGB and DEM. For instance, Slope (S) dropped only 0.049 in F1, whereas RGB dropped 0.267.
• Raw Inputs Degrade: Raw spectral and elevation inputs are highly sensitive to local appearance and geographic context, leading to poor generalization.

B. Multi-Scale and Multimodal Fusion

• Multi-Scale Benefit: Combining terrain features across multiple scales improved cross-region robustness.
• Best Configuration: The combination of Multi-scale EP + Slope + SDS (Standard Deviation of Slope) achieved the highest performance:
  • In-Domain F1: 0.657
  • Cross-Domain F1: 0.598
  • Performance Drop: Only 0.059 (the smallest among all configurations).
• Fusion Strategy: Early channel stacking (concatenating inputs before the encoder) consistently outperformed attention-based fusion and mid-level concatenation. This suggests that for geologic tasks, simple, dense integration of shape-based cues is more effective than complex attention mechanisms in this specific context.

C. Comparison with Foundation Models

• SGMap-Net vs. Pretrained Models: SGMap-Net significantly outperformed adapted foundation models (SatMAE++, DOFA, Panopticon-FM).
  • Pretrained models relying on spectral inputs suffered severe cross-domain collapse (e.g., Panopticon-FM dropped 0.257 in F1).
  • Even when extended to accept terrain channels, SatMAE++ showed a large drop (0.202) compared to SGMap-Net's stability.
• Conclusion: Architectures pre-trained on spectral satellite imagery are insufficient for surface-aware geologic tasks; models must be explicitly designed to leverage geomorphic shape descriptors.

D. Class-Level Analysis

• Frequency vs. Separability: Performance did not strictly correlate with class frequency. Rare classes like Alluvial Fans (Qaf, 0.9%) and Terrace Deposits (Qat, 4.6%) achieved competitive AUC scores, indicating that distinct spatial expressions can offset low prevalence.
• Backbone Differences: ResNeXt-50 generally performed better on smaller, linear features (e.g., artificial fill), while ViT-B/16 excelled on broader, regional geomorphic patterns.

5. Significance and Impact

• Advancing Geospatial AI: EarthScape bridges the gap between computer vision and geoscience, providing a rigorous testbed for domain adaptation and multimodal fusion in a physically constrained environment.
• Scalable Mapping: The results suggest that automated SG mapping can be made scalable and reproducible by focusing on terrain morphology rather than just spectral signatures, potentially reducing the cost and time of creating geologic maps.
• Living Dataset: Designed as a "living dataset," EarthScape is extensible to new regions and modalities, fostering a community-driven approach to geospatial benchmarking.
• Generalizability: The findings highlight that shape-based descriptors (derived from DEMs) are more invariant to geographic shifts than raw imagery, a principle that could benefit other surface-aware tasks like autonomous navigation and medical imaging.

In summary, EarthScape demonstrates that for surficial geologic mapping, multi-scale terrain analysis is the critical signal, and simple, geologically-informed fusion strategies outperform complex, generic foundation models when dealing with cross-region generalization.