SSL4EO-S12 v1.1: A Multimodal, Multiseasonal Dataset for Pretraining, Updated

This paper introduces SSL4EO-S12 v1.1, an updated multimodal and multiseasonal Earth observation dataset covering the world's 10,000 largest cities that features corrected geospatial alignment, an optimized data structure, and new elevation, land-cover, and vegetation modalities to facilitate efficient pretraining of large-scale foundation models.

The Gist

Imagine you are trying to teach a robot how to understand the Earth. You want it to look at satellite photos and instantly know if it's seeing a city, a forest, a flood, or a farm. To do this, the robot needs to "study" a massive library of pictures.

This paper introduces an upgraded, super-charged library called SSL4EO-S12 v1.1. Think of it as taking a good textbook and turning it into an interactive, 3D multimedia encyclopedia.

Here is the breakdown of what they did, using some everyday analogies:

1. The Problem: The "Misaligned Puzzle"

The previous version of this dataset (v1) was like a jigsaw puzzle where the pieces were slightly the wrong size.

• The Issue: They had two types of satellite cameras: one that sees visible light (like a normal camera) and one that uses radar (like a night-vision camera that sees through clouds). In the old version, the radar picture didn't perfectly line up with the light picture. It was like trying to overlay a map on a photo, but the roads were slightly off.
• The Fix: In v1.1, the team went back and "re-aligned" the pieces. They downloaded bigger chunks of the Earth and carefully stretched or shifted the radar images so they fit perfectly on top of the light images. Now, every pixel matches up exactly.

2. The Upgrade: From "Raw Ingredients" to "Ready-to-Eat Meals"

• Old Way: The old dataset was like giving a chef a bag of raw, unpeeled potatoes and a bag of uncut carrots. The chef (the AI) had to spend a lot of time cleaning and chopping them before they could even start cooking (training the model).
• New Way (ARD): The new v1.1 dataset is "Analysis-Ready Data." It's like a meal kit where the potatoes are already peeled, the carrots are diced, and the spices are measured out. The AI can start learning immediately without wasting time on messy data cleaning.

3. The New Features: Adding "Super Senses"

The original dataset only had two "senses":

1. Eyes: Optical photos (Sentinel-2).
2. Radar: Night-vision style photos (Sentinel-1).

v1.1 adds three new senses:

• The "Height" Sense (Elevation/DEM): Imagine looking at a flat map vs. a 3D model of a mountain. This new data tells the AI how high the ground is, helping it distinguish between a flat road and a hill.
• The "Green" Sense (Vegetation/NDVI): This is like a special filter that highlights how healthy the plants are. It helps the AI tell the difference between a lush forest and a dry patch of dirt.
• The "Map" Sense (Land Cover): This is like a color-coded legend that tells the AI, "This pixel is water," or "This pixel is a building."

4. The Time Machine: Seeing Seasons Change

Instead of just showing one photo of a city, this dataset shows four photos of the same spot, taken in different seasons (Spring, Summer, Autumn, Winter).

• Why it matters: A forest looks green in summer and brown in winter. A field might be flooded in spring and dry in summer. By showing the AI the same place over time, it learns how the world changes, rather than just memorizing a single snapshot.

5. The Delivery System: The "Streaming Box"

Storing nearly a million image patches is like trying to store a million books in a library. If you just pile them in a room, it's a nightmare to find anything.

• The Solution: The team packed these images into special digital boxes called Zarr files and WebDataset shards.
• The Analogy: Imagine instead of giving the AI a library card to walk into a warehouse and find a book, they give it a high-speed streaming service (like Netflix). The AI can "stream" the data it needs instantly, without waiting for the whole library to load. This makes training the AI much faster and cheaper.

The Big Picture: Why Does This Matter?

This dataset is a foundation for the next generation of AI.

• Before: AI models were like students who only studied one subject (just photos).
• Now: With v1.1, the AI is a student studying photos, radar, height maps, and vegetation all at once, across different seasons.

The result? These AI models become much smarter. They can better predict floods, track deforestation, plan city growth, and detect crop diseases. The authors have made this "textbook" free for everyone to use, hoping it helps researchers build better tools to understand and protect our planet.

In short: They fixed the alignment errors, added new "senses" (height and plant health), organized the data so it's easy to use, and made it available for free to help AI learn how to see the Earth more clearly.

Technical Summary

1. Problem Statement

While the original SSL4EO-S12 dataset successfully facilitated the development of Earth Observation (EO) Foundation Models (FMs) by providing multi-seasonal, multi-modal data, it suffered from two critical limitations that hindered its utility for large-scale pre-training:

• Geospatial Misalignment: There were spatial inaccuracies between Sentinel-1 (S-1) and Sentinel-2 (S-2) patches, particularly due to projection distortions (EPSG:4326) and coordinate mismatches.
• Data Readiness: The dataset lacked "Analysis-Ready Data" (ARD) standards. Issues included inefficient data structures, unfiltered NaN values, and suboptimal radiometric scaling for RGB visualization, making direct integration into modern training pipelines difficult.
• Modal Limitations: The original version lacked auxiliary modalities (elevation, land cover, vegetation indices) necessary for robust multimodal pre-training.

2. Methodology

The authors developed SSL4EO-S12 v1.1 by reprocessing data acquired from Google Earth Engine (2019–2021) with a focus on spatial accuracy, data quality, and multimodal alignment.

• Spatial Alignment & Projection:

  • Instead of projecting all data to EPSG:4326 (which distorts distances near poles), the dataset retains native UTM zones to preserve spatial accuracy.
  • For each location, a larger area of interest is downloaded for both S-1 and S-2 to buffer reprojection mismatches.
  • S-1 GRD data is reprojected to match S-2 L2A coordinates. If a location spans multiple UTM zones, the majority zone is selected, and timestamps are reprojected to a common 264x264 pixel grid (chosen for divisibility by 6 to accommodate 10m, 20m, and 60m resolutions).
  • All bands are upsampled to 10m resolution using nearest neighbor interpolation.

• Data Cleaning & Filtering:

  • NaN Handling: Locations with >1% missing values (NaN) in any timestamp/channel are discarded. Remaining NaNs are filled via nearest neighbor interpolation.
  • Cloud Masking: Original S-2 cloud masks are deemed unreliable for training. The authors use the SEnSeI v2 model to generate accurate cloud, snow, and shadow masks.
  • Radiometric Scaling: To prevent saturation in bright regions (clouds/snow) or darkness in others, a custom quantile-based scaling is applied to RGB data. Values are adjusted based on 2% and 98% quantiles, then mapped to 8-bit (0–255) using 0.2% and 99.8% quantiles as bounds.

• Multimodal Integration:

  • New modalities were aligned with the S-2 grid:
    • NDVI: Computed from S-2 L2A reflectances.
    • LULC (Land Use/Land Cover): Sourced from ESRI, augmented with SEnSeI v2 masks for temporal consistency.
    • DEM (Digital Elevation Model): Copernicus DEM at 30m, resampled to 10m.
  • The dataset covers 246,144 distinct locations (time series of 4 timestamps each) within a 50km radius of the world's 10,000 largest cities, resulting in 984,576 total samples.

• Data Packaging & Storage:

  • Data is stored in Zarr format (ZipStore, Zarr v2) within WebDataset tar shards.
  • This structure allows for efficient streaming, lazy loading (via Dask/xarray), and reduced file counts for HPC/HuggingFace storage.
  • Metadata (coordinates, timestamps, file IDs) is embedded directly with the image data.

3. Key Contributions

1. Geospatial Correction: A rigorous reprojection pipeline ensuring perfect alignment between S-1 and S-2 modalities across global locations.
2. Analysis-Ready Format (ARD): Strict filtering of NaNs, accurate cloud masking via SEnSeI v2, and optimized radiometric scaling for RGB data.
3. Multimodal Expansion: Introduction of aligned NDVI, LULC, and DEM layers, enabling richer self-supervised learning tasks beyond raw spectral/radar data.
4. Efficient Infrastructure: Implementation of a Zarr/WebDataset pipeline that supports streaming and efficient data loading for large-scale foundation model training.
5. Open Release: The dataset is released under CC-BY-4.0 on Hugging Face, covering 246k time series with nearly 1 million image patches.

4. Experimental Results

The paper evaluates the impact of v1.1 on the PANGAEA benchmark across multiple downstream tasks (e.g., land cover, flood detection, burn scars).

• Baseline Comparison: Models trained on v1 data (MAE, DINO, MoCo) using only S-2 data showed lower performance compared to CROMA (trained on S-1 and S-2).
• Multimodal Superiority: The TerraMind model, trained on SSL4EO-S12 v1.1 using all available modalities (S-1, S-2, LULC, DEM, NDVI), achieved the best performance on nearly all benchmark tasks compared to single-modality or dual-modality baselines.
• Key Insight: The addition of semantic (LULC) and topographic (DEM) modalities directly enabled token prediction tasks that improved representation learning, confirming the value of multimodal pre-training.

5. Significance

SSL4EO-S12 v1.1 represents a significant step forward in Earth Observation data science by:

• Solving Alignment Issues: It removes the technical barrier of spatial misalignment that previously complicated multimodal fusion.
• Enabling Next-Gen FMs: By providing a clean, multimodal, and temporally rich dataset, it serves as a robust foundation for training the next generation of EO foundation models capable of complex reasoning (e.g., change detection, urban planning, disaster monitoring).
• Standardizing ARD: It sets a new standard for "Analysis-Ready" remote sensing data, facilitating easier integration into modern deep learning pipelines (PyTorch/TensorFlow) via WebDataset streaming.
• Community Impact: As part of the open-source SSL4EO-* family, it democratizes access to high-quality, pre-processed global data, fostering research in multimodal fusion and geospatial representation learning.