🤖 machine learning

SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining

The paper introduces SpectralEarth-FM, a hierarchical transformer model and its accompanying 40TB SpectralEarth-MM dataset, which enable state-of-the-art joint pretraining of hyperspectral imagery with multispectral, SAR, and thermal Earth observation data to overcome the limitations of single-sensor or single-modality foundation models.

Original authors: Nassim Ait Ali Braham, Aaron Banze, Conrad M. Albrecht, Julien Mairal, Jocelyn Chanussot, Xiao Xiang Zhu

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nassim Ait Ali Braham, Aaron Banze, Conrad M. Albrecht, Julien Mairal, Jocelyn Chanussot, Xiao Xiang Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the Earth as a giant, complex puzzle. For years, scientists trying to understand this puzzle have been using different types of "lenses" to look at it. Some lenses see the world in broad, colorful strokes (like standard satellite photos), while others see it in black-and-white radar or heat maps.

For a long time, there was a very special, high-powered lens called Hyperspectral Imagery (HSI). This lens doesn't just see colors; it sees hundreds of tiny, specific shades of light, allowing it to identify exactly what materials are on the ground (like distinguishing between two types of wheat or spotting a specific mineral). However, this special lens was mostly used in isolation. Scientists had powerful models for the "broad" lenses and separate models for the "special" lens, but they rarely taught them to work together.

SpectralEarth-FM is the new system designed to fix this. Here is how it works, broken down into simple concepts:

1. The Massive Library (SpectralEarth-MM)

Before a student can learn, they need a library of books. The authors built a massive digital library called SpectralEarth-MM.

The Collection: It contains over 40 Terabytes of data—enough to fill thousands of hard drives.
The Mix: It doesn't just have the "special" hyperspectral photos. It pairs them with standard photos, radar images, and heat maps, all taken from the exact same spot on Earth at the same time.
The Scale: It covers about 2 million different locations across the globe, creating 25 million matching picture sets. Think of it as a global photo album where every page has a high-definition color photo, a radar scan, and a heat map of the same backyard, all perfectly aligned.

2. The Smart Translator (The Architecture)

The problem is that these different "lenses" speak different languages. The hyperspectral camera speaks in hundreds of tiny frequency words, while the radar camera speaks in a few loud, broad words. If you try to feed them into a standard brain, it gets confused.

SpectralEarth-FM acts like a brilliant translator with a unique brain structure:

Specialized Assistants: It has different "assistants" (branches) for each type of camera. The assistant for the hyperspectral camera is trained to understand complex, detailed spectral patterns. The assistant for the radar camera is trained to understand texture and shape.
The Fusion Table: Once each assistant understands its own piece of the puzzle, they bring their notes to a central table. Here, a "fusion module" combines their insights into a single, unified understanding.
The Shared Brain: Finally, this combined understanding is passed to a shared "brain" (a hierarchical encoder) that learns the big picture, allowing the system to answer questions about the Earth using any combination of these sensors.

3. The Learning Method (JEPA Pretraining)

How do you teach a model this without a teacher giving it the answers? The authors used a game called JEPA (Joint-Embedding Predictive Architecture).

Imagine you are looking at a landscape.

The Teacher: The "Teacher" looks at the whole landscape using all available sensors (the full picture) and forms a deep, abstract idea of what is there.
The Student: The "Student" is given a tiny, blurry crop of the landscape using only one sensor (e.g., just the radar or just the hyperspectral view).
The Challenge: The Student must guess the Teacher's deep idea based on that tiny, single-sensor view.
The Result: By playing this game millions of times, the model learns to understand the essence of the Earth. It learns that a specific radar texture combined with a specific heat signature usually means "forest," even if it hasn't seen the full color photo yet. It learns to predict the "meaning" of a scene rather than just copying the pixels.

4. The Results

The authors tested this new system on two types of challenges:

Hyperspectral Tasks: Can it identify specific minerals or crop types using the high-detail lens? Yes. It outperformed all previous models that were trained only on hyperspectral data.
General Earth Tasks: Can it handle standard tasks like flood monitoring or land cover classification using the mix of sensors? Yes. It performed as well as, or better than, the best existing models that use standard satellite data.

The Bottom Line

SpectralEarth-FM is a new kind of "Earth brain" that finally learns to read the Earth using all its senses at once. By combining the ultra-detailed vision of hyperspectral cameras with the robust, all-weather vision of radar and standard cameras, it creates a more complete and accurate understanding of our planet than ever before. The authors have also promised to release the massive library and the brain's code to the public so others can continue to build on this foundation.

Technical Summary: SpectralEarth-FM

Problem Statement

Earth Observation (EO) foundation models (FMs) have increasingly adopted multi-sensor pretraining, integrating multispectral imagery (MSI), synthetic aperture radar (SAR), and derived geospatial layers. However, hyperspectral imagery (HSI), which offers high-dimensional, spectrally contiguous measurements (~10nm bandwidth), remains underrepresented in these multimodal frameworks. Conversely, existing HSI foundation models are typically trained on HSI data alone, leaving the joint pretraining and fusion of HSI with co-located MSI, SAR, and thermal observations unexplored.

Current modeling approaches face specific challenges when integrating HSI:

Spectral Complexity: HSI contains strong inter-band correlations and sensor-dependent noise, requiring explicit spectral–spatial processing rather than simple patch projection.
Heterogeneity: Integrating HSI with lower-channel sensors (MSI, SAR, Land Surface Temperature) requires an architecture that preserves fine-grained spectral processing while supporting broadband optical inputs.
Data Scarcity: There is a lack of large-scale, spatially aligned datasets that pair spaceborne HSI with other major EO modalities (Sentinel-2, Landsat, Sentinel-1) over common geographic footprints.

Methodology

1. Dataset: SpectralEarth-MM

To enable large-scale pretraining, the authors curated SpectralEarth-MM, a multi-sensor dataset centered on spaceborne HSI missions.

Sources: The dataset anchors on HSI from three sensors: EnMAP, EMIT, and DESIS.
Co-location: These are spatially aligned with Sentinel-2, Landsat-8/9 (MSI and Land Surface Temperature), and Sentinel-1 (SAR).
Scale: The dataset covers approximately 2 million globally distributed locations, comprising 25 million georeferenced patches and over 40 TB of data.
Construction: The pipeline involves sensor-specific quality control (cloud screening, georeferencing checks), patch extraction into non-overlapping $(3.84 \text{ km})^2$ footprints, and hierarchical rebalancing using AlphaEarth embeddings to reduce spatial and semantic redundancy while preserving rare sensor configurations.

2. Architecture: SpectralEarth-FM

The authors propose SpectralEarth-FM, a hierarchical transformer designed for heterogeneous spectral dimensionality. The architecture consists of four main components:

Spectral Tokenization (HSI): Unlike standard patch projections, HSI inputs undergo a sensor-specific spectral tokenizer. Bands within a spatial patch are partitioned into contiguous groups, embedded via dedicated projections, and processed by a Spectral Transformer (SpecTr). Learnable spectral position encodings inject order information. The resulting group tokens are aggregated into a single spatial token.
Sensor-Specific Local Branches: Lower-dimensional sensors (MSI, SAR, LST) use standard convolutional patch projections. All branches then pass through Local Hierarchical Blocks (LHBs) based on the Hiera design. These branches operate independently to handle sensor-specific statistics before fusion.
Cross-Sensor Fusion: After local encoding, branch embeddings are fused at each spatial position. A learned query attends over the projected sensor tokens (augmented with sensor-specific embeddings) to produce a single fused token grid.
Shared Hierarchical Encoder: The fused tokens are passed to a shared Hiera encoder to produce the final representation $Z$ .

3. Pretraining Strategy

The model is pretrained using a Joint-Embedding Predictive Architecture (JEPA)-style objective, adapted for multimodal EO fusion.

Objective: Instead of reconstructing raw pixels (which is difficult across modalities with different noise characteristics), the model learns to match representations between global views and single-sensor local views from the same location.
Teacher-Student Design: An asymmetric setup is used where an Exponential Moving Average (EMA) teacher processes only global views (containing multiple modalities) to define a stable target. The student processes global, local, and sensor-dropped views to predict this target.
Loss Function: The objective combines an invariance loss (MSE between student and teacher projections) with a SIGReg penalty to prevent representation collapse.

Key Contributions

SpectralEarth-MM: A globally balanced, multi-temporal corpus of 40+ TB spanning HSI, MSI, SAR, and LST, specifically designed to bridge the gap between hyperspectral and multi-sensor EO.
SpectralEarth-FM: A novel foundation model architecture featuring explicit spectral tokenization for HSI, sensor-specific local branches, and a cross-sensor fusion module within a hierarchical transformer framework.
Multimodal JEPA Pretraining: A tailored pretraining strategy that avoids direct pixel reconstruction across heterogeneous sensors, instead focusing on aligning latent representations between global and local views.
Comprehensive Evaluation: Extensive benchmarking on ten hyperspectral downstream tasks and the standard PANGAEA suite of EO benchmarks.

Results

The paper evaluates SpectralEarth-FM under two settings: single-branch routing (routing inputs to the closest spectral branch) and multi-branch routing (concatenating features from multiple branches).

Hyperspectral Benchmarks: On ten downstream tasks (including CDL, BD-Foret, EuroCrops, and OxHyperMinerals), SpectralEarth-FM-a (multi-branch) achieved the best overall average rank (1.4), outperforming state-of-the-art models like HyperSIGMA, Panopticon, and CARL. It showed particular strength on EnMAP-based tasks and demonstrated successful transfer to unseen sensors (e.g., EO-1 Hyperion, GF-5) via spectral interpolation.
General EO Benchmarks (PANGAEA): Evaluated on seven tasks (e.g., HLS Burns, Sen1Floods11, AI4Farms), SpectralEarth-FM-a achieved an average rank of 3.43, placing it among the top-performing models (surpassing TerraMind, CROMA, and Prithvi in the comparison).
Ablation Studies:
- Branch Selection: The EnMAP branch generally yielded the highest performance on HSI tasks, while combining all HSI branches further improved results.
- Sensor Fusion: Fusing EnMAP with Sentinel-2 or Landsat OLI provided complementary information, yielding significant gains (e.g., +3.4 mIoU on EuroCrops) compared to single-sensor inputs.

Significance and Claims

The authors claim that SpectralEarth-FM represents a significant step forward in integrating hyperspectral data into the broader ecosystem of Earth Observation foundation models. By explicitly addressing the high spectral dimensionality of HSI through dedicated tokenization and fusion mechanisms, the model achieves state-of-the-art performance on hyperspectral tasks while maintaining competitive performance on standard multi-sensor benchmarks.

The work highlights that:

Joint pretraining of HSI with MSI, SAR, and LST is feasible and beneficial, allowing models to learn representations that generalize across sensors.
The proposed JEPA-style objective effectively handles the heterogeneity of sensor noise and spectral resolution without requiring pixel-level reconstruction.
The release of SpectralEarth-MM and SpectralEarth-FM provides a necessary resource for future research in multimodal EO, addressing the historical gap where HSI was either isolated or excluded from large-scale foundation models.

The paper concludes modestly, noting that while the model achieves strong in-distribution performance, the complex interplay of spatial and spectral features in fused representations warrants further investigation into explainability. Additionally, the current framework is dedicated to specific pretraining sensors, trading out-of-the-box flexibility for stronger spectral representations, a trade-off the authors acknowledge for future work.