🤖 machine learning

Now We Know? A Systematic Comparison of TerraMind and THOR

This paper presents a systematic comparison of two European Space Agency Geospatial Foundation Models, THOR and TerraMind, across ten diverse use cases to demonstrate that architectural design choices—specifically patch size and decoder complexity—explain more performance variance than model identity, revealing complementary investment strategies and establishing a diagnostic methodology for future GFM evaluation.

Original authors: Frederick Schindlegger, Kenzo Bounegta, Eva Gmelich Meijling, Johannes Jakubik, Arnt-Børre Salberg, Theodor Forgaard, Nicolas Longepe, Valerio Marsocci

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Frederick Schindlegger, Kenzo Bounegta, Eva Gmelich Meijling, Johannes Jakubik, Arnt-Børre Salberg, Theodor Forgaard, Nicolas Longepe, Valerio Marsocci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to understand the Earth from space. For years, scientists have been building "Geospatial Foundation Models" (GFMs). Think of these models as super-smart, pre-trained brains that have already looked at billions of satellite photos. They are like a student who has read every geography textbook in the library before ever taking a specific test. Because they've seen so much, they can learn new tasks—like finding floods or tracking icebergs—much faster than a model starting from scratch. But here's the tricky part: we have many of these "super-brains," and they all get different scores on tests. Sometimes one is better at finding floods, and another is better at spotting fires. The big question is: Why? Is one model just smarter because of its brain architecture? Is it because it has a better "decoder" (the part that translates brain signals into a map)? Or is it just lucky with the specific pictures it was tested on? Until now, we've mostly just looked at the final scores, like a sports leaderboard, without understanding the mechanics behind the win.

This paper, titled "Now We Know?", is like a detective story where two rival detectives, THOR and TerraMind, are put through a series of controlled experiments to see what actually makes them tick. Both were built by teams funded by the European Space Agency, but they have very different philosophies. THOR is the "chameleon": it can change its "patch size" (how it slices up an image) on the fly, trading computer power for sharper details when it needs to. TerraMind is the "generative artist": it was trained to imagine missing pieces of a puzzle, allowing it to fill in gaps or switch between different types of sensors (like turning a radar image into an optical one) using a trick called "Thinking-in-Modalities."

The researchers didn't just let them fight it out on a leaderboard. Instead, they ran a massive "ablation study," which is a fancy way of saying they took the models apart and reassembled them piece by piece to see which part mattered most. They tested these models on ten different real-world missions, from tracking methane leaks to mapping snow cover.

Here is what they found, and it's a bit of a plot twist. First, they discovered that the "leaderboard" score is often a liar. The biggest factor determining who wins isn't necessarily which model (THOR or TerraMind) you pick, but how you slice the image and how complex your decoder is. If you slice an image into tiny pieces (small patch size), THOR becomes incredibly powerful, especially for spotting small, tricky things like icebergs or flood zones, because it sees the fine details. However, this comes at a high cost: it requires a lot of computer power. TerraMind, on the other hand, uses a fixed slice size. It shines when you need to process optical images (like standard photos) efficiently, but it struggles a bit more with radar-only data unless you give it a very powerful decoder.

The study also ruled out a few common assumptions. For instance, many people thought that simply fusing data from two different satellites (like combining radar and optical) would always make the model smarter. The researchers found that, with full datasets, this wasn't always true; sometimes, just using the optical data alone was better because the fusion method they used was too simple to handle the extra noise. They also found that "freezing" the model's brain (not letting it learn new things during the test) worked surprisingly well for TerraMind on optical tasks, but THOR often needed to be fully "unfrozen" (allowed to learn) to reach its full potential, especially on radar tasks.

Ultimately, the paper suggests there is no single "winner." It's not about which model is the best in the world; it's about matching the tool to the job. If you have a powerful computer and need to find tiny, specific objects in radar images, THOR with a tiny patch size is your hero. If you need a fast, efficient model for general optical mapping, TerraMind might be the better choice. The authors conclude that the future of these models isn't just about building bigger brains, but about understanding the specific trade-offs between how we slice the data, how we decode the answers, and what kind of "brain" we are using. It's a reminder that in the world of AI, context is everything.

Title: Now We Know? A Systematic Comparison of TerraMind and THOR

Problem Statement
Current benchmarks for Geospatial Foundation Models (GFMs) increasingly rely on aggregate scores to rank models. The authors argue that such rankings obscure the underlying reasons for performance differences, conflating the contributions of the pretrained backbone, decoder capacity, fine-tuning regimes, and spatial tokenization schemes. This makes it difficult to determine whether performance gaps stem from genuine architectural advantages or dataset-specific artifacts. Furthermore, existing benchmarks often fail to answer why models differ, particularly regarding how design choices like patch size, decoder complexity, and input modality influence real-world performance across diverse Earth Observation (EO) domains.

Methodology
The study conducts a controlled ablation study comparing two GFMs developed under the European Space Agency's $\phi$ -lab, which embody contrasting design philosophies:

THOR: A Vision Transformer (ViT) utilizing a compute-adaptive architecture (FlexiViT) that supports variable patch sizes at inference. It unifies Sentinel-1, -2, and -3 data at their native resolutions (10 m–1 km) using ground-sampling-distance (GSD)-aware positional encoding.
TerraMind: A multimodal generative GFM pretrained on 500 billion tokens across eight modalities (including SAR, optical, LULC, DEM, and text). It employs a dual-scale token/pixel objective and a "Thinking-in-Modalities" (TiM) mechanism to infer missing sensors or generate auxiliary modalities during inference.

The evaluation spans ten use cases covering semantic segmentation, regression, object detection, and change detection across diverse domains (e.g., flood mapping, methane detection, snow monitoring, sea ice). The methodology adheres to the following principles:

Controlled Variables: Hyperparameters not intrinsic to the architecture are fixed (following PANGAEA protocols), while structural specificities (e.g., THOR's variable patch sizes) are exploited.
Capacity Matching: Models are allowed to run at their best achievable configuration within hardware constraints (single RTX A6000 GPU), rather than degrading one to match another.
Ablation Axes: The study systematically varies patch size (for THOR), decoder complexity (Linear vs. UperNet), backbone adaptation (frozen vs. fine-tuned), input modality, and model scale (Tiny vs. Base).
Diagnostic Approach: Rather than a single leaderboard, the study isolates marginal contributions of specific design axes to understand performance variance.

Key Contributions

Large-Scale Controlled Ablation: The authors executed a comprehensive comparison across two architecturally distinct GFMs and ten EO use cases.
Diagnostic Methodology: They introduced a framework to isolate the marginal contribution of architectural deployment choices (e.g., patch size, decoder type) from the model identity itself.
Dataset-Level Characterization: The study demonstrates that correctly interpreting ablation results requires characterizing use cases at the dataset level, arguing this is a necessary component of rigorous benchmarking rather than a preprocessing detail.
Generalizable Hypotheses: The authors distilled patterns into hypotheses expected to generalize to ViT-based GFMs beyond THOR and TerraMind, offering practical deployment guidance.

Results

Architectural Design vs. Model Identity: Architectural design choices, particularly patch size and decoder type, explain more performance variance than the model identity itself.
Complementary Strategies: The models represent complementary investment strategies. TerraMind invests in pretraining-time scale (500B tokens, dual-scale objectives) to produce strong fixed-resolution features, while THOR shifts the burden to inference-time tokenization (variable patch sizes) to trade compute for resolution.
Patch Size Dominance: Spatial tokenization is the dominant performance lever. Halving the patch size (e.g., from 32 to 16) often yields the largest accuracy gains, particularly for tasks with spatially compact or minority-class targets (e.g., flood extents, icebergs). However, the relationship is not strictly monotonic and is task-dependent.
Decoder Interaction: The performance gap between complex decoders (UperNet) and simple linear decoders shrinks as patch size decreases. For THOR, finer tokenization (e.g., ps4/ps8) reduces the need for complex decoders, as the backbone features already encode sufficient local detail. Conversely, TerraMind (fixed ps16) consistently benefits from multi-scale decoders, especially in optical segmentation tasks.
Modality and Sensor Integration:
- Optical vs. SAR: In full-data regimes, simple fusion of Sentinel-1 (SAR) and Sentinel-2 (Optical) did not consistently outperform Sentinel-2 alone, suggesting shallow fusion mechanisms may introduce noise when optical signals are abundant. However, SAR contribution was positive in low-data regimes.
- Cross-Sensor Generalization: TerraMind, despite lacking Sentinel-3 in its pretraining corpus, remained competitive on snow monitoring tasks when Sentinel-3 bands were mapped to Sentinel-2 wavelengths, suggesting spectral compatibility can outweigh native spatial resolution.
Model Scale: Base models generally outperform Tiny models, but the gap is task-dependent. Notably, in frozen regimes for SAR-heavy tasks (Sea Ice), Tiny models sometimes outperformed Base models, challenging the assumption that capacity always guarantees better transferability outside principal training domains.
Data Efficiency: In low-data regimes, THOR with fine-tuning and fine patch sizes (ps4) demonstrated significant data efficiency, achieving performance levels with 5% of the data that UNet required 50% to reach.

Significance and Claims
The paper claims that the resulting picture is not a single "winner" but a set of hypotheses and a diagnostic ablation methodology. The authors assert that:

Aggregate leaderboards are insufficient for understanding model behavior; performance must be analyzed through the lens of specific design axes and dataset characteristics.
The trade-off between pretraining investment (TerraMind) and inference-time adaptability (THOR) is a critical decision for practitioners, dependent on deployment constraints (training budget vs. inference latency) and sensor modalities.
The diagnostic methodology developed here is expected to generalize to future GFMs, providing a framework for attributing performance differences to specific, nameable design choices rather than treating models as black boxes.
The study highlights that "correctly interpreting results requires dataset-level characterisation," positioning this as a fundamental requirement for rigorous GFM benchmarking.

More like this