🤖 machine learning

On What Depends the Robustness of Multi-source Models to Missing Data in Earth Observation?

This study evaluates six state-of-the-art multi-source Earth Observation models to reveal that their robustness to missing data depends on task nature, source complementarity, and model design, while challenging the assumption that using all available data always improves performance.

Original authors: Francisco Mena, Diego Arenas, Miro Miranda, Andreas Dengel

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Francisco Mena, Diego Arenas, Miro Miranda, Andreas Dengel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to identify a specific type of tree in a forest. To do this, you have a team of four experts:

The Optical Expert: Takes photos of the leaves (but gets blinded by clouds).
The Radar Expert: Uses sound waves to see through the fog (but sometimes the sensor breaks).
The Weather Expert: Knows the temperature and rain history.
The Map Expert: Knows the shape of the hills and valleys.

In the world of Earth Observation (EO), scientists build "super-models" that listen to all four experts at once to make the best guess possible. But what happens if the Radar Expert goes on vacation, or the Weather Expert loses their notebook? Does the whole team fail, or can they still guess correctly?

This paper is like a stress test for these teams. The researchers built six different types of "super-teams" (models) and asked: "How well do you perform if one of your experts suddenly disappears?"

Here is what they found, explained simply:

1. The "Missing Person" Problem

In the real world, data often goes missing. Clouds hide satellite photos, sensors break, or satellites stop working. The researchers tested what happens when they pretend one expert is gone during the "exam" (prediction).

They discovered that not all teams handle missing members the same way. Some teams crumble when the Optical Expert leaves; others barely notice.

2. The Three Rules of a Strong Team

The paper found that a model's ability to survive without a data source depends on three main things:

The Job Matters (The Task):
Sometimes, an expert is crucial for one job but useless for another.
- Analogy: If you are trying to guess what crop is growing (e.g., wheat vs. corn), the Weather Expert is very helpful because different crops need different rain. But if you are just trying to guess if there is a crop at all (yes/no), the Weather Expert matters less.
- The Finding: A model that is great at identifying specific crops might fail if the weather data is missing, while a model just checking for "crop presence" might not care.
The Team Chemistry (Complementarity):
This is about how well the experts fill in each other's gaps.
- Analogy: Imagine the Map Expert and the Optical Expert. If the Map Expert knows the terrain is steep, and the Optical Expert sees green, they might guess "forest." But if the Map Expert is missing, the Optical Expert might get confused by shadows.
- The Surprise: Sometimes, an expert who is terrible at working alone (low individual score) is actually the most important to have on the team because they provide unique information the others don't have. Conversely, sometimes an expert who is great alone is actually redundant when the whole team is present.
The Team Rules (Model Design):
How the team is organized matters.
- Some teams are built to "ignore" missing members automatically (like a committee that just votes on who is present).
- Other teams are built to try and "reconstruct" the missing person's opinion based on what the others said.
- The Finding: The paper found that no single team design wins every time. Some models are super-robust when a source is missing, but others are better when you only have one source left.

3. The Big Surprise: Less is Sometimes More

The most shocking discovery was that having all the data isn't always better.

Analogy: Imagine a cooking competition. You have a recipe that uses salt, pepper, garlic, and sugar. But what if the sugar actually ruins the dish? If you remove the sugar, the dish tastes better.
The Finding: In some cases, the researchers found that removing a specific data source (like the weather data) actually made the model's predictions more accurate. The extra data was just adding "noise" or confusion. This challenges the idea that "more data is always good."

4. The Takeaway for the Future

The paper concludes that we can't just throw all our data into a "black box" and hope for the best.

One size does not fit all: A model that is robust for one type of missing data might fail miserably for another.
Specialization is key: If you know you will only ever have one type of data (e.g., only radar, no photos), it's better to train a model specifically for that scenario rather than a general "all-data" model.
Quality over Quantity: Sometimes, carefully selecting which data sources to keep is more important than collecting everything.

In short, building a robust Earth Observation model is like building a sports team. You don't just want the players with the highest individual stats; you need players who fit together, you need to know which position is most critical for the specific game you are playing, and sometimes, you need to realize that a smaller, more focused team actually wins the game.

Technical Summary: On What Depends the Robustness of Multi-source Models to Missing Data in Earth Observation?

Problem Statement
In Earth Observation (EO), multi-source models leverage diverse data streams (e.g., optical, radar, weather, topography) to enhance predictive accuracy. However, these models often suffer from a critical vulnerability: they typically assume the availability of all data sources during inference. In real-world scenarios, data loss is inherent due to factors such as cloud occlusion, sensor failures (e.g., Landsat 7 SLC-off), or satellite operational halts. While recent literature has introduced robust multi-source models designed to handle missing data, the specific factors determining why some models outperform others in varying missing-data scenarios remain poorly understood. Existing studies have not sufficiently assessed the interplay between task nature, data source complementarity, and model architecture in the context of source-wise missing data.

Methodology
The authors conducted a comprehensive evaluation of six state-of-the-art multi-source models across three distinct EO classification tasks involving both static (single-date) and temporal data sources.

Datasets:
- CropH-b & CropH-m: Binary and multi-class crop recognition tasks using four sources: multi-spectral optical Satellite Image Time Series (SITS), radar SITS, weather time series, and topographic information.
- TreeSatAI-TS: Multi-label tree species classification using optical SITS, radar SITS, and high-resolution single-date aerial imagery.
Models Evaluated:
- ISensD: Input-level fusion using sensor dropout (replacing missing features with zeros).
- FSensD: Feature-level fusion with cross-attention and sensor dropout (masking missing features).
- OOD-f: Feature-level fusion with concatenation, simulating all source-wise missing cases.
- TIMML: Feature-level fusion with cross-attention, sensor dropout, and auxiliary prediction losses per source.
- ESensI: Ensemble-based model using shared prediction heads in per-source layers.
- Co-Ens: Ensemble-based model using similarity constraints in per-source predictions.
Experimental Design:
- Models were trained on full-data scenarios.
- Validation was performed under two primary missing-data conditions: (1) Single source missing (one source removed while others remain), and (2) Single source available (only one source provided for prediction).
- Performance was measured using the Weighted F1 score.
- A complementarity score was calculated to quantify the relative drop in performance when a specific source is missing, defined as $(P_A - P_{m(s)}) / P_{m(s)}$ , where $P_A$ is performance with all sources and $P_{m(s)}$ is performance with source $s$ missing.

Key Results
The evaluation revealed that the robustness of multi-source models is not uniform but depends on three intricate factors:

Predictive Task Nature: The utility of a data source varies by task. For instance, weather time series proved more effective for binary crop classification than for multi-class crop-type identification, whereas radar SITS was more discriminative for distinguishing specific crop types.
Data Source Complementarity: High individual performance of a data source does not guarantee high complementarity. In some cases (e.g., OOD-f model on CropH-b), weather data showed higher complementarity scores than optical data despite lower single-source performance. Conversely, in tree classification, high-resolution aerial data provided the highest complementarity despite not being the top single-source performer.
Model Design:
- Missing Source Scenarios: Models like OOD-f and TIMML demonstrated superior robustness when a single data source was missing.
- Single Source Scenarios: Models like ESensI and Co-Ens performed better when only a single data source was available for prediction.
- Counter-intuitive Findings: The study observed instances where removing a data source led to improved predictive performance. For example, in the TreeSatAI-TS dataset, using only radar SITS with ESensI and Co-Ens models outperformed the full-data configuration. Similarly, weather data degraded the performance of TIMML, ESensI, and Co-Ens in the CropH-m multi-class task.

Significance and Claims
The paper claims to provide a structured understanding of the factors governing the robustness of multi-source models in EO. The authors emphasize that:

Robustness is context-dependent: No single model architecture is universally effective for all missing-data scenarios; the optimal choice depends on the specific task and data characteristics.
More data is not always better: The assumption that incorporating all available data sources is always beneficial is challenged. The study demonstrates that specific data sources can introduce noise or redundancy, leading to better performance when excluded.
Specialization is necessary: If a deployment scenario expects only a single data source, training a model specialized for that specific scenario is more effective than relying on a general robust multi-source model, which may be unreliable in single-source conditions.

The authors conclude that these findings suggest a need for more streamlined approaches in EO applications, potentially involving careful data source selection based on missing data analysis, akin to feature selection methods in Explainable AI (XAI).

1. The "Missing Person" Problem

2. The Three Rules of a Strong Team

3. The Big Surprise: Less is Sometimes More

4. The Takeaway for the Future

More like this