🤖 machine learning

Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors

This paper introduces DeluluNet, an end-to-end trained architecture that adapts remote sensing models to changing satellite modalities (substitution, addition, or subset) by leveraging modality hallucination to predict missing sensor representations from unlabeled data, thereby enabling model deployment without costly re-labeling or re-training.

Original authors: Tim G. Zhou, Anthony Fuller, Geoff Pleiss, Evan Shelhamer

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tim G. Zhou, Anthony Fuller, Geoff Pleiss, Evan Shelhamer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has spent years perfecting a recipe using a specific set of ingredients: fresh basil, tomatoes, and mozzarella. You know exactly how to make the perfect Caprese salad.

Now, imagine the grocery store changes. The basil is retired, and a new, exotic herb is introduced. Or maybe they stop selling tomatoes but start selling a new type of pepper. Or perhaps they have both the old tomatoes and the new peppers, but you've never cooked with them together before.

In the world of satellite imaging, this is exactly what happens. Satellites are like grocery stores that constantly update their "sensors" (the ingredients). One day, they send back clear photos (optical data); the next, they send back radar images (which see through clouds) or hundreds of different color bands.

The problem is that the "chefs" (AI models) trained on the old ingredients often fail when handed the new ones. Usually, to fix this, you'd have to hire a whole new team of chefs to taste-test and label thousands of new dishes (re-labeling data) and train a new model from scratch. This is expensive and slow.

Enter "DeluluNet" (a playful name from the paper, playing on the idea of "delusion" or "hallucination").

The Core Idea: "Hallucinating" the Missing Ingredients

The paper proposes a clever trick: Modality Hallucination.

Instead of waiting for the new ingredients to be labeled, DeluluNet learns to "imagine" or "hallucinate" what the missing ingredients would look like based on the ones it does have.

Here is how it works in three simple scenarios, using the paper's analogies:

1. Modality Transfer (The Swap)

The Situation: The old satellite (Satellite A) is retired, and a new one (Satellite B) takes its place. You have a model trained on A, but now you only have data from B.
The Analogy: You are used to cooking with Basil. Now, the store only has Rosemary. You don't have a recipe for Rosemary, but you have a huge pile of unlabelled photos showing both Basil and Rosemary growing side-by-side.
The Solution: DeluluNet looks at the Rosemary and says, "I've seen this next to Basil before. I can guess what the Basil would have looked like here." It uses that guess to update its recipe so it can cook with Rosemary alone, without needing a new human chef to taste-test it.

2. Modality Addition (The Mix)

The Situation: The old satellite is still working, but a new one joins the team. You want to use both to make better predictions.
The Analogy: You have your Basil, and now you also have a new, spicy Pepper. You want to make a salad that uses both.
The Solution: DeluluNet learns how the Basil and Pepper interact. It takes the unlabelled photos of them growing together and learns how to combine them into a super-salad. It doesn't need a human to label every single leaf; it figures out the relationship on its own.

3. Modality Peeking (The Sneak Peek)

The Situation: You still need to use the old satellite (Satellite A) for your daily work, but a new, high-tech satellite (Satellite B) is available. Satellite B is too expensive or rare to use for everything, but it has "super-vision" (like seeing fine details).
The Analogy: You are cooking with standard tomatoes, but you have a friend with a high-tech microscope that can see the tiny details inside a tomato. You can't use the microscope for every tomato, but you can look at a few.
The Solution: DeluluNet "peeks" at the high-tech data (Satellite B) while it has the standard data (Satellite A). It learns the secret patterns from the high-tech view and then applies that knowledge to improve its cooking with the standard tomatoes. It gets smarter without needing to switch to the expensive tool permanently.

How Does It Learn? (The "Magic" Training)

The paper explains that DeluluNet uses a special training method called Batch Mixing:

The Teacher: It starts with a smart model trained on the old data (the "Teacher").
The Masking Game: During training, the system sometimes hides (masks) one type of data. For example, it shows the model a radar image but hides the optical image.
The Guess: The model tries to "hallucinate" (predict) what the hidden image should look like based on the radar.
The Check: It compares its guess to the real hidden image (which it has in the training data, just not the labels). If the guess is close, it learns.
The Result: The model becomes so good at guessing the missing pieces that when it goes out into the real world and only gets one type of data, it can still "fill in the blanks" in its own mind and make accurate predictions.

Why Is This a Big Deal?

No New Labels Needed: Usually, to teach a model a new sensor, you need humans to label thousands of images. DeluluNet does this using unlabeled data (just raw images from two satellites), which is abundant and free.
One Model to Rule Them All: Instead of building a new model for every new satellite, you can take your existing model and "upgrade" it with these modular parts.
It Works: The paper tested this on real satellite data (Sentinel-1 and Sentinel-2). In many cases, DeluluNet performed better than models that were fully re-trained from scratch with expensive labeled data.

The Bottom Line

The world of satellites is constantly changing. Old sensors die, new ones are born, and data formats shift. DeluluNet is a flexible, "hallucinating" AI that allows us to adapt our existing models to these changes instantly, without the heavy cost of re-labeling the world. It's like teaching a chef to adapt to a changing grocery store by letting them practice guessing the missing ingredients, rather than hiring a new chef every time the menu changes.

Technical Summary: Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors

Problem Statement

Remote sensing (RS) models are typically trained and deployed on static sets of modalities (e.g., specific satellite sensors or spectral bands). However, the Earth observation landscape is dynamic: satellites are launched with novel sensors, old ones are retired, and data availability shifts (e.g., free data becoming paywalled). This creates a critical challenge: existing models fail when presented with modalities unseen during training. For instance, a model trained on Sentinel-2's 12-band imagery cannot predict on a new hyperspectral sensor's hundreds of bands.

The paper identifies three specific scenarios where models must adapt to these changing modalities without the luxury of re-labeling data or training from scratch due to computational and annotation constraints:

Modality Transfer: Switching a model from an old modality ( $M_A$ ) to a new one ( $M_B$ ) while maintaining predictive ability (e.g., moving from LandSAT to Sentinel-2).
Modality Addition: Incorporating a new modality ( $M_B$ ) into predictions alongside the existing one ( $M_A$ ) to leverage complementary information.
Modality Peeking: Improving predictions on an old modality ( $M_A$ ) by observing a new, unlabeled modality ( $M_B$ ) during adaptation, then reverting to $M_A$ for inference.

Existing literature addresses these scenarios in isolation (e.g., Knowledge Distillation for transfer, Semi-Supervised Learning for peeking, Multimodal Knowledge Expansion for addition) or assumes fixed modalities (Continual Learning, Domain Adaptation). No unified framework exists to handle all three scenarios in a label-free setting using unlabeled multi-modal data.

Methodology: DeluluNet

The authors propose DeluluNet, a unified architecture designed to adapt an existing unimodal predictive model ( $f_0$ ) to changing modalities using unlabeled paired data ( $D_a \subset M_A \times M_B$ ). The core concept is Modality Hallucination: predicting missing modality representations from those that are present.

Architecture

DeluluNet augments a pre-trained unimodal teacher ( $f_0$ , typically a ViT) with three modular components:

Modality-Specific Extraction:
- $M_A$ components are initialized from the weights of $f_0$ .
- $M_B$ components (tokenizer and transformer layers) are randomly initialized to learn features specific to the new modality.
Cross-Modality Fusion via Masking Transformers:
- Inspired by masked image modeling, this component facilitates learning between modalities.
- It employs token-level masking (randomly masking tokens across modalities) and modality-level masking (treating an entire modality as absent).
- A Masking Transformer uses cross-attention to "hallucinate" (impute) masked tokens or entire modalities based on visible tokens from the other modality. A learned query attends to visible tokens from the other modality to predict the missing ones.
Predictor Head:
- A task-specific head ( $Head_{M_B}$ ) that predicts from $M_B$ representations.

Training Strategy

DeluluNet is trained end-to-end using a batch mixing strategy that alternates between two data splits:

Unlabeled Adaptation Split ( $D_a$ ): Contains paired $M_A$ and $M_B$ data. The model applies masking (token and modality level) and uses Distillation Loss. The model learns to predict from the hallucinated representations, distilling knowledge from the unimodal teacher $f_0$ .
Labeled Split ( $D_l$ ): Contains labeled $M_A$ data. The model treats $M_B$ as masked (hallucinated) and applies Cross-Entropy Loss against real labels. This allows the model to learn from real labels using pseudo-multi-modal inputs.

Loss Functions

The training objective combines four loss terms:

Distillation Loss: On unlabeled batches, aligning predictions with the teacher $f_0$ .
Cross-Entropy Loss: On labeled batches, aligning predictions with ground truth.
Pre-fusion MSE: Supervises the masking transformer directly, encouraging predicted CLS and patch tokens to match the true tokens of the masked modality.
Latent MSE: Anchors the fused representation to the teacher's frozen penultimate representations via a decoder.

Key Results

The authors evaluated DeluluNet on three datasets (EuroSAT, reBEN, DFC2020) covering cross-band changes (within Sentinel-2) and cross-satellite changes (Sentinel-1 to Sentinel-2).

Modality Transfer: DeluluNet consistently outperformed label-free baselines like Knowledge Distillation (KD) and Transformed Teacher Matching (TTM). Notably, in several transfer scenarios (e.g., S2-rgb to S2-non-rgb), DeluluNet surpassed fully supervised fine-tuned DINOv3 models, achieving performance close to oracle multi-modal foundation models (RSFMs) without requiring labels on the target modality.
Modality Addition: DeluluNet improved performance over the unimodal baseline $f_0$ and outperformed Multimodal Knowledge Expansion (MKE) baselines. In some cases, it matched or exceeded supervised RSFM oracles (e.g., Panopticon, OlmoEarth) trained on labeled multi-modal data.
Modality Peeking: DeluluNet consistently improved predictions on the original modality ( $M_A$ ) by leveraging unlabeled observations of $M_B$ , outperforming Semi-Supervised Learning baselines like MixMatch.
Ablation Studies: Removing any component (Masking Transformers, Pre-fusion MSE, Latent MSE, or Batch Mixing) resulted in performance degradation across all scenarios, confirming the necessity of the hallucination mechanism and the specific loss terms.
Computational Cost: While DeluluNet is more computationally expensive than unimodal models due to hallucination, its cost is comparable to existing multi-modal RSFMs (like Panopticon) when handling multi-modal inputs.

Significance and Claims

The paper claims that DeluluNet provides a practical, label-free alternative to re-labeling and re-training in a changing world. Its primary significance lies in:

Unified Framework: It is the first method to jointly address transfer, addition, and peeking scenarios under a single architecture.
Sustainability: It offers a path for adapting existing models to new sensors without the unsustainable cost of pre-training foundation models from scratch or collecting massive labeled datasets for every new sensor.
Immediate Deployability: Practitioners can augment existing fine-tuned models with modular components using widely available unlabeled paired satellite data, rather than waiting for new foundation models to be developed.

The authors acknowledge limitations, noting that the current work assumes static land-use tasks and paired co-located observations. They highlight that temporal misalignment (e.g., in disaster monitoring) and the need for continual self-supervised learning of the foundation model itself are important directions for future work. The paper concludes by calling for long-term benchmarks that cover diverse modalities to support the sustainable evolution of remote sensing models alongside satellite constellations.