💻 computer science

CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts

The paper introduces CSMoE, an efficient remote sensing foundation model that combines a Soft Mixture-of-Experts architecture with a thematic-climatic sampling strategy to achieve state-of-the-art performance across diverse tasks while significantly reducing computational complexity and pretraining costs.

Original authors: Leonard Hackel, Tom Burgert, Begüm Demir

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leonard Hackel, Tom Burgert, Begüm Demir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Earth is constantly watched from above by a vast network of satellites, capturing images of forests, cities, deserts, and oceans in a continuous stream of data. For decades, scientists have relied on artificial intelligence to make sense of this deluge, teaching computers to recognize patterns like crop types, flood zones, or urban sprawl. However, a new challenge has emerged: the models powerful enough to understand this complex global view have become so massive that they are difficult to run, requiring immense computing power and time. These "foundation models" are designed to learn general knowledge from unlabeled images, much like a child learns to recognize objects before being taught specific names, but their sheer size often makes them impractical for everyday use. Researchers are now asking a critical question: can we build models that are just as smart but far more efficient, capable of running on standard hardware without sacrificing their ability to see the world clearly?

A team of researchers has answered this by developing a new type of foundation model called CSMoE, which achieves high performance while using significantly less computing power than current state-of-the-art systems. The core of their innovation lies in a technique called a "soft mixture of experts." Imagine a large team of specialists, each with a unique skill set, working together to solve a problem. In traditional large models, every single piece of information is processed by the entire team, which is slow and energy-intensive. The new model, however, uses a smart routing system that gently directs each piece of information to only the few specialists best suited to handle it, while still allowing them to share insights. This approach, known as a soft mixture, allows the model to maintain a high level of intelligence and understanding without the heavy computational cost of activating every part of its brain for every single image. By integrating this mechanism into a system designed to learn from multiple types of satellite sensors at once, the researchers created a model that is both versatile and lightweight.

The researchers did not stop at improving the model's architecture; they also addressed the inefficiency of the data used to train it. Training these massive systems usually involves feeding them billions of images, many of which are nearly identical, such as endless stretches of ocean or uniform desert sand. This redundancy wastes time and resources without teaching the model anything new. To solve this, the team created a new method for selecting training images based on their climate and land-cover characteristics. Instead of grabbing images randomly, their system ensures that the training set includes a balanced mix of different environments, from tropical rainforests to arid deserts, while removing duplicate or overly similar pictures. This strategy, which uses a genetic algorithm to maximize the diversity of the selected images, allows the model to learn from a much smaller, more representative dataset, drastically cutting down the time and energy required for training.

When tested on a variety of real-world tasks, the new model proved its worth. In experiments involving the classification of land use, such as identifying whether an area is a city, a forest, or a farm, the model performed as well as the largest and most powerful existing systems. It was also tested on the more difficult task of semantic segmentation, where the computer must draw precise boundaries around objects like individual trees or buildings in an image. Here, the model again matched or exceeded the performance of its much larger competitors. Perhaps most impressively, it demonstrated strong capabilities in content-based image retrieval, a task where a user provides an image and the system must find similar images from a massive archive, even if the images come from different types of sensors. In these tests, the new model achieved results comparable to the best systems available, but it required up to ten times fewer computational operations to do so.

The study also explored how different design choices affected the model's performance, revealing that smaller image patches used during processing led to better results in detailed tasks like segmentation, though at a slightly higher computational cost. The researchers found that the model's internal "experts" did not specialize in specific types of land cover as one might expect; instead, they worked together as a flexible, unified team, with the routing system distributing the workload evenly. This suggests that the efficiency gains come from the architecture itself rather than from rigid specialization. The findings indicate that it is possible to build remote sensing models that are both powerful and practical, capable of running on standard hardware while delivering the high-quality analysis needed for disaster response, environmental monitoring, and urban planning. By combining a smarter internal structure with a more efficient way of learning from data, this work offers a clear path forward for making advanced artificial intelligence accessible for observing and understanding our planet.

Technical Summary: CSMoE – An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts

1. Problem Statement

The development of Remote Sensing (RS) Foundation Models (FMs) has been driven by self-supervised learning (SSL) and the availability of large-scale Earth Observation (EO) data. However, existing RS FMs face two primary limitations that hinder their practical applicability:

Computational Inefficiency: Current models often require substantial computational resources (measured in Floating-Point Operations or FLOPs) for both training and inference. While some models achieve high representational capacity through massive parameter counts (e.g., Prithvi V2-600 with 631M parameters), their inference costs exceed 160 billion FLOPs per image. Conversely, models with lower computational budgets often suffer from limited representational capacity.
Training-Data Redundancy: Pretraining archives for RS FMs frequently contain redundant images (e.g., vast homogeneous regions like deserts or forests under the same climate zone). This redundancy increases training costs and environmental impact without significantly improving the learned representations.

These issues are particularly critical for tasks like Content-Based Image Retrieval (CBIR), where inference must be performed on every image in a large archive, and for pretraining, where the size and redundancy of the dataset directly drive computational costs.

2. Methodology

The authors propose CSMoE (Cross-Sensor Mixture-of-Experts), a computationally efficient RS foundation model that addresses the above limitations along two axes: Model Efficiency and Training-Data Efficiency.

A. Model Efficiency: Soft Mixture-of-Experts (Soft MoE)

The core architectural innovation is the integration of the Soft MoE mechanism into the Cross-Sensor Masked Autoencoder (CSMAE) baseline. Unlike sparse MoEs that route tokens to a single top- $k$ expert, Soft MoE employs a two-stage routing mechanism:

Slot-Based Routing: Input tokens are softly routed to $S$ intermediate "slots" rather than directly to experts. Each slot summarizes a subset of input features via weighted aggregation.
Expert Processing: Each slot is processed by a single assigned expert.
Reconstruction: The final output is reconstructed through weighted aggregation of the expert outputs.

This approach reduces the number of expert calls from $k \cdot P$ (where $P$ is the number of tokens) to $S$ , significantly lowering computational complexity while maintaining high representational capacity.

Architecture Adaptations in CSMoE:

Sensor-Specific Encoders: Each image modality (e.g., Sentinel-1 SAR, Sentinel-2 Optical) utilizes its own encoder with dedicated Soft MoE layers and separate expert networks to model sensor-specific characteristics.
Cross-Sensor Encoder: A shared encoder with shared Soft MoE layers processes features from all modalities to facilitate inter-modal pattern learning.
Regularization: To ensure stable training and balanced expert usage, the authors introduce two additional loss functions:
- Slot Repulsion Loss ( $L_{REP}$ ): Encourages reduced correlation between slot embeddings.
- Entropy Loss ( $L_{ENT}$ ): Promotes balanced usage of experts.

B. Training-Data Efficiency: Thematic-Climatic Descriptor-Driven Sampling

To reduce pretraining costs and redundancy, the authors introduce a sampling strategy that constructs a reduced, representative training set from a large-scale archive (specifically the Major TOM Core dataset). The strategy involves two stages:

Auxiliary Descriptor Generation: Each image is assigned a climatic descriptor (derived from Köppen–Geiger climate zones) and a thematic descriptor (derived from land-cover products like ESA WorldCover).
Entropy-Maximizing Stratified Sampling:
- The dataset is stratified by the combination of climatic and thematic descriptors.
- Within each stratum, a genetic algorithm is employed to select a subset of samples that maximizes spatial dispersion (geodesic distance) while maintaining a target sample count.
- This process discards semantically similar samples (redundancy) while preserving the geographic and thematic-climatic diversity of the original archive.

3. Key Contributions

The paper claims the following contributions:

First Computationally Efficient Multi-Modal FM: CSMoE is presented as the first RS foundation model to inject Soft MoE into a multi-modal architecture (CSMAE), achieving increased representational capacity while significantly reducing computational complexity during training and inference compared to models of similar size.
Efficient Sampling Strategy: The introduction of a thematic-climatic descriptor-driven sampling strategy that constructs a reduced training set from large-scale archives, retaining diversity while minimizing redundancy and pretraining costs.
Comprehensive Evaluation: Extensive experiments across scene classification, semantic segmentation, and uni-modal/cross-modal CBIR tasks, demonstrating that CSMoE achieves competitive or superior downstream performance with substantially fewer FLOPs than state-of-the-art FMs.

4. Experimental Results

The authors evaluated CSMoE on multiple benchmarks (e.g., m-bigearthnet, m-eurosat, m-cashew-plant, BigEarthNet-v2) and compared it against models like Prithvi V2, Satlas, DOFA, TerraMind, and the baseline CSMAE.

Performance vs. Efficiency:
- On m-bigearthnet, CSMoE variants achieved micro-mean average precision (mAP $\mu$ ) comparable to Prithvi V2-600 but required an order of magnitude fewer FLOPs.
- On semantic segmentation (m-cashew-plant), the CSMoE variant with patch size $\rho=14$ achieved an IoU of 59.4%, outperforming TerraMind (58.8%) and Prithvi V2-300 (58.5%) while using significantly fewer FLOPs (13.40G vs. 17.84G and 59.85G).
- In CBIR, CSMoE variants generally surpassed the baseline CSMAE in uni-modal retrieval tasks. In cross-modal tasks, performance varied by patch size, but lightweight variants (e.g., $\rho=32$ ) remained competitive while using 35–48% fewer FLOPs than CSMAE.
Sensitivity Analysis:
- Patch Size: Smaller patch sizes ( $\rho=14, 16$ ) generally yielded better downstream performance, particularly for segmentation, though at the cost of higher FLOPs. $\rho=16$ was identified as offering the best trade-off between efficiency and performance.
- Sampling Strategy: The entropy-maximizing stratified sampling outperformed random stratified sampling on five of six downstream tasks, confirming the benefit of reducing redundancy while preserving diversity.
- Routing Behavior: Analysis of expert routing revealed that experts operate as a soft ensemble with near-uniform routing weights and near-orthogonal parameters, rather than as semantically specialized units. The performance gains are attributed to the architectural integration and soft-mixing capacity rather than discrete expert specialization.

5. Significance and Claims

The paper positions CSMoE as a significant step toward making RS foundation models more practical for operational deployment. By decoupling representational capacity from computational cost through Soft MoE and reducing data redundancy through intelligent sampling, the authors demonstrate that high-performance RS models do not necessarily require massive computational budgets.

The authors modestly claim that:

CSMoE is the first computationally efficient FM in RS to integrate Soft MoE.
The proposed sampling strategy effectively reduces pretraining costs without sacrificing the diversity required for generalization.
The model achieves competitive or superior performance compared to state-of-the-art models while requiring substantially fewer FLOPs.

The work suggests that future RS FM development should prioritize efficiency-oriented architectures and data curation strategies alongside scaling, noting that the proposed Soft MoE adaptation is generalizable to other transformer-based FMs and learning objectives beyond masked image modeling.