Technical Summary: Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classification

1. Problem Definition

Multimodal remote sensing classification relies on complementary data sources (e.g., optical, SAR, hyperspectral, LiDAR) to achieve robust Earth observation. However, real-world scenarios frequently suffer from missing modalities due to sensor failures, atmospheric interference, or budget constraints. This incompleteness leads to severe performance degradation in conventional multimodal models, which typically assume full-modality availability during both training and inference.

Existing solutions generally follow a two-stage paradigm: training a model on complete data and then adapting it to missing scenarios via additional modules or retraining. This approach suffers from two primary limitations:

1. Computational Overhead: Repeated retraining or adaptation for multiple missing-modality patterns is inefficient.
2. Generalization Gap: Models trained on complete data fail to account for the inherent incompleteness of real-world observations, limiting their ability to handle unseen modality-missing conditions.

While prompt-based parameter-efficient fine-tuning (PEFT) offers a lightweight alternative, it often requires exponential configuration growth as the number of modalities increases, hindering scalability.

2. Methodology: MaMOL

The authors propose Missing-aware Mixture-of-LoRAs (MaMOL), a parameter-efficient framework that reframes missing-modality learning as a conditional computation problem. Instead of treating missing modalities as a data reconstruction task, MaMOL utilizes a Mixture-of-Experts (MoE) architecture to dynamically select computation paths based on available inputs.

Core Architecture

MaMOL is built upon a frozen pretrained Vision Transformer (ViT) backbone. It injects low-rank adaptation (LoRA) experts into the feed-forward layers, decomposing the transformation into three structured components:
$$f(z, m) = f_{shr}(z) + f_{mod}(z, m) + f_{pat}(z, m)$$
Where $z$ is the input representation and $m$ is the modality availability indicator.

The framework employs a dual-routing mechanism consisting of two complementary expert categories:

1. Static Experts (Modality-Invariant & Specific):

   • Shared Expert: Always activated to encode modality-invariant knowledge common across all scenarios, ensuring global stability.
   • Modality-Specific Experts: Activated only when their corresponding modality is present ($\alpha_i = m_i$). These preserve modality-dependent priors and stabilize feature alignment when specific sensors are available.

2. Dynamic Experts (Pattern-Aware):

   • A set of $K_d$ experts that adaptively specialize in different missing-pattern configurations.
   • Routing Mechanism: Unlike standard MoE, the router explicitly incorporates both the semantic content of the input ($z$) and the missing-modality configuration ($\psi(m)$). It selects the top-$K$ experts via a gating function:
     $$g = \text{TopK}(\text{softmax}(W \phi([z; \psi(m)])))$$
   • This allows the model to learn specific adaptations for missing patterns (e.g., "missing HS only" vs. "missing LiDAR only") without retraining.

Parameter Efficiency

By utilizing LoRA (Low-Rank Adaptation), MaMOL introduces minimal trainable parameters. The experts are decomposed into low-rank matrices ($A \in \mathbb{R}^{r \times d}, B \in \mathbb{R}^{d \times r}$ with $r \ll d$), enabling the unification of $2^M - 1$ possible modality combinations within a single model structure.

3. Key Contributions

1. Reformulation of the Problem: The paper reframes missing-modality learning in remote sensing as a conditional expert selection problem, providing the first systematic study of MoE paradigms under diverse missing-modality settings.
2. MaMOL Framework: Introduction of a dual-routing, parameter-efficient MoE framework that unifies multiple missing-modality scenarios. It decouples modality-invariant shared knowledge from modality-aware dynamic adaptations.
3. Empirical Validation: Extensive experiments demonstrate that MaMOL significantly improves robustness and generalization under diverse missing-modality scenarios with minimal computational overhead. The framework is shown to scale to natural image tasks, validating its cross-domain applicability.

4. Experimental Results

The authors evaluated MaMOL on three public multimodal remote sensing datasets: Houston2013 (HS + LiDAR), Trento (HS + LiDAR), and Augsburg (HS + SAR).

• Performance: MaMOL consistently outperformed state-of-the-art baselines (including MMP, DCP, and various MoE variants like Moerep, Moeada, Moetask) across all missing rates (50%, 70%, 90%) and missing configurations.
  • On the Houston2013 dataset with 50% missing rates, MaMOL achieved 98.40% OA (vs. 97.60% for the next best), and 88.00% OA in the 50% train/100% test split scenario, significantly outperforming baselines that dropped to ~83-85%.
  • On Augsburg, where modality discrepancies are large, MaMOL showed clear advantages, achieving 79.05% OA compared to 78.57% for the best baseline under 50% missing rates.
• Generalization: Transfer experiments on the MM-IMDb natural image dataset confirmed that MaMOL generalizes beyond remote sensing, achieving the highest F1-Macro scores across all train/test splits.
• Ablation Studies:
  • Removing Dynamic Experts caused a substantial performance drop, confirming their role in adapting to missing patterns.
  • Removing Static Experts reduced stability, highlighting their importance in preserving invariant knowledge.
  • Placing experts in the last six Transformer layers yielded the best results, suggesting that dynamic routing is most effective for modeling high-level cross-modal interactions.

5. Significance and Claims

The paper claims that MaMOL offers a lightweight, robust, and extensible solution for real-world multimodal incompleteness. By treating missing modalities as a conditional computation problem rather than a data reconstruction issue, the authors demonstrate that:

• MoE architectures can inherently adapt to diverse modality-missing scenarios without the need for separate retraining or exponential prompt configurations.
• The decoupling of shared and dynamic experts allows for stable knowledge transfer while maintaining flexibility for specific missing patterns.
• The approach is scalable, capable of handling complex multi-modality combinations (up to three modalities in experiments) and transferring to natural image domains.

The work positions MaMOL as a step toward building missing-aware and generalizable multimodal learning systems, addressing the fundamental challenge of reliability in sensor-constrained remote sensing applications.