Technical Summary: Design Choices That Matter in Remote Sensing Multi-Label Classification

Problem Statement

Benchmarking deep learning (DL) models for multi-label classification (MLC) of remote sensing images (RSI) typically produces rankings of model performance. However, these rankings often fail to generalize beyond the specific datasets on which they were evaluated. The performance of DL models is influenced by a complex interplay of factors, including network architecture, training strategies (e.g., fine-tuning vs. pre-trained), and initialization. Furthermore, intrinsic dataset properties—such as scale, spatial resolution, and label space complexity—significantly affect which design choices are effective. Current benchmarking approaches, which rely on simple performance rankings, do not provide actionable insights into why certain choices work for specific datasets or how these choices interact. There is a need to move beyond static rankings toward a systematic understanding of how design choices and their interactions contribute to performance variability across different data regimes.

Methodology

The authors employ Functional Analysis of Variance (fANOVA) to decompose the variance in model performance into contributions from individual design choices and their interactions. The study utilizes an extended fANOVA framework applied to two distinct benchmarking scenarios across seven RSI datasets.

Experimental Scenarios

1. Scenario 1 (End-to-End vs. Feature Extraction): Based on Stoimchev et al. [9], this scenario evaluates 48 DL models. It investigates whether CNNs used as end-to-end predictors outperform those used as feature extractors combined with tree-based models (Random Forest, Extra Trees). The design choices analyzed are:
   • Network Architecture: VGG, ResNet, EfficientNet variants.
   • Fine-tuning Strategy: Pre-trained (frozen layers) vs. Fine-tuned (all layers updated).
   • Learning Strategy: End-to-end vs. Feature extraction with downstream classifiers.
2. Scenario 2 (Architecture and Initialization): Based on Dimitrovski et al. [8], this scenario evaluates 20 DL models, including CNNs and Transformers (ViT, SwinT, etc.). The design choices are:
   • Network Architecture: AlexNet, VGG, ResNet, DenseNet, EfficientNet, ViT, MLP-Mixer, ConvNeXt, SwinT.
   • Initialization Strategy: From scratch vs. Pre-trained.

Analysis Pipeline

1. Dataset Meta-Representations: For each dataset, fANOVA is applied to the performance scores of the evaluated models. This generates a meta-representation vector capturing the importance of main effects (individual modules) and interaction effects (pairwise and higher-order).
2. Clustering: Hierarchical clustering is applied to these meta-representations to group datasets based on their "design-choice sensitivity profiles" rather than raw performance levels.
3. Correlation with Meta-Features: The resulting clusters are analyzed against intrinsic dataset meta-features (e.g., number of samples, spatial resolution, label cardinality) to identify patterns.

Key Contributions

The paper makes four primary contributions:

1. Quantification of Design Choice Impact: It applies fANOVA to two MLC benchmarking scenarios (48 and 20 models) across seven RSI datasets, quantifying the specific contribution of individual design choices and their interactions to performance variability.
2. Dataset Meta-Representations: It constructs meta-representations from fANOVA importance scores. Hierarchical clustering of these representations reveals that datasets naturally group by their sensitivity to design choices, rather than by overall performance magnitude.
3. Identification of Performance Regimes: The study demonstrates that the dominant factor governing performance shifts based on the dataset regime:
   • Large-scale: Driven by fine-tuning strategy and architecture.
   • Data-limited: Governed by initialization strategy.
   • Intermediate: Characterized by interactions between architecture and learning/initialization strategies.
4. Reframing Benchmarking Conclusions: It reinterprets conclusions from established studies [8, 9] as dataset-conditional rather than universal, providing dataset-aware guidelines for model selection.

Results

The analysis reveals distinct patterns in how design choices influence performance across different dataset scales:

• Raw Performance Variability: Datasets like Ankara and AID showed tight performance distributions (low sensitivity to design choices), while datasets like BigEarthNet and MLRSNet showed wide distributions (high sensitivity).
• Variance Decomposition:
  • In Scenario 1, individual effects dominated, with single choices explaining over 80% of variance in some cases. Pairwise interactions were substantial but lower; three-way interactions were negligible.
  • In Scenario 2, individual effects also dominated, but pairwise interactions remained non-negligible (8–20%), indicating that the combination of architecture and initialization significantly impacts performance.
• Clustering and Regime Analysis:
  • Cluster 1 (Large-scale: BigEarthNet-19/43, MLRSNet): Performance is primarily driven by the fine-tuning strategy (Scenario 1) and architecture choice (Scenario 2). Adapting pre-trained weights is crucial, and fine-tuning improves performance uniformly across architectures in this regime.
  • Cluster 2 (Data-limited: Ankara, UCM, AID): In small datasets, initialization (pre-training) is decisive. Learning strong representations from scratch is difficult; pre-training significantly reduces overfitting and improves generalization. In Scenario 1, architecture becomes more influential here, with simpler models (e.g., VGG) sometimes outperforming deeper ones on small, low-resolution data.
  • Cluster 3 (Intermediate: DFC-15, PlanetUAS): Both architecture and initialization/learning strategies are important, and their interactions become non-negligible. For instance, certain architectures (e.g., ConvNeXt, VGG16) benefit more from pre-training than others (e.g., ResNet152), requiring careful joint configuration.

Significance and Claims

The paper claims that its findings provide actionable, dataset-aware insights that go beyond traditional benchmarking rankings. By identifying that design choice importance is not universal but depends on intrinsic dataset properties (scale, resolution, complexity), the study argues that:

• Benchmarking conclusions should be reframed as conditional on the dataset regime.
• Practitioners should prioritize fine-tuning and architectural capacity for large-scale datasets.
• For data-limited regimes, the choice of initialization (pre-training) is the most critical factor.
• For intermediate regimes, the interaction between modules must be considered.

The authors note limitations, including the restriction to CNNs and early Transformers, the exclusion of recent visual foundation models, and the reliance on fixed training elements (e.g., augmentation, optimizers) from prior studies. They suggest future work should focus on predicting sensitivity profiles directly from dataset meta-features and extending the analysis to modern foundation models.