Technical Summary: A Comparative Study of Transformer and Convolutional Models for Crop Segmentation from SITS

Problem Statement

Crop segmentation from Satellite Image Time Series (SITS) is a critical task for agricultural monitoring and land-use analysis. While Convolutional Neural Networks (CNNs) have long been the standard for learning hierarchical spatial representations from multispectral imagery, the integration of temporal data remains a challenge. Specifically, it is unclear how self-attention-based transformer models compare to CNNs when spatial and temporal information must be modeled jointly. Furthermore, existing transformer architectures vary significantly in their strategies for handling the temporal dimension—some treating time as an additional spatial dimension, while others explicitly factorizing temporal and spatial dependencies. This paper addresses the gap in understanding which architectural strategies yield the best performance for crop mapping using Sentinel-2 time series.

Methodology

The study conducts a comparative evaluation of three transformer-based architectures against established 3D CNN baselines. The models are trained and tested on two public datasets: the Munich dataset (48×48 patches, 13 spectral bands, 18 crop classes) and the Lombardia dataset (48×48 patches, 9 spectral bands, 7 crop classes, spanning 2016–2019).

Architectures Evaluated

1. CNN Baselines:

   • 3D U-Net: A strong baseline treating the time series as a 3D volume.
   • 3D FPN (Feature Pyramid Network): Utilizing feature pyramids for multi-scale detection.
   • 3D DeepLabv3: Employing atrous convolutions for receptive field expansion.

2. Transformer Architectures:

   • Swin UNETR (Adapted): Originally designed for 3D medical imaging, this model was modified for SITS. The temporal axis is treated as the depth dimension, and spectral bands as input channels. Modifications included reducing the encoder depth (removing the fourth stage) to preserve spatial detail on 48×48 inputs and adapting the output layer for dense pixel-wise prediction. It processes the input as a 3D volume using hierarchical Swin Transformer blocks with shifted windows.
   • TSViT: This architecture explicitly factorizes temporal and spatial attention. It first learns temporal signatures for each spatial location (using acquisition dates as positional encoding) and then models spatial relationships. This approach is designed to capture seasonal patterns and irregular temporal sampling.
   • VistaFormer: This model utilizes 3D gated convolutions to downsample spatiotemporal inputs, followed by efficient self-attention over spatial tokens, with the temporal dimension collapsed in the decoder.

Experimental Setup

• Training: Models were trained using Stochastic Gradient Descent (SGD) with momentum (0.9) and a cosine annealing learning rate scheduler for 200 epochs with a batch size of 32.
• Data Augmentation: Limited to horizontal and vertical flips.
• Metrics: Performance was evaluated using Overall Accuracy (OA), Cohen's Kappa, mean Intersection over Union (mIoU), and weighted F1-score (wF1).

Key Results

Munich Dataset

On the Munich dataset, TSViT achieved the highest overall performance, slightly surpassing the 3D U-Net baseline.

• TSViT: 94.50% OA, 93.17% Kappa, 83.01% mIoU, 94.47% wF1.
• 3D U-Net: 94.39% OA, 93.04% Kappa, 81.25% mIoU, 94.28% wF1.
• VistaFormer: Demonstrated the best efficiency, achieving 93.60% OA with only 0.57M parameters and 1.97 GFLOPs, offering a superior efficiency-accuracy trade-off.
• Swin UNETR: Performed competitively (92.51% OA) with low parameter count (4.50M) but did not outperform TSViT.
• 3D DeepLabv3: Performed the worst (83.43% OA), likely due to the ineffectiveness of atrous convolutions on small 48×48 input patches.

Lombardia Dataset

The Lombardia dataset presented a more challenging scenario with lower overall scores across all models, attributed to higher intra-class variability and potential ground truth inconsistencies.

• TSViT again achieved the highest OA (74.74%), Kappa (69.26%), and wF1 (74.12%).
• 3D U-Net attained the highest mIoU (58.50%), indicating a slight trade-off between pixel-wise accuracy and region overlap compared to TSViT.
• VistaFormer maintained strong performance (74.24% OA) with minimal computational cost.
• Swin UNETR remained competitive (71.61% OA) but did not fully exploit the temporal structure compared to TSViT.

Qualitative Observations

Qualitative analysis revealed that accurate predictions are generally found within homogeneous parcels. Errors were most frequent along field borders where crop transitions are ambiguous. In challenging cases, misclassifications affected larger contiguous regions, suggesting confusion between crops with similar spectral or temporal signatures. The models performed better on the Munich dataset, suggesting they benefit from its more consistent spatial and temporal structure.

Key Contributions

1. Comprehensive Comparison: The paper provides a systematic comparison of CNNs and diverse transformer architectures specifically adapted for SITS crop segmentation.
2. Architectural Adaptation: It details the specific modifications required to adapt Swin UNETR (a medical imaging model) for Sentinel-2 time series, including input tensor redefinition and encoder depth reduction.
3. Temporal Modeling Analysis: The study isolates the impact of temporal modeling strategies, comparing explicit temporal factorization (TSViT) and spatiotemporal downsampling (VistaFormer) against treating time as a spatial dimension (Swin UNETR, 3D CNNs).
4. Efficiency Benchmarking: The work highlights the computational efficiency of VistaFormer, demonstrating that high accuracy can be achieved with significantly lower FLOPs and parameter counts.

Significance and Claims

The paper claims that temporal modeling is critical for accurate crop segmentation from SITS. The results indicate that:

• Architectures that explicitly model temporal dynamics (such as TSViT) outperform both standard CNNs and transformers that treat time merely as an additional spatial dimension (like Swin UNETR).
• While 3D U-Net remains a strong baseline, the explicit separation of temporal and spatial attention in TSViT provides a clear advantage, particularly in handling seasonal patterns.
• VistaFormer offers a compelling solution for scenarios requiring high efficiency without sacrificing significant accuracy.
• The choice of temporal modeling strategy plays a decisive role, especially in heterogeneous and challenging environments like the Lombardia dataset.

The authors conclude that effectively modeling temporal dynamics is not just beneficial but essential for fully exploiting the information contained in satellite image time series, particularly in scenarios characterized by complex seasonal patterns and high intra-class variability. Future work is proposed to extend this analysis to higher-resolution imagery and more diverse geographic regions.