Bayesian Transformer for Pan-Arctic Sea Ice Concentration Mapping and Uncertainty Estimation using Sentinel-1, RCM, and AMSR2 Data

Technical Summary: Bayesian Transformer for Pan-Arctic Sea Ice Concentration Mapping and Uncertainty Estimation

Problem Statement
Accurate, high-resolution mapping of Pan-Arctic sea ice concentration (SIC) with reliable uncertainty quantification is critical for climate change studies, Arctic navigation, and adaptation strategies. However, achieving this is hindered by three primary challenges:

1. Subtle Feature Signatures: Distinguishing between SIC classes in satellite imagery (Synthetic Aperture Radar [SAR] and Passive Microwave [PM]) is difficult due to noise and the dynamic marine environment.
2. Model Ambiguity: Traditional deep learning models often lack robust uncertainty estimation, leading to ambiguous SIC maps. Existing methods like Monte Carlo (MC) simulation and ensemble generation provide heuristic approximations that may not represent true posterior distributions.
3. Data Heterogeneity: Integrating diverse data sources—Sentinel-1, RADARSAT Constellation Mission (RCM), and Advanced Microwave Scanning Radiometer 2 (AMSR2)—is complex due to differences in resolution, frequency, and noise characteristics.

Methodology
The authors propose a novel Bayesian Transformer approach that addresses these challenges through three integrated components:

1. High-Resolution Transformer Architecture:
   To improve feature extraction, the authors designed a Transformer model capable of discerning subtle ice patterns. The architecture consists of two distinct modules:

   • GloFormer (Global Module): An "among token" block utilizing multi-head self-attention to model global context across all tokens.
   • LoFormer (Local Module): A "within token" block designed to learn local features by applying self-attention across patches inside each token.
     These blocks are repeated four times to capture both high and low-frequency features. The model is trained using a geographically-weighted L1 loss, which assigns higher weights to confident regions (open water, ice pack) and lower weights to the marginal ice zone, based on U.S. National Ice Center (NIC) charts.

2. Bayesian Extension for Uncertainty Quantification:
   The Transformer is extended into a Bayesian Neural Network (BNN) by treating model parameters ($\omega$) as random variables rather than fixed values.

   • Variational Inference: The model employs Bayes by Backpropagation (BBB) to approximate the true posterior distribution of the parameters using a diagonal Gaussian variational distribution.
   • Loss Function: The training objective combines the geographically-weighted L1 loss with a Kullback-Leibler (KL) divergence loss to minimize the difference between the variational distribution and the prior.
   • Uncertainty Output: After training, model parameters are sampled multiple times. The predictive mean ($\mu$) provides the SIC estimate, while the variance ($\sigma^2$) quantifies the epistemic uncertainty.

3. Decision-Level Data Fusion:
   To address data heterogeneity, the authors fuse Sentinel-1, RCM, and AMSR2 data at the decision level. Rather than fusing raw features, the models generate SIC maps and uncertainty estimates for each data source independently. These are then combined, with a hierarchy established based on the uncertainty levels: Sentinel-1 (lowest uncertainty) is layered on top, followed by RCM, and finally AMSR2.

Key Contributions

• Novel Architecture: Introduction of a high-resolution Transformer with dual global (GloFormer) and local (LoFormer) modules specifically tailored for sea ice pattern recognition.
• Robust Uncertainty Estimation: Implementation of a Bayesian framework (BBB) that explicitly models parameter uncertainty, offering more reliable and calibrated uncertainty maps compared to heuristic methods like MC Dropout or epoch ensembles.
• Effective Fusion Strategy: Demonstration that decision-level fusion of heterogeneous SAR and PM data, guided by model-derived uncertainty, improves both SIC mapping accuracy and uncertainty quantification.

Results and Analysis
The model was tested on Pan-Arctic datasets from September 2021. Key findings include:

• Uncertainty Distribution: The Bayesian Transformer correctly identified higher uncertainty in the dynamic marginal ice zone (10%–90% concentration) compared to open water or dense ice packs.
• Performance Comparison:
  • Uncertainty Magnitude: The Bayesian Transformer produced significantly lower and more consistent uncertainty values (0.01%–0.41%) across data sources compared to MC Dropout (0.17%–29.16%) and Epoch Ensemble (0.64%–14.29%).
  • Data Source Sensitivity: Sentinel-1 data yielded the lowest uncertainty, followed by RCM, with AMSR2 showing the highest uncertainty (likely due to cloud/fog impacts and thermal noise). This validated the decision-level fusion hierarchy.
  • Accuracy: The Bayesian Transformer achieved the highest accuracy for Sentinel-1 and AMSR2 datasets and outperformed deterministic Transformers, MC Dropout, and epoch ensembles for the majority of datasets.
• Visual Quality: The model successfully preserved small ice features (leads and floes) visible in SAR imagery while mitigating the impact of wind effects and thermal noise, demonstrating effective regularization against overfitting.

Significance
The paper claims that the proposed Bayesian Transformer offers a more reliable framework for Pan-Arctic SIC mapping than existing approaches. By integrating uncertainty estimation directly into the training objective, the model provides consistent uncertainty quantification across heterogeneous datasets and demonstrates stronger generalization capabilities. This approach supports the creation of high-resolution SIC maps with robust uncertainty metrics, which are essential for long-term environmental monitoring and safe navigation in a changing Arctic. The authors note that future work will explore heteroscedastic Bayesian Neural Networks and multi-modal feature-level fusion.