Technical Summary: Explainable Comparison of Feature-Based and Deep Learning Models for TROPOMI Methane Plume Screening

Problem Statement

Continuous global detection of large methane emissions is essential for mitigating global warming, particularly given that a significant fraction of emissions originates from "super-emitters" such as coal mines and oil/gas infrastructure. The TROPOspheric Monitoring Instrument (TROPOMI) on the Sentinel-5P satellite provides daily global observations capable of detecting these plumes. However, a major challenge in automated screening is distinguishing genuine methane plumes from retrieval artifacts. These artifacts, which mimic plume-like features, arise from elevation gradients, surface albedo variations, high aerosol loading, coastlines, and water bodies.

Current operational workflows, such as the Copernicus Atmospheric Monitoring Service (CAMS) Methane Hotspot Explorer, rely on a machine learning pipeline to filter these artifacts. The existing second-stage classifier uses a Support Vector Machine (SVM) trained on hand-crafted scalar features. While effective, this approach has limitations: it restricts the algorithm to expert-defined information, disrupts spatial relationships between pixels, and loses information through statistical aggregation. This study addresses the need to evaluate whether advanced image-based deep learning models can outperform or complement these feature-based methods while maintaining interpretability.

Methodology

The study compares two families of machine learning models for binary classification (methane plume vs. retrieval artifact) using TROPOMI data:

1. Feature-Based Models: These operate on pre-extracted numerical descriptors derived from the image patches. The study evaluates:

   • Support Vector Classifier (SVC): Used as a baseline to ensure direct comparability with the benchmark study [5].
   • Random Forest (RF) and Extreme Gradient Boosting (XGBoost): Selected for their demonstrated superiority over SVMs in similar TROPOMI-related tasks.
   • Input: A set of 32 hand-crafted features including statistical moments of methane concentration, plume geometry (elongation, length), wind vectors, and correlations between methane and retrieval diagnostics (e.g., albedo, aerosol optical thickness).

2. Image-Based Models: These are deep learning architectures trained directly on image patches, allowing the capture of spatial patterns and contextual information.

   • ResNet-18 and ResNet-34: Variants of residual neural networks chosen for their ability to handle deep networks via skip connections.
   • Input: Multi-channel image patches including methane concentration, bias-corrected XCH4, surface albedo, aerosol optical thickness (AOT), surface pressure, wind components, and a specific "plume mask" channel. The plume mask encodes the confidence score of the initial plume detection CNN, multiplied by a binary mask.

Experimental Design:

• Dataset: An expanded dataset of 8,883 image patches (6,085 plumes, 2,798 artifacts) covering April 2019 to August 2024.
• Evaluation Settings:
  • Imbalanced Setting: Reflects real-world class ratios (approx. 1:2) using 5-fold cross-validation.
  • Balanced Setting: Isolates the intrinsic discriminative capacity of models on a hold-out test set with balanced classes.
• Hyperparameter Optimization: A random search technique was used to maximize Average Precision (AP) under 5-fold cross-validation.
• Explainability: SHapley Additive exPlanations (SHAP) were applied to both model families. Tree-based models used the TreeExplainer, while image-based models utilized the GradientExplainer to analyze channel-level and pixel-level attributions.

Key Results

Performance Metrics

• Imbalanced Setting (Cross-Validation): Feature-based models, particularly tree-based ensembles, demonstrated the most robust and stable performance.
  • Random Forest (RF) achieved the highest Average Precision (0.948 ± 0.008) and ROC-AUC (0.908 ± 0.014).
  • XGBoost performed similarly, slightly trailing RF.
  • ResNet-18/34 showed lower average precision and ROC-AUC in this setting, with higher standard deviations, suggesting sensitivity to class imbalance and limited effective training data per fold.
• Balanced Setting (Hold-out Test): The relative ranking shifted, highlighting the discriminative capacity of deep learning when class domination is removed.
  • ResNet-18 outperformed all other models, achieving the highest Average Precision (0.937) and ROC-AUC (0.939).
  • ResNet-34 performed strongly but did not surpass ResNet-18, indicating that increased depth did not yield further gains for this specific task.
  • XGBoost achieved the highest Balanced Accuracy (0.859) but lagged behind ResNet-18 in overall discriminative metrics (AP and ROC-AUC).

Explainability Insights

• Feature-Based Models: SHAP analysis identified the CNN plume score (the confidence of the initial detection model) as the most influential feature. Other significant features included Mean QA (quality assurance) values and correlations between methane and aerosol optical thickness (AOT). XGBoost relied more heavily on wind-related context, while RF prioritized enhancement-shape statistics.
• Image-Based Models: The plume mask channel (encoding the CNN plume score and spatial structure) was the most important input. Unlike tree-based models, ResNet-18 effectively utilized surface albedo as a primary feature, learning the spatial correspondence between albedo patterns and methane enhancements to identify artifacts. The model demonstrated balanced positive and negative contributions within channels, reflecting its ability to interpret spatial context and localized patterns.

Significance and Conclusions

The paper concludes that both model families offer complementary strengths for operational methane screening:

• Robustness vs. Discriminative Power: Tree-based models (RF/XGBoost) are preferable when robustness under class imbalance and limited training data is the primary requirement, as seen in the current operational pipeline. Conversely, ResNet-18 is optimal when the goal is to maximize discriminative performance and threshold flexibility, provided sufficient balanced data is available.
• Operational Implications: For the Methane Hotspot Explorer, where minimizing false positives is critical to reduce manual verification workload, the robustness of tree-based models under imbalance is a key asset. However, the superior performance of ResNet-18 in balanced settings suggests that deep learning could advance the pipeline's detection capabilities if data availability and balance improve.
• Interpretability: The study demonstrates that SHAP can unify the interpretation of disparate model types. Both approaches confirm that the confidence of the initial plume detection is the primary driver of classification. However, image-based models leverage spatial coherence and shape information that feature-based models must approximate through statistical aggregation.

The authors emphasize that these findings provide practical guidance for model selection in operational workflows and highlight that the first-stage detection model remains a critical component of the overall pipeline. Future work could focus on improving the first-stage pipeline or exploring more complex deep learning architectures.