Clay-CNN Hybrids: Leveraging Geo-Foundational Models as Auxiliary Context for Landslide Detection

Technical Summary: Clay-CNN Hybrids for Landslide Detection

Problem Statement
Rapid post-event landslide mapping is critical for disaster response but remains difficult to automate due to two primary technical obstacles: spectral similarity between landslide scars and bare soil, and extreme class imbalance (landslide pixels comprising only ~2% of the dataset). Traditional deep learning approaches, particularly U-Net architectures, rely heavily on task-specific training data, which is often unavailable during active disasters. While Geospatial Foundation Models (GFMs) like Clay v1.5 offer generalizable representations pretrained on massive, diverse satellite imagery, there is no systematic evaluation of how to best integrate these models with convolutional architectures to optimize landslide detection in data-scarce regimes.

Methodology
This study evaluates the integration of Clay v1.5, a geospatial Vision Transformer (ViT), with convolutional neural networks (CNNs) using the Landslide4Sense (L4S) benchmark. The dataset consists of 3,799 training chips (128×128 pixels) containing 14 input channels: 12 Sentinel-2 optical bands and 2 topographic bands (DEM and slope). The study employs a geographically stratified "leave-out-location" strategy to prevent spatial leakage.

Three architectural strategies were compared:

1. Baseline (U-Net): A standard U-Net with four encoder stages, full skip connections, and 14 input bands, serving as the CNN-only reference.
2. Architecture 1 (Clay Backbone): Clay v1.5 serves as the primary encoder for the 12 optical bands, producing a single 16×16 feature map. Topographic data is processed via a separate "TerrainPyramid" and fused residually at coarse decoder scales using a learnable scalar $\alpha$. This architecture lacks encoder skip connections.
3. Architecture 2 (Hybrid U-Net + Clay): A U-Net backbone processes all 14 bands through four downsampling stages. Clay v1.5 independently processes the 12 optical bands. The Clay output is projected, downsampled to 8×8, and concatenated with the U-Net bottleneck. The fused representation is then decoded using standard U-Net skip connections.

Training and Optimization
All models utilized a two-stage training protocol to stabilize learning:

• Stage 1: The Clay encoder was frozen while training the decoder, terrain branch, and fusion layers.
• Stage 2: Low-Rank Adaptation (LoRA) adapters were inserted into the Clay transformer blocks (targeting Q/K/V projections and MLPs) to fine-tune the encoder at a lower learning rate.
• Loss Function: A combination of Binary Cross-Entropy (with a positive weight of 25 to address imbalance) and Lovász hinge loss.
• Evaluation: Models were evaluated using pixel-wise F1 score, with classification thresholds optimized on the validation set to account for severe class imbalance.

Key Contributions

• Novel Architectures: Introduction of two hybrid designs: one using Clay as a primary encoder with multi-scale terrain fusion, and a second using Clay as auxiliary bottleneck context within a U-Net.
• Systematic Ablation: Comprehensive analysis of encoder freezing, LoRA fine-tuning, two-stage training strategies, loss formulations, and terrain integration.
• Interpretability and Uncertainty: Application of Monte Carlo Dropout (MC Dropout) for uncertainty quantification and Gradient-weighted Class Activation Mapping (Grad-CAM) to diagnose model behavior and feature contributions beyond aggregate metrics.

Results

• Performance: The Hybrid U-Net + Clay model (Architecture 2) achieved the best test F1 score of 64.5 ± 1.8% (across three seeds), surpassing both the U-Net baseline (59.9%) and the Clay-only backbone (55.2 ± 3.6%).
• Architectural Limitations: Clay as a standalone encoder underperformed the U-Net baseline, confirming that the ViT's single-scale output without skip connections is insufficient for precise pixel-level segmentation.
• Hybrid Advantage: Injecting Clay's pretrained representations as auxiliary context at the bottleneck significantly improved performance. The hybrid model traded a modest reduction in recall for a substantial increase in precision, reducing false positives in spectrally similar bare soil.
• Uncertainty Analysis: MC Dropout analysis revealed that the model's uncertainty correctly concentrated on error-prone regions (false positives and false negatives) and landslide boundaries, indicating the model is aware of its own limitations.
• Grad-CAM Findings: Analysis showed that the fusion of Clay context acted as a corrective prior, broadening contribution outside scar interiors to suppress false positives in geomorphically implausible areas (e.g., flat agricultural fields), while the CNN decoder handled pixel-level localization.

Significance and Claims
The paper concludes that Geospatial Foundation Models are most effective for landslide detection when they complement rather than replace spatially-detailed convolutional architectures. The study demonstrates that GFMs provide necessary spectral generalization, while CNNs provide the spatial precision required for segmentation. This hybrid approach outperforms both standalone GFMs and traditional CNNs in data-scarce regimes, offering a scalable solution for rapid post-event mapping without relying on extensive task-specific labeled data. The results suggest this principle of combining spectral generalization with spatial precision is applicable to other geohazard and planetary surface mapping tasks.