Building Damage Detection using Satellite Images and Patch-Based Transformer Methods

Technical Summary: Building Damage Detection using Satellite Images and Patch-Based Transformer Methods

Problem Statement
Rapid building damage assessment following natural disasters is critical for effective post-disaster response. While satellite imagery offers a scalable method for situational awareness, developing reliable damage classifiers is hindered by significant data challenges. Specifically, satellite datasets often suffer from severe class imbalance (a heavy skew toward "no-damage" labels) and label noise. Furthermore, the visual distinctions between damage categories—particularly between "minor damage," "major damage," and "destroyed"—are often subtle. The xBD dataset, a standardized benchmark for building-level damage across diverse geographies, exemplifies these issues, presenting a difficult testbed for machine learning models.

Methodology
The study evaluates the performance of Vision Transformer (ViT) architectures on the xBD dataset, specifically focusing on DeiT-Tiny and DINOv2-small. The research employs a novel, targeted pre-processing pipeline and specific fine-tuning strategies to address data limitations within constrained computational environments (Google Colab Pro/Pro+).

• Patch-Based Pre-processing: To mitigate background noise (e.g., clouds, roads) and isolate structural features, the authors designed a custom pipeline. This process filters images to retain only those containing buildings with specific damage labels. It then extracts random patches centered on building centroids. A critical step involves calculating an "empty ratio" (the proportion of transparent or black pixels); patches exceeding a 0.01 threshold are re-cropped to ensure the building remains the primary focus. This reduces the dataset size by approximately 50% but increases signal density.
• Model Architectures:
  • DeiT-Tiny: A 5-million parameter model pre-trained on ImageNet, utilizing a teacher-student distillation approach with a [CLS] token and a distinct distillation token.
  • DINOv2-small: A 22-million parameter model pre-trained in a self-supervised manner on the LVD-142M dataset. It utilizes Swish-Gated Linear Units (SwiGLUs) and a [CLS] token but omits the distillation token used in DeiT.
• Training Strategies: The authors compared two training approaches:
  1. End-to-End Fine-tuning: Full model training with hyperparameter grid searches (learning rates, batch sizes) constrained by GPU memory and time limits.
  2. Frozen-Head Fine-tuning: Only the classifier head was trained while the backbone weights remained frozen, a strategy adopted to manage computational costs.
• Evaluation: Models were assessed using accuracy, precision, recall, and macro-averaged F1 scores. To mitigate the effects of class imbalance during evaluation, the testing process was repeated across five independent runs.

Key Contributions

1. Targeted Pre-processing Pipeline: The introduction of a patch-based method that isolates building structures and minimizes background noise, specifically designed to handle the noisy, imbalanced nature of the xBD dataset.
2. Transformer Benchmarking: A comparative evaluation of DINOv2-small and DeiT for multi-class building damage classification, providing empirical evidence of their performance on highly imbalanced disaster data.
3. Resource-Constrained Optimization: An analysis of fine-tuning strategies (End-to-End vs. Frozen Head) specifically tailored for environments with limited compute resources, such as Google Colab.

Results
The study found that small ViT architectures, when combined with the proposed pre-processing and training methods, achieved competitive performance against prior Convolutional Neural Network (CNN) baselines.

• Performance Metrics: The DeiT (End-to-End) model achieved the highest overall performance with an accuracy of 0.782 and a macro-averaged F1 score of 0.599. The DINOv2 (End-to-End) model followed with an F1 score of 0.565.
• Comparison to Baselines:
  • Both transformer models outperformed the original ResNet baseline (Gupta et al., 2019), which reported an F1 of 0.320.
  • The DeiT (E2E) model slightly underperformed the few-shot baseline (Koukouraki et al., 2021) which achieved an F1 of 0.638, but remained competitive.
  • Compared to the UNet Encoder + ResNet18 method (Chen, 2020), the transformer models showed lower overall macro-averaged metrics but demonstrated superior individual class performance for "No-damage" (0.90 vs 0.88), "Major damage" (0.73 vs 0.53), and "Destroyed" (0.73 vs 0.55).
• Class-Specific Challenges: The models struggled significantly with the "Minor damage" class (F1 score of 0.14), where visual cues are most subtle. Conversely, performance was strong for "No damage," "Major damage," and "Destroyed" classes.

Significance and Claims
The paper claims that small ViT architectures, when paired with a targeted patch-based pre-processing pipeline, can effectively handle the noisy and imbalanced data typical of disaster response scenarios. The authors assert that their approach achieves competitive macro-averaged F1 scores relative to prior CNN baselines, particularly improving the detection of severe damage categories.

The authors remain modest regarding the "Minor damage" classification, acknowledging that the subtle visual differences between minor damage and no damage remain a challenge. They suggest that future work could address this by explicitly rebalancing the dataset (e.g., reducing "no-damage" samples or using generative AI for augmentation), modeling broader scene contexts rather than single building crops, and potentially simplifying the label space to align better with human triage protocols. The study concludes that these steps could further establish transformer-based models as strong benchmarks for reliable disaster classification.