Technical Summary: DamageArbiter

Problem Statement

Rapid, hyperlocal disaster damage assessment using street-view imagery is critical for emergency response and resource allocation. While computer vision models offer a promising avenue for this task, existing approaches face three primary limitations:

1. Reliance on "Black Box" Models: Current methods often depend on pre-trained vision models (e.g., VGG, Swin Transformer) that lack interpretability and reliability.
2. Overconfidence Errors: Models are frequently evaluated solely on accuracy or F1-scores, ignoring the risk of "overconfidence"—where a model assigns high confidence to incorrect predictions. In disaster contexts, a confident but wrong prediction can be more detrimental than an uncertain one.
3. Modality Vulnerability: Single-modality approaches (image-only) are prone to errors when visual evidence is ambiguous, while text-only approaches may miss critical visual cues. Furthermore, the application of Vision-Language Models (VLMs) like CLIP in this domain is nascent, hindered by a lack of high-quality image-text datasets and insufficient evaluation of their reliability.

Methodology

The study proposes DamageArbiter, a multimodal, disagreement-driven arbitration framework designed to enhance both the accuracy and reliability of damage assessment.

Data

The framework was validated using the Milton-SV dataset, comprising 2,556 post-disaster street-view images from Horseshoe Beach, Florida, following Hurricane Milton (2024). Images were manually labeled into three severity levels: mild, moderate, and severe. Each image was paired with two types of textual descriptions:

• Human Annotations: Fine-grained descriptions by trained annotators.
• LLM Annotations: Descriptions generated by GPT-4o-mini, selected for their high semantic alignment (CLIPScore) with the images.

Model Architectures

The study compared four architectural approaches:

1. Unimodal Image-Only: A fine-tuned Vision Transformer (ViT) model (ViT-B/32) serving as the visual baseline.
2. Unimodal Text-Only: A CLIP text encoder fine-tuned on either human or LLM-generated captions.
3. Multimodal CLIP: A cross-modal contrastive learning framework aligning image and text embeddings, followed by a classification head.
4. DamageArbiter (The Proposed Framework): A meta-classifier that arbitrates between the ViT and CLIP-LLM models.
   • Mechanism: When the ViT and CLIP-LLM models agree, their shared prediction is accepted. When they disagree, a lightweight logistic regression arbitrator is activated.
   • Arbitration Features: The arbitrator uses the max-softmax confidence scores of the two base models to decide which prediction to trust. Ablation studies confirmed that adding uncertainty metrics (entropy, decision margin) did not significantly improve performance over confidence scores alone.
   • Training: The arbitrator was trained on out-of-fold disagreement samples, learning to trust the ViT prediction when $g(x) > 0.5$ and the CLIP prediction otherwise.

Evaluation Metrics

Beyond standard classification metrics (Accuracy, Recall, Precision, SW-F1, MCC), the study introduced a rigorous reliability analysis focusing on confidence-based errors:

• Overconfidence Errors: Incorrect predictions with a high confidence margin (0.4–1.0).
• Ambiguity Errors: Incorrect predictions with low confidence margins (0–0.1), indicating indistinguishable classes.
• Semantic Alignment: Measured via CLIPScore to evaluate the quality of LLM-generated text.

Key Results

Baseline Performance

• Visual Dominance: The image-only ViT-B/32 model achieved the highest baseline accuracy (74.33%) and MCC (0.5947).
• Multimodal Trade-offs: The multimodal CLIP model with human annotations achieved comparable accuracy (74.22%) but exhibited a significantly different error profile.
• Text Limitations: Text-only models performed the worst (63.07% accuracy for LLM text), confirming that visual cues are essential for this task.

Reliability and Arbitration

• Overconfidence Reduction: The image-only ViT baseline suffered from a high rate of overconfidence errors (70.58% of its errors). In contrast, the multimodal CLIP model produced zero overconfidence errors, though it had higher rates of ambiguity.
• DamageArbiter Performance:
  • Accuracy: Improved to 75.85% (up from 74.33% for ViT).
  • Reliability: The Matthews Correlation Coefficient (MCC) increased to 0.6188.
  • Overconfidence Mitigation: The framework reduced the overconfidence error rate from 70.58% (ViT baseline) to 16.45%.
• Ablation Findings: The logistic regression arbitrator, using only confidence scores, outperformed a naive heuristic (trusting the more confident model) and either base model in isolation.

Spatial Deployment

When deployed on the Horseshoe Beach dataset, DamageArbiter successfully mapped damage severity, with the arbitration mechanism engaging primarily in interior corridors where visual and textual evidence conflicted. The resulting error map showed sparse overconfident misclassifications compared to the dense errors of the image-only baseline.

Significance and Contributions

The paper claims three primary contributions:

1. Redefining Evaluation: It demonstrates that accuracy alone is insufficient for disaster damage classification. The study highlights the necessity of measuring overconfidence errors as a critical component of model reliability, particularly in high-stakes scenarios where false certainty can mislead resource allocation.
2. The DamageArbiter Framework: It introduces a lightweight, deployable arbitration mechanism that effectively combines the strong visual discrimination of image-only models with the conservative, semantic reliability of multimodal CLIP models. This approach improves both predictive accuracy and balanced classification (MCC).
3. Reliability via Arbitration: The study shows that disagreement-driven arbitration can substantially mitigate overconfidence (reducing it from ~70% to ~16%) without sacrificing accuracy, offering a more trustworthy framework for rapid, hyperlocal disaster assessment.

Limitations and Future Directions

The authors acknowledge that the study is limited to a single hurricane event and a specific geographic area, requiring further testing for generalizability across different disaster types and urban forms. Additionally, the current framework assesses damage by severity level rather than specific damage types (e.g., fallen trees vs. flooding). Future work is proposed to extend the framework to type-specific mapping using open-vocabulary detection and to integrate uncertainty-aware calibration strategies into operational deployment.