Using an On-Device VLM-Based Edge Computing System for Real-Time Disaster Detection

Technical Summary: On-Device VLM-Based Edge Computing for Real-Time Disaster Detection

Problem Statement
Natural disaster detection systems have traditionally relied on Convolutional Neural Networks (CNNs) for image classification. While effective, CNNs typically require extensive fine-tuning on specific datasets to achieve high accuracy and lack the ability to generate natural language responses or integrate flexibly with other sensors post-detection. Conversely, Vision-Language Models (VLMs) offer multimodal understanding and natural language generation capabilities but are often assumed to be too computationally heavy for real-time, on-device edge deployment. This study addresses the gap in verifying whether VLMs can match or exceed the inference performance of CNNs for disaster detection while operating entirely on resource-constrained edge hardware, thereby enabling zero-shot detection and natural language-based downstream tasks without network dependency.

Methodology
The research was conducted using an on-device edge computing environment to ensure real-time applicability and energy efficiency.

• Hardware: All inference experiments were executed on a Raspberry Pi 5 to simulate a low-cost, scalable edge device.
• Dataset: The study utilized the AIDER (Aerial Image Dataset for Emergency Response Applications), filtering the data to four disaster-related classes: fire, flood, building collapse, and traffic accident. "Normal conditions" were excluded.
• Models:
  • Baseline (CNNs): Three architectures were evaluated: GoogLeNet (Inception v1), ResNet-18, and MobileNet V2.
  • Experimental (VLM): Gemma 3 (4B parameter variant), a multimodal model capable of processing text and images.
• Experimental Design:
  • Fairness Control: To ensure a fair comparison with the zero-shot VLM, CNNs were first evaluated using ImageNet-pretrained weights without any exposure to the AIDER dataset.
  • Fine-Tuning Comparison: CNNs were subsequently fine-tuned on the AIDER dataset (80 images per class for training, 20 for validation) to establish an upper-bound performance baseline.
  • Reproducibility: Experiments were run across five distinct random seeds (42–46) to rule out incidental variation.
  • Metrics: Performance was measured using Accuracy and F1 Score.

Key Results

• ImageNet-Pretrained CNNs: Without fine-tuning, the CNN models performed poorly, with accuracy ranging between 0.20 and 0.35 and F1 scores between 0.10 and 0.30. This indicates that standard ImageNet weights are insufficient for specific disaster classification tasks without adaptation.
• Fine-Tuned CNNs: After fine-tuning on the AIDER dataset, CNN performance improved significantly. ResNet-18 achieved the highest performance among CNNs (accuracy in the high 0.8s), followed by MobileNet V2 and GoogLeNet. All fine-tuned models stabilized in the 0.85–0.90 accuracy range.
• Zero-Shot VLM (Gemma 3): The Gemma 3 model, operating in a zero-shot setting without any training on the AIDER dataset, achieved a stable accuracy and F1 score between 0.92 and 0.94 across all seeds.
• Comparative Performance: The VLM outperformed the ImageNet-pretrained CNNs by a margin of approximately 60 percentage points (0.92–0.94 vs. 0.20–0.35). Crucially, the VLM's zero-shot performance was comparable to, and in some seeds slightly superior to, the fine-tuned CNNs. The VLM demonstrated consistent performance regardless of the data split, suggesting robust generalization of disaster-related visual patterns learned during its large-scale pretraining.

Significance and Claims
The paper claims that VLMs are a viable and potentially superior alternative to CNNs for real-time disaster detection on edge devices. The primary significance lies in three areas:

1. Performance Viability: The study demonstrates that a 4B parameter VLM can achieve high-accuracy disaster detection on a Raspberry Pi 5 without the need for dataset-specific fine-tuning, challenging the notion that VLMs are too heavy or require extensive retraining for edge tasks.
2. Operational Efficiency: By operating entirely on-device, the system ensures functionality in scenarios with limited or no network connectivity, a critical factor for disaster response.
3. Functional Expansion: Unlike traditional CNNs which output only class labels, the VLM-based system enables natural language-based response generation. This allows for flexible integration with other sensors and the potential for automated, language-driven downstream actions (e.g., voice or text-based reporting) directly at the edge.

The author acknowledges limitations, specifically the inability to test larger models (e.g., DeepSeek, GPT) due to the memory constraints of the Raspberry Pi 5, and note that performance gaps may vary with different data compositions. However, the study concludes that for immediate deployment where setup time and cost are factors, VLMs offer a compelling advantage over the traditional CNN pipeline.