LLM-Powered Flood Depth Estimation from Social Media Imagery: A Vision-Language Model Framework with Mechanistic Interpretability for Transportation Resilience

1. Problem Statement

Urban flooding poses a critical threat to transportation networks, yet current systems lack real-time, street-level flood depth information at the centimeter resolution required for:

• Dynamic Routing: Navigation apps (e.g., Google Maps) currently rely on binary road closures, missing nuanced depth data.
• Electric Vehicle (EV) Safety: EV batteries are sensitive to water ingress at depths of 15–25 cm; precise depth estimation is vital for safety.
• Autonomous Vehicle (AV) Operations: AVs lack robust perception for flooded roads, often excluding them from operational design domains.

Existing computer vision approaches are limited by:

• Coarse Categorization: Most methods classify depth into broad categories (e.g., "tire-level") rather than continuous measurements.
• Data Scarcity: No public datasets exist linking street-level images to centimeter-resolution depth labels.
• Black-Box Nature: Vision-Language Models (VLMs) applied to this task lack interpretability and mechanistic understanding, hindering safe deployment.

2. Methodology

The study proposes FloodLlama, an end-to-end framework integrating social media sensing, synthetic data generation, and mechanistic interpretability.

A. Data Pipeline & Synthetic Dataset Generation

• Social Media Sensing: A multimodal pipeline collects TikTok videos (visuals, OCR text, ASR transcripts, comments) to detect flood events and geolocate them.
• Synthetic Data (Unreal Engine 5): To overcome the lack of labeled real-world data, the authors generated a massive synthetic dataset:
  • Scale: ~1.9 million images (1.7M single-vehicle, 1.1M mixed-vehicle).
  • Resolution: Continuous depth labels from 0 to 40 cm at 1 cm increments.
  • Variability: Covers 7 vehicle types (Sedan, SUV, Truck, Bus, etc.), 4 weather conditions (Sunny, Rainy, Night, Afternoon), and mixed-vehicle occlusions.
  • Realism: Validated via Kolmogorov–Smirnov (KS) tests against real flood images, showing the mixed-vehicle dataset achieves high fidelity in brightness and saturation.

B. Model Architecture & Training

• Base Model: LLaMA 3.2-11B Vision Instruct, chosen for its multimodal reasoning and open weights.
• Fine-Tuning Strategy:
  • QLoRA: Parameter-Efficient Fine-Tuning using 4-bit NF4 quantization.
  • Curriculum Learning: A two-phase approach:
    1. Phase 1: Train on single-vehicle synthetic data to learn coarse depth mapping.
    2. Phase 2: Progressive training on mixed-vehicle data chunks to handle occlusion and complex scenes.
  • Prompt Engineering: A structured "Schema-First" prompt providing vehicle metadata (tire height, ground clearance) and enforcing a strict JSON output format ({"flood_depth_cm": <int>}).

C. Mechanistic Interpretability Framework

The authors developed a five-phase framework to understand how the model learns depth estimation, moving beyond black-box evaluation:

1. Adapter Weight Analysis: Identified that adaptation concentrates in specific cross-attention layers (18, 23, 28, 33, 38) and MLP gates.
2. Prompt Sensitivity: Analyzed how different prompts affect attention maps. Found that Simple prompts work best for shallow floods (clear visual cues), while Chain-of-Thought (CoT) prompts improve performance for deep floods (requiring reasoning through occlusion).
3. Layer Ablation: Systematically tested subsets of cross-attention layers. Found that using all 8 layers is suboptimal; optimal subsets vary by depth (e.g., early layers for shallow, late layers for deep).
4. Linear Probing & Logit Lens: Revealed a phase transition at Layer 23 (L23). Below L23, depth is not linearly decodable; above L23, depth information becomes explicitly encoded.
5. Centered Kernel Alignment (CKA): Confirmed that early layers reorganize features, while upper layers (L23–L38) perform the specific task of depth encoding.

D. Selective Fine-Tuning (Tier Architecture)

Based on the interpretability findings, the authors designed Tier 2 and Tier 3 configurations that train only the critical layers (e.g., L8, L18, L23, L28, L33, L38) rather than the whole model. This reduces trainable parameters by 76–80% while maintaining accuracy.

3. Key Results

• Accuracy:
  • Mean Absolute Error (MAE): Achieved < 0.97 cm for deep flooding and < 0.40 cm overall on synthetic test sets.
  • Accuracy @ 5cm: Exceeded 93.7% for deep flooding and 96.8% for shallow depths.
  • Real-World Validation: On real-world TikTok data (Detroit), the Tier 3 configuration achieved 98.62% accuracy in classifying flood levels (No Flood, Partial, Deep).
• Prompt-Depth Interaction:
  • Shallow Floods (<27 cm): Simple prompts outperform CoT.
  • Deep Floods (>27 cm): CoT prompts significantly reduce error by guiding the model through occlusion reasoning.
• Efficiency:
  • Selective fine-tuning (Tier 2/3) reduced trainable parameters from ~54M to 6.5M–7.8M (0.1% of total parameters) with negligible performance loss.
• Robustness:
  • Under visual occlusion (masking vehicle tires), the Tier 3 model showed only a 2.3% accuracy drop, compared to a 9.4% drop for the full-model baseline, indicating more distributed and stable representations.
• Synthetic-to-Real Transfer: A model trained entirely on synthetic data achieved >95% accuracy on real-world images, validating the simulation's fidelity.

4. Key Contributions

1. First Centimeter-Resolution VLM for Floods: Introduced the first fine-tuned open-source VLM capable of continuous flood depth estimation (0–40 cm) rather than categorical classification.
2. Large-Scale Synthetic Dataset: Created a novel dataset of ~1.9 million images with precise 1 cm depth labels, addressing the critical data scarcity in this domain.
3. Mechanistic Interpretability for Safety: Provided the first deep-dive into where and how VLMs encode flood depth, identifying Layer 23 as the critical transition point for depth encoding.
4. Parameter-Efficient Deployment: Demonstrated that selective layer fine-tuning can reduce computational costs by ~80% without sacrificing accuracy, making deployment feasible for resource-constrained agencies.
5. Social Media Sensing Pipeline: Validated a TikTok-based pipeline for real-time, crowdsourced flood monitoring, integrating visual, textual, and audio data.

5. Significance & Implications

• Transportation Resilience: Enables dynamic routing and real-time decision-making for traffic management, moving beyond binary "road closed" signals to graded depth warnings.
• EV & AV Safety: Provides the centimeter-level precision needed to determine if specific EV battery packs or AV sensors are at risk of submersion.
• Infrastructure-Free Monitoring: Offers a scalable solution that utilizes existing smartphones and social media, eliminating the need for expensive, fixed sensor networks.
• AI Methodology: Establishes a protocol for using mechanistic interpretability to guide efficient model design in safety-critical transportation applications, proving that understanding internal model mechanics leads to better, cheaper, and more robust AI systems.