Technical Summary: FloodReasonBench

Problem Statement

Embodied agents, such as unmanned aerial vehicles (UAVs) and mobile robots, are increasingly critical for flood response operations where human access is unsafe. These agents require a perception interface capable of translating mission-relevant natural language requests (e.g., "locate submerged vehicles") into pixel-level visual grounding. While Vision-Language Models (VLMs) coupled with reasoning segmentation offer this capability, existing benchmarks and models face two primary challenges in the flood-response domain:

1. Domain Gap: Current reasoning-segmentation benchmarks (e.g., ReasonSeg) focus on generic visual scenes. When applied to flood scenarios without specific adaptation, models like LISA suffer significant performance degradation.
2. Resource Constraints: Deploying these models on edge platforms (e.g., NVIDIA Jetson) is hindered by the high computational cost of heavy image encoders (like SAM ViT-H) and the communication overhead of fully remote execution. While split computing offers a middle ground, existing approaches often lack a systematic evaluation of how hierarchical partitioning and feature compression interact with domain-specific task requirements.

Methodology

The authors introduce FloodReasonBench, a benchmark designed to evaluate VLM reasoning segmentation for embodied flood response under resource constraints. The methodology encompasses three core components:

1. FloodResponseSeg Dataset

A new, flood-specific reasoning-segmentation dataset was constructed via a semi-automated pipeline:

• Curation: Real-world flood imagery was filtered using CLIP, localized with Grounding DINO, and segmented using SAM2.
• Annotation: Manual inspection ensured quality, followed by the authoring of natural-language queries describing targets (people, buildings, vehicles) within the flood context.
• Composition: The dataset includes 532 original training samples (augmented to 2,128 instances) and 100 evaluation samples, covering three critical categories for flood response.

2. Model Architecture and Split Design

The benchmark evaluates the LISA framework with specific modifications for edge deployment:

• Lightweight Backbone: The heavy SAM ViT-H encoder is replaced with MobileSAM, utilizing a hierarchical TinyViT backbone (12 blocks: 2 MBConv + 10 Transformer blocks).
• Hierarchical Split Inference: The TinyViT backbone is partitioned at 12 candidate points. The edge device executes the prefix, compresses the intermediate feature via a learned Latent Autoencoder (AE), and transmits it to a remote server for reconstruction and completion of the inference.
• Compression Strategy: Separate AEs are trained for each partition point to match specific feature dimensions, trained on ADE20K for feature reconstruction rather than the downstream segmentation task.

3. Evaluation Protocol

The evaluation proceeds in two stages:

• Pre-adaptation: Evaluating split configurations on the generic ReasonSeg dataset to characterize partition sensitivity without flood-specific tuning.
• Split-aware Flood Adaptation: Fine-tuning the reasoning-segmentation components on FloodResponseSeg while keeping the learned compression bottlenecks fixed. This assesses the design space under the target workload.
• System Metrics: Experiments are conducted on an NVIDIA Jetson AGX Xavier across multiple power modes (MAXN, 10W, 15W, 30W). Metrics include task accuracy (gIoU, cIoU), edge-side latency, energy consumption per frame, and communication footprint (compressed feature size).

Key Contributions

1. FloodReasonBench & FloodResponseSeg: The introduction of the first benchmark and dataset specifically for reasoning segmentation in flood response, addressing the gap between generic VLM benchmarks and domain-specific flood vision tasks.
2. Systematic Characterization of Partition Sensitivity: A detailed analysis of how hierarchical split inference performs under feature compression. The study reveals that while generic workloads show strong partition-dependent accuracy variation, flood-adapted workloads exhibit a much more compact accuracy range across different split points.
3. Quality-Constrained Design Space: A joint evaluation of task quality, latency, energy, and communication costs. This enables the selection of "operating points" that satisfy specific perception quality requirements while minimizing resource usage on constrained edge platforms.

Results

• Domain Adaptation Necessity: Without flood-specific fine-tuning, LISA-SAM achieves only 0.7223 gIoU on FloodResponseSeg. Flood-specific adaptation improves this to 0.8423 gIoU (a 12.0-point gain), demonstrating the critical need for domain adaptation.
• Lightweight Encoding: Replacing SAM ViT-H with MobileSAM (TinyViT) after adaptation results in a minor accuracy drop (0.8423 to 0.8202 gIoU) but enables the hierarchical split design space.
• Partition Sensitivity:
  • Generic Setting (ReasonSeg): Accuracy varies widely across partitions (gIoU spread of 18.24 points), indicating high sensitivity to where the split occurs.
  • Flood-Adapted Setting: The accuracy range becomes substantially more compact (gIoU spread of 4.26 points). This implies that in the target domain, multiple partition points offer comparable task quality, shifting the selection criteria to system-level costs.
• Tradeoff Analysis:
  • Deeper partitions increase edge latency and energy but do not always reduce communication size due to hierarchical feature dimensions.
  • For a quality constraint of gIoU ≥ 0.79, Block 1 (an early split) offers the most efficient operating point, achieving 0.7941 gIoU with 50.1 ms latency, 0.461 J/frame, and a 0.125 MiB transmission size. Compared to the highest-quality Block 9, this reduces latency by 50.5%, energy by 54.8%, and feature size by 50% with negligible accuracy loss.
  • Platform power modes significantly impact costs; for instance, running Block 1 at 10W requires 142.8 ms and 0.947 J, compared to 50.1 ms and 0.461 J at 30W.

Significance and Claims

The paper claims that FloodReasonBench provides a necessary task- and system-level characterization of reasoning segmentation for resource-constrained embodied flood response. Its primary significance lies in demonstrating that:

1. Domain adaptation is non-negotiable for effective flood response perception.
2. Flood-adapted workloads exhibit robustness to partition selection, unlike generic workloads. This allows system designers to prioritize latency, energy, and bandwidth constraints when selecting split points, rather than being forced into a single "accuracy-optimal" partition.
3. Joint optimization of perception quality and edge resource costs is essential for practical deployment, enabling the selection of operating points that satisfy specific mission requirements under varying connectivity and power envelopes.

The authors note that future work will extend the dataset to additional entities, incorporate emerging models, and evaluate end-to-end closed-loop execution under varying network conditions.