Technical Summary: HorusEye – Language as Dynamic Attention for Emergency Visual Analysis

1. Problem Statement

Emergency rescue operations frequently occur under visual conditions that deviate significantly from the clean, well-lit distributions used to train Vision-Language Models (VLMs). Factors such as fog, smoke, and the fundamental modality shift of thermal imaging (which removes color and alters texture) pose critical challenges for computer vision systems. While VLMs have demonstrated strong capabilities in standard visual understanding, their robustness under these degraded conditions remains unquantified.

The core research question addressed by this study is: To what extent can natural language feedback serve as a dynamic attention mechanism to re-evaluate and refine visual analytical tasks in real-time under degraded emergency conditions? The authors propose that natural language prompts can act as external signals to guide a model's visual processing, analogous to verbal instructions given by human rescuers in chaotic scenes.

2. Methodology

2.1 Dataset: RefCOCO-Degraded

To systematically evaluate performance under degradation, the authors constructed RefCOCO-Degraded, a benchmark dataset comprising 15,244 images derived from 3,811 base images from the RefCOCO dataset. Each base image was systematically degraded into four conditions:

• Clean: The original unmodified image (baseline).
• Fog: Simulated using the Koschmieder visibility model to reduce contrast while preserving color.
• Smoke: Simulated using particle-based occlusion with Perlin noise to introduce local variations and gray color shifts.
• Thermal: Simulated by converting RGB images to grayscale intensity values, removing all color information.

2.2 Models Evaluated

Five VLMs representing diverse architectural approaches were evaluated:

• Gemini 2.0 Flash: Proprietary multimodal architecture.
• Qwen2-VL-2B: Open-source, instruction-tuned VLM (2B parameters).
• BLIP-2: Utilizes a Q-Former to bridge frozen image encoders with LLMs.
• LLaVA-1.6: Directly projects visual features into LLM embedding space.
• Kosmos-2: A grounding model with explicit grounding capabilities.

2.3 Experimental Pipeline

The investigation followed a five-stage pipeline addressing four sequential research questions (RQs):

1. RQ1 (Visual Grounding Baseline): Measured the baseline degradation of visual grounding performance (using Intersection over Union, IoU) across the four conditions.
2. RQ2 (Language Feedback as Dynamic Attention): Tested an iterative three-round feedback protocol. If an initial prediction failed ($IoU < 0.5$), the model received directional language feedback (e.g., "look more to the left") to correct the bounding box. The metric was the percentage gain in IoU from Round 1 to Round 3.
3. RQ3 (Health VQA Generalization): Evaluated posture classification (standing, sitting, lying) to test if explicit spatial attention (via cropping the region of interest) generalizes to health tasks. This compared Full Image inputs against Cropped Image inputs.
4. RQ4 (Hallucination Analysis): Quantified the risk of hallucination under degradation using a Hallucination Score (H-Score). This score combined fabrication counts, overconfidence penalties, and uncertainty bonuses to determine if models became more dangerous (confidently incorrect) or appropriately uncertain under stress.

3. Key Contributions

1. RefCOCO-Degraded Benchmark: The introduction of a systematic benchmark dataset specifically designed for emergency VLM evaluation, covering fog, smoke, and thermal degradations.
2. Model-Dependent Efficacy of Language Feedback: Empirical evidence demonstrating that language feedback is not a universal solution. While Gemini achieved a +47.3% improvement in thermal conditions through iterative feedback, Qwen2-VL suffered a -5.1% degradation under the identical protocol.
3. The "Thermal Paradox": The discovery that cropping strategies, which improve performance in RGB conditions (e.g., +4% for Gemini), cause catastrophic failure in thermal imagery (-26% for Gemini). This highlights that thermal imagery relies on contextual cues (e.g., horizon, surrounding objects) that are lost when the image is cropped.
4. Safety Identification of BLIP-2: The identification of BLIP-2 as uniquely dangerous for emergency deployment. Unlike other models that increased uncertainty under degradation, BLIP-2 exhibited increased hallucinations (+0.34 H-Score increase) and a complete lack of hedging language, making it prone to confidently fabricating information.

4. Key Results

• Visual Grounding (RQ1): Thermal conditions caused the most severe degradation. Qwen2-VL lost 63.6% of its grounding accuracy in thermal conditions, while Gemini lost 37.7%. Fog and smoke caused moderate degradation (10–15% IoU drop).
• Language Feedback (RQ2):
  • Gemini: Showed a strong positive correlation between degradation severity and feedback utility. Improvement ranged from +0.5% (Clean) to +47.3% (Thermal).
  • Qwen2-VL: Showed positive gains in Clean/Fog conditions (+18.7%, +17.0%) but degraded by -5.1% in Thermal conditions, indicating the feedback mechanism confused the model in low-visibility thermal scenarios.
• Cropping Strategies (RQ3):
  • Gemini: Cropping helped in Clean conditions (+4%) but destroyed performance in Thermal conditions (-26%).
  • BLIP-2: Uniquely robust to cropping, benefiting from focused attention in all conditions (+12% to +16%).
  • Qwen2-VL: Consistently harmed by cropping in RGB conditions (-12%).
• Hallucination (RQ4):
  • BLIP-2: Classified as DANGER. Its H-Score increased significantly under thermal degradation, and it failed to express uncertainty (0.00 uncertainty score).
  • Gemini & Qwen2-VL: Classified as SAFER. Both models appropriately increased uncertainty expressions under degraded conditions, reducing the risk of confident hallucinations.

5. Significance and Claims

The paper claims that natural language can function as a dynamic attention mechanism for VLMs, but its success is critically tied to the specific model architecture and visual conditions.

• No Universal Solution: Techniques proven successful for one model (e.g., iterative feedback for Gemini, cropping for BLIP-2) cannot be assumed to transfer to others. In fact, the same mechanism can yield opposite results (improvement vs. degradation) depending on the model.
• Context is Critical in Thermal Imaging: The "Thermal Paradox" establishes that for thermal imagery, full-image processing is essential. Cropping removes the spatial and contextual cues necessary for tasks like posture classification when color and texture are absent.
• Safety Implications: For emergency system designers, the selection of the model is a safety-critical decision. The authors recommend Gemini for thermal conditions due to its responsive feedback and calibrated uncertainty, while explicitly advising against the deployment of BLIP-2 in emergency scenarios unless hallucination mitigation is implemented.

The study concludes that emergency system designers must rigorously validate language feedback mechanisms and attention strategies for each unique model-condition combination prior to deployment, rather than relying on general assumptions about VLM robustness. The RefCOCO-Degraded benchmark and experimental protocols provide a foundation for future research into robust VLMs for safety-critical applications.