Assessing Vision-Language Models for Perception in Autonomous Underwater Robotic Software

1. Problem Statement

Autonomous Underwater Robots (AURs) operate in harsh, low-visibility environments where traditional deep learning-based perception modules face significant challenges. These challenges include:

• Data Scarcity and Noise: The lack of large, high-quality labeled datasets for underwater environments makes training robust deep learning models difficult.
• Generalization Limits: Traditional models often fail to detect objects outside their specific training distribution (e.g., unseen trash types or unexpected debris).
• Trustworthiness: In safety-critical maritime applications (e.g., trash collection, subsea inspection), unreliable perception can lead to severe operational failures.
• The Gap: While Vision-Language Models (VLMs) offer promising generalization and reasoning capabilities, their performance, uncertainty quantification, and calibration in underwater contexts remain understudied from a software engineering perspective. Industrial partners (like DNV) lack empirical evidence to determine if VLMs can be safely integrated into AUR software.

2. Methodology

The authors conducted a systematic empirical evaluation of VLMs as perception modules for AURs, focusing on underwater trash detection.

• Models Evaluated: Four open-source VLMs (released 2024 or later) with similar parameter counts (~7B) to ensure fair comparison:
  • InstructBLIP (BLIP)
  • LLaVA-1.6 (LLaVA)
  • DeepSeek-VL2 (DeepSeek)
  • QWen2.5-VL (QWen)
• Datasets: Two benchmark underwater datasets were used in a zero-shot setting (no fine-tuning):
  • TrashCan1.0: Contains marine animals, vegetation, man-made objects, and trash.
  • SeaClear: Focuses on deep/shallow scenes with mixed natural and artificial clutter.
  • Classes: The study categorized outputs into four classes: Trash, Object, Animal, and Vegetation.
• Experimental Setup:
  • Prompting: A standardized zero-shot prompt instructed models to identify and list visible objects in the four categories.
  • Execution: All models ran on an NVIDIA RTX-5090 GPU with deterministic settings (temperature = 0) to ensure reproducibility.
• Evaluation Metrics:
  • Performance: F1-Score (Macro/Micro) and Jaccard Index (Macro/Micro).
  • Uncertainty Quantification:
    • Confidence: Max Softmax Probability (MSP) and Probability Confidence Score (PCS).
    • Uncertainty: Entropy (ENT) and Deep Gini (DG).
    • Calibration: Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) to measure the alignment between predicted confidence and actual accuracy.

3. Key Contributions

1. Empirical Evaluation: The first systematic assessment of four state-of-the-art VLMs for underwater object detection and trash classification in a zero-shot setting.
2. Uncertainty & Calibration Analysis: A comprehensive study of the relationship between model performance, confidence, uncertainty, and calibration, providing a framework for selecting VLMs based on reliability rather than just accuracy.
3. Industrial Guidance: Evidence-based recommendations for software engineers and industrial stakeholders (e.g., DNV) on integrating VLMs into Autonomous Cyber-Physical Systems (ACPS) for maritime safety.

4. Key Results

Performance (RQ1)

• Top Performers: BLIP and DeepSeek achieved the highest overall performance. BLIP slightly outperformed DeepSeek in aggregated metrics (F1 Micro: 0.67 vs. 0.66).
• Class-Specific Variance:
  • Trash & Objects: BLIP and DeepSeek significantly outperformed LLaVA and QWen (e.g., BLIP F1 for Trash = 0.76; for Objects = 0.87).
  • Animals & Vegetation: All models performed poorly (F1 scores generally < 0.43), indicating VLMs currently struggle with biological classification in underwater settings without fine-tuning.
• Worst Performers: LLaVA generally ranked lowest in performance, except in specific metrics on SeaClear where QWen was slightly worse.

Uncertainty and Calibration (RQ2)

• Overconfidence: LLaVA was the most confident and "certain" model (highest MSP/PCS, lowest ENT/DG) but was poorly calibrated. It exhibited high confidence even when incorrect, making it unreliable for safety-critical tasks.
• Reliability: BLIP was the best-calibrated model. While it showed slightly lower confidence and higher uncertainty (more cautious) compared to LLaVA, its probability estimates aligned much better with actual performance.
• DeepSeek showed similar calibration to BLIP but slightly lower confidence.

Performance-Uncertainty Relationship (RQ3)

• Key Finding: High performance does not correlate with high confidence or low uncertainty. Instead, high performance correlates strongly with good calibration.
• Implication: The best models (BLIP/DeepSeek) were not the most confident, but they were the most reliable because they correctly expressed uncertainty when unsure. LLaVA's high confidence was a liability due to miscalibration.

5. Significance and Implications

• Selection Criteria for AURs: The study argues that when selecting VLMs for AUR software, calibration and performance should be prioritized over raw confidence. A model that admits uncertainty (like BLIP) is safer than an overconfident model (like LLaVA) that may make silent errors.
• Hybrid Deployment Strategy: Due to poor performance on "Animal" and "Vegetation" classes, VLMs should not yet be used as standalone perception systems for all tasks. Instead, they are best suited as supporting components in a hybrid architecture:
  • Triggered when traditional deep learning models fail or exhibit high uncertainty.
  • Used for specific, scoped tasks like trash and object identification where they show high efficacy.
• Digital Twins: The paper suggests a practical industrial use case: integrating VLMs into Digital Twins of AURs. Since Digital Twins have access to powerful cloud computing, they can run larger VLMs to validate perception or identify unknown objects before the physical robot takes action.
• Regulatory Compliance: The empirical evidence provided helps address the "black box" nature of AI in maritime regulations (IMO), offering a pathway to certify VLMs based on quantified uncertainty and calibration metrics.

Conclusion

The paper concludes that while VLMs hold great promise for AUR perception due to their generalization capabilities, they are not yet a "drop-in" replacement for all perception tasks. BLIP emerges as the most reliable candidate for industrial deployment due to its superior calibration and performance on trash/object detection. Future work should focus on lightweight fine-tuning to improve performance on biological classes and integrating these models into real-time AUR control loops with uncertainty-aware safety mechanisms.