Exploring the Underwater World Segmentation without Extra Training

1. Problem Statement

Accurate segmentation of marine organisms is critical for biodiversity monitoring and ecological assessment. However, existing research faces two major bottlenecks:

• Data Scarcity and Coarseness: Most existing underwater datasets are limited in scale (often <40 categories) and rely on coarse labeling (e.g., labeling all aquatic life simply as "fish"). They lack the fine-grained diversity required for open-vocabulary (OV) evaluation.
• Domain Gap and Training Costs: Current segmentation models are predominantly trained on terrestrial data. Adapting them to underwater environments typically requires extensive retraining on scarce underwater data. Furthermore, underwater imagery presents unique challenges such as color attenuation, light scattering, low visibility, and blurred textures, which degrade the performance of standard Vision-Language Models (VLMs) like CLIP.

2. Methodology: Earth2Ocean

The authors propose Earth2Ocean, a training-free open-vocabulary segmentation framework designed to transfer terrestrial VLMs to underwater domains without any additional underwater training. The framework consists of two core modules:

A. Geometric-guided Visual Mask Generator (GMG)

• Goal: To refine visual features and improve mask generation by leveraging geometric priors, which remain relatively stable in underwater scenes despite color and texture degradation.
• Mechanism:
  • It utilizes a pre-trained geometric encoder (based on DINO features) to extract multi-scale geometric features.
  • It computes a self-similarity map from the last layer of the geometric encoder to serve as an attention prior.
  • This geometric attention map is used to correct and refine the visual embeddings extracted from the CLIP visual encoder.
  • Result: This produces spatially consistent, geometry-aware visual masks that are robust to underwater visual noise.

B. Category-visual Semantic Alignment (CSA)

• Goal: To bridge the semantic gap between terrestrial VLMs and underwater concepts, ensuring that text embeddings align correctly with underwater visual features.
• Mechanism:
  • Scene Context Enhancement: It employs a template-based approach to inject underwater-specific context (e.g., "swimming in clear water," "resting on the seabed") into text prompts.
  • MLLM-Guided Reasoning: It leverages a Multimodal Large Language Model (MLLM) to generate rich, instance-specific descriptions. The MLLM outputs a caption, a list of objects, and detailed attributes (color, shape, size).
  • Fusion: These reasoning-aware embeddings are fused with the original category embeddings using a similarity-guided weighting mechanism.
  • Result: The text features are enriched with domain-specific knowledge and fine-grained attributes, significantly improving the alignment with underwater visual features.

C. Final Prediction

The refined visual features (from GMG) and the enhanced text embeddings (from CSA) are projected and matched via a linear layer followed by a softmax operation to generate the final pixel-level segmentation map.

3. Key Contributions

A. AquaOV255 Dataset

• The first large-scale, fine-grained underwater segmentation dataset.
• Scale: Contains 20,723 high-resolution images.
• Diversity: Covers 255 distinct categories (including background), ranging from specific fish species (e.g., Zebrafish, Goblin Shark) to invertebrates, corals, and artificial objects (e.g., shipwrecks, plastic debris).
• Annotation: Created via a semi-automatic pipeline using Segment Anything (SAM) and verified by human experts.

B. UOVSBench Benchmark

• A unified benchmark integrating AquaOV255 with five existing underwater datasets (USIS16K, SUIM, MAS3K, USIS10K, DUT-USEG).
• Standardizes these datasets into an Open-Vocabulary Segmentation (OVS) format to enable comprehensive evaluation across diverse scenes and categories.

C. Training-Free Framework

• Demonstrates that high-performance underwater segmentation can be achieved without collecting new training data or fine-tuning models, addressing the "data-hungry" limitation of current deep learning approaches.

4. Experimental Results

The authors evaluated Earth2Ocean on the UOVSBench benchmark against state-of-the-art training-free OVS methods (e.g., ClearCLIP, SCLIP, ProxyCLIP, Trident, CorrCLIP) using various backbones (ViT-B/16, ViT-L/14, ViT-H/14).

• Performance Gains: Earth2Ocean consistently outperformed all baselines.
  • Average Improvement: Achieved an average increase of ~6 mIoU and significant gains in accuracy (aAcc) and mean accuracy (mAcc).
  • Robustness: The method showed particularly strong performance on larger backbones (ViT-H/14), achieving an average mIoU of 49.67% compared to ~29.56% for the previous best (CorrCLIP).
• Fine-Grained Capability: The model excelled in distinguishing rare and specialized categories (e.g., specific fish species), demonstrating superior long-tail recognition capabilities.
• Efficiency: By pre-computing MLLM reasoning outputs and caching them, the framework maintains efficient inference speeds suitable for real-world deployment.

5. Significance and Impact

• Ecological Utility: Provides a scalable solution for monitoring marine biodiversity without the prohibitive cost of data annotation and model retraining.
• Methodological Advancement: Proves that geometric priors and MLLM reasoning can effectively compensate for the domain shift between terrestrial and underwater environments.
• Resource Availability: The release of AquaOV255 and UOVSBench sets a new standard for future research in underwater computer vision, moving the field from coarse, closed-set recognition to fine-grained, open-vocabulary understanding.

In conclusion, Earth2Ocean represents a paradigm shift in underwater segmentation, offering a practical, training-free solution that leverages the power of terrestrial VLMs and MLLMs to solve complex marine monitoring challenges.