🤖 AI

MARIS: Marine Open-Vocabulary Instance Segmentation with Geometric Enhancement and Semantic Alignment

This paper introduces MARIS, the first large-scale fine-grained benchmark for underwater open-vocabulary instance segmentation, and proposes a unified framework featuring a Geometric Prior Enhancement Module and a Semantic Alignment Injection Mechanism to overcome visual degradation and semantic misalignment in marine environments.

Original authors: Bingyu Li, Feiyu Wang, Da Zhang, Zhiyuan Zhao, Junyu Gao, Xuelong Li

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingyu Li, Feiyu Wang, Da Zhang, Zhiyuan Zhao, Junyu Gao, Xuelong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a diver exploring the deep ocean. You see a colorful fish, a strange coral, and maybe a lost treasure chest. Now, imagine you have a smart assistant in your helmet that can tell you exactly what everything is.

The problem is, most of these assistants were trained in a library full of books about land animals and everyday objects. If you show them a fish that looks like a rock, or a coral that looks like a brain, they get confused. They might say, "I don't know," or worse, they might guess "It's a rock" when it's actually a rare fish.

This paper introduces MARIS, a new project designed to fix this confusion and teach computers how to "see" the underwater world much better. Here is how they did it, explained simply:

1. The Problem: The "Blurry Glasses" and the "Vague Dictionary"

The authors identified two main reasons why current AI fails underwater:

The Blurry Glasses (Visual Degradation): Underwater is a messy place. The water acts like a dirty filter. Colors fade (reds turn gray), things look hazy, and light scatters. It's like trying to read a book through a foggy window. Because the AI relies on clear colors and sharp edges, it gets lost.
The Vague Dictionary (Semantic Misalignment): The AI's "dictionary" is built on land. It knows what a "dog" or a "car" is. But it doesn't know the difference between a "Blue Parrotfish" and a "Clownfish." It just sees "Fish." It's like having a dictionary that only has the word "Animal" instead of "Lion," "Tiger," and "Bear."

2. The Solution: A New "Underwater School" (The MARIS Dataset)

First, the researchers built a massive new library of underwater pictures called MARIS.

Before: Old datasets were like a photo album with only 20 pages, where everything was just labeled "Fish" or "Plant."
Now: MARIS has over 16,000 photos with 158 specific labels. It doesn't just say "Fish"; it says "Blue Parrotfish," "Hammerhead Shark," or "Sea Slug." This gives the AI a much richer vocabulary to learn from.

3. The Two Super-Powers (The New Framework)

To teach the AI to work in this messy, blurry underwater world, they gave it two special tools:

Tool A: The "Skeleton Tracker" (Geometric Prior Enhancement Module)

Since the water messes up colors, the AI needs to stop relying on paint and start looking at shape.

The Analogy: Imagine trying to recognize a friend in a dark room where you can't see their face or clothes. You can't, but you can recognize them by their height, how they walk, or the shape of their shoulders.
How it works: This tool teaches the AI to ignore the fading colors and focus on the geometry (the shape and structure). Even if a fish's color is washed out, its body shape and fin structure remain stable. The AI learns to "feel" the shape of the object to know what it is, even when the water is murky.

Tool B: The "Underwater Translator" (Semantic Alignment Injection Mechanism)

Since the AI's dictionary is too land-focused, they had to translate it into "underwater language."

The Analogy: If you ask a land-dwelling AI, "What is a fish?", it thinks of a goldfish in a bowl. But in the ocean, a fish might be swimming in a coral reef, in low light, or near a shipwreck.
How it works: This tool adds "context clues" to the AI's thinking. It tells the AI: "When you see a fish, remember it might be in 'low visibility,' 'near a reef,' or 'in deep water.'" It enriches the AI's understanding so it doesn't get confused by the weird lighting or the specific environment. It acts like a translator that explains the underwater scene to the AI in terms it can understand.

4. The Result: A Smarter Diver

When they tested this new system:

It got much better at identifying things: It could tell the difference between similar-looking fish that confused other models.
It handled the "unknown": Because it learned the concept of shapes and underwater context, it could guess what new, unseen creatures were, even if it had never seen them before.
It worked across different oceans: It didn't just work in one specific type of water; it generalized well, meaning it could handle different levels of murkiness and light.

In a Nutshell

Think of MARIS as giving a computer a pair of specialized underwater goggles (that ignore color and focus on shape) and a new underwater encyclopedia (that understands the specific context of the ocean). This allows the computer to finally "see" and understand the mysterious world beneath the waves, helping us explore, protect, and study the ocean like never before.

1. Problem Statement

The paper addresses the critical limitations of existing underwater instance segmentation methods, which are primarily constrained by close-vocabulary prediction. Traditional models rely on a fixed set of training categories, making them unable to recognize novel marine species or adapt to diverse exploration scenarios without exhaustive retraining.

Furthermore, the direct transfer of Open-Vocabulary (OV) segmentation models (successful in natural images) to underwater environments faces three specific challenges:

Data Scarcity and Coarseness: Existing underwater datasets (e.g., UIIS, USIS10K) contain fewer than 20 categories, often grouping diverse organisms into broad, coarse classes (e.g., "fish" or "plants"), which hinders fine-grained OV learning.
Visual Degradation: Underwater images suffer from severe optical distortions, including color attenuation, low contrast, and light scattering. These factors render standard visual appearance cues unstable, causing models to fail when object colors blend with the background.
Semantic Misalignment: Vision-Language Models (VLMs) are typically trained on terrestrial data. They lack domain-specific knowledge regarding marine environments, leading to semantic ambiguity and an inability to distinguish between fine-grained underwater species.

2. Methodology

The authors propose MARIS, a unified framework designed to tackle these challenges through two complementary modules: Geometric Prior Enhancement Module (GPEM) and Semantic Alignment Injection Mechanism (SAIM).

A. The MARIS Benchmark

Before detailing the model, the authors introduce MARIS, the first large-scale, fine-grained benchmark for underwater OV instance segmentation.

Scale: Comprises over 16,000 images with 158 fine-grained subclasses (e.g., distinguishing specific fish species like "Blue Parrotfish" vs. "Bicolor Angelfish") across 9 super-classes.
Split: Divided into training (43 exclusive classes), validation (74 exclusive classes), and overlapping (41 classes) sets to rigorously test In-Domain and Cross-Domain generalization.

B. The Unified Framework

The architecture processes an input image $I$ and text descriptions $C$ to generate instance masks $M$ and labels $Y$ .

Geometric Prior Enhancement Module (GPEM):
- Goal: To counteract visual degradation by leveraging stable structural cues that persist even when color and texture are distorted.
- Mechanism: It fuses multi-scale visual features from a frozen CLIP encoder with geometric features (depth maps) generated by a frozen depth encoder (e.g., Depth Anything).
- Fusion: Uses a multi-scale deformable attention mechanism and an adaptive weighting strategy to inject geometric priors into visual features. This ensures that object consistency is maintained based on shape and structure (e.g., fin structures, coral growth patterns) rather than unreliable color cues.
Semantic Alignment Injection Mechanism (SAIM):
- Goal: To resolve semantic ambiguity caused by the lack of underwater-specific definitions in pre-trained VLMs.
- Mechanism:
  - Prompt Engineering: Introduces underwater-aware textual prompts that encode five dimensions: environmental context, water medium/visibility, illumination, depth cues, and scene interactions.
  - Adaptive Template Selection: Instead of using all prompts, the model computes similarity between visual features and text templates, selecting the top- $N$ most reliable templates to reduce noise in degraded conditions.
  - Global Priors: Fuses global depth tokens with mask features to enrich category representations, bridging the gap between terrestrial VLM knowledge and marine semantics.

3. Key Contributions

New Benchmark (MARIS): The first large-scale, fine-grained dataset for underwater OV instance segmentation, addressing the coarse-grained limitations of prior datasets.
Novel Framework: A unified approach combining GPEM (to handle visual degradation via geometric stability) and SAIM (to handle semantic ambiguity via domain-specific language priors).
Comprehensive Evaluation: Extensive experiments demonstrating state-of-the-art performance in both In-Domain and Cross-Domain (trained on COCO, tested on MARIS) settings.

4. Experimental Results

The framework was evaluated on the MARIS dataset using ConvNeXt-B and ConvNeXt-L backbones, comparing against strong baselines like OVSeg, ODISE, SAN, and FCCLIP.

In-Domain Performance:
- With ConvNeXt-L, MARIS achieved 61.55 mAP on intersection classes and 54.02 mAP on Open-Vocabulary classes.
- This represents a significant improvement over the second-best method (MAFT+), surpassing it by 6.23 mAP (Intersection) and 2.48 mAP (OV).
- Notable gains were observed in AP75, indicating more precise and robust mask predictions.
Cross-Domain Performance:
- Models trained on the terrestrial COCO dataset and tested on MARIS showed a natural performance drop due to the domain gap.
- However, MARIS still achieved the best results, reaching 46.18 mAP (ConvNeXt-L), outperforming the second-best by 5.91 mAP. This highlights the framework's ability to generalize from natural scenes to underwater environments.
Ablation Studies:
- Removing either GPEM or SAIM resulted in performance drops, confirming their complementary nature.
- The adaptive template selection strategy in SAIM was proven crucial for handling diverse underwater conditions.
- The method maintained high efficiency, with significantly fewer trainable parameters compared to other SOTA methods.

5. Significance

This work represents a pivotal step forward in underwater computer vision:

Bridging the Gap: It successfully adapts Open-Vocabulary learning to the challenging underwater domain, moving beyond the limitations of fixed-category models.
Robustness: By leveraging geometric priors, the method provides a solution to the inherent instability of visual appearance in turbid or low-light underwater environments.
Foundation for Future Research: The introduction of the MARIS dataset establishes a rigorous standard for evaluating fine-grained marine perception, enabling future advancements in autonomous underwater vehicles (AUVs), marine biodiversity monitoring, and environmental conservation.

In conclusion, MARIS provides a robust methodology and benchmark that effectively tackles the dual challenges of visual degradation and semantic misalignment in underwater open-vocabulary instance segmentation.