Technical Summary: ShellfishNet

Problem Statement

The management of coastal ecosystems and the conservation of marine biodiversity rely heavily on the accurate identification of benthic organisms, particularly shellfish. Traditional monitoring methods are labor-intensive, requiring specialized taxonomic expertise for morphological identification, which creates a scalability bottleneck for broad ecological assessments. While artificial intelligence (AI) and computer vision offer potential for automated monitoring, existing marine datasets are ill-suited for real-world deployment. Current benchmarks often feature samples captured under controlled, ideal conditions, failing to account for the complexities of underwater and coastal environments, such as variable lighting, water turbidity, diverse species postures, and complex habitat backgrounds. Furthermore, general-purpose foundation models often exhibit "domain knowledge deficits" when confronted with the high intra-class variance and minute inter-class variance characteristic of marine molluscs. There is a critical lack of high-quality, domain-specific benchmarks that reflect the visual variability of practical ecological monitoring.

Methodology

To address these gaps, the authors constructed ShellfishNet, a comprehensive domain-specific benchmark designed for the visual recognition and description of marine molluscs.

Dataset Construction

• Scope: The dataset comprises 8,691 images spanning 32 distinct taxa of shellfish commonly encountered in coastal conservation, aquaculture, and monitoring.
• Data Sources: Images were aggregated through a combination of controlled field photography (standardized specimen shots against clean backgrounds) and web scraping (in-the-wild imagery).
• Curation: A multi-stage curation process removed irrelevant or degraded samples. Taxonomic labels were verified against visible morphological evidence and cross-checked by multiple annotators to minimize label noise.
• Real-World Complexity: To simulate the physical constraints of real-world monitoring, the authors applied an offline data augmentation pipeline. This included geometric transformations (random rotation, cropping) to mimic varying viewpoints and photometric adjustments (lighting variations, blur) to simulate weather conditions and water turbidity.
• Multimodal Annotation: A curated subset of 500 images was annotated with descriptive captions by human experts, assisted by a black-box commercial LLM. These captions were structured to cover four professional dimensions: Morphology, Color & Texture, Anatomical Features, and Environment & Background. Human experts maintained full control to eliminate hallucinations.

Benchmarking Framework

The authors established two primary benchmark tasks:

1. Shellfish Classification: A systematic evaluation of 80 representative neural network models, including Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), State Space Models (SSMs), Self-Supervised Learning (SSL) frameworks, and Fine-Grained Visual Categorization (FGVC) models.
2. Shellfish Image Captioning: An evaluation of 13 state-of-the-art Multimodal Large Language Models (MLLMs) (including Qwen, LLaVA, InternVL, IDEFICS2, Gemini, and GPT variants) on the curated captioning subset.

Evaluation Metrics

• Classification: Accuracy was measured on clean data and under 15 types of image corruptions (noise, blur, weather, digital distortions) to assess robustness. Long-tail performance was also analyzed for the five least frequent classes.
• Captioning: Evaluation utilized a multi-dimensional approach:
  • Lexical/Syntactic: BLEU, METEOR, ROUGE-L, CIDEr (noted to suffer from "long-text penalties").
  • Semantic Equivalence: BERTScore (Precision, Recall, F1) to measure deep semantic similarity.
  • Cross-Modal Alignment: CLIP Score to verify visual grounding.
• Robustness: Mean Performance under Corruption (mPC) and Mean Corruption Error (mCE) were calculated relative to a ResNet-50 baseline.

Key Results

Classification Performance

• Architectural Superiority: Hierarchical Vision Transformers (e.g., MViTv2-T at 95.66%) and modern macro-architectural designs (e.g., MambaOut at 96.47%) demonstrated superior performance over early isotropic models (e.g., ViT at 88.35%) and legacy CNNs (e.g., VGG at 89.43%).
• CNN Evolution: Modernized CNNs (e.g., ConvNeXt, ConvFormer-S18) effectively narrowed the performance gap with Vision Transformers, achieving accuracies around 95.12%.
• Robustness: Under severe visual degradations (corruptions), models like MViTv2-T (mPC 72.19%) and MambaOut (mPC 69.73%) exhibited the highest absolute robustness. In contrast, legacy models like VGG suffered catastrophic failures (e.g., accuracy dropping to 7.37% under impulse noise). Notably, some Self-Supervised Learning models (e.g., MoCoV3) showed higher clean accuracy but significantly lower robustness compared to their supervised counterparts.

Captioning Performance

• Semantic Alignment: Proprietary models Gemini 3.1 Pro (BERTScore-F1: 0.9006) and GPT-5.4 (0.8880) achieved the highest semantic equivalence. Among open-weight models, Qwen2.5-7B (0.8750) and InternVL2.5-4B (0.8712) performed best.
• Metric Limitations: Traditional n-gram metrics (BLEU, CIDEr) yielded universally low scores across all models (e.g., CIDEr < 0.11 for top models), highlighting a severe "long-text penalty" and evaluation distortion. The authors conclude that semantic metrics like BERTScore are more reliable for evaluating detailed ecological descriptions.
• Failure Modes: Qualitative analysis revealed that even high-performing models suffer from visual hallucinations (e.g., claiming shells are empty when they contain tissue) and logical errors (e.g., counting hallucinations like "five shells in a 2x2 grid").

Significance and Contributions

The paper positions ShellfishNet as a critical resource for advancing intelligent monitoring in marine ecology. Its primary contributions are:

1. A Comprehensive Multi-Scenario Benchmark: ShellfishNet provides a data foundation that reflects the full spectrum of visual complexities in coastal management, from natural field observations to aquaculture and post-harvest handling. It addresses the scarcity of domain-specific data for fine-grained benthic recognition.
2. Detailed Model Evaluation: By establishing robust baselines across 80 diverse architectures, the work clarifies which model paradigms (hierarchical attention, modern macro-architectures) are best suited for fine-grained marine recognition and robustness against environmental degradation.
3. Exploration of Multimodal Ecological Reporting: The inclusion of a captioning subset facilitates research into vision-language alignment for ecological tasks. It demonstrates the potential of MLLMs for generating detailed environmental reports while highlighting current limitations in visual reasoning and hallucination control.

The authors conclude that while current models show promise, the field requires continued development to mitigate domain-specific knowledge deficits and visual hallucinations. They identify the on-device deployment of lightweight models on edge sensors as a promising future direction for continuous, low-cost ecological monitoring.