An interpretable framework using foundation models for fish sex identification

The paper introduces FishProtoNet, an interpretable, non-invasive computer vision framework leveraging foundation models to accurately identify the sex of endangered delta smelt across various life stages while minimizing stress and mortality risks.

The Gist

Imagine you are a zookeeper trying to tell the difference between male and female fish in a tank. For some species, it's easy: the males are bright red, and the females are blue. But for the Delta Smelt—a tiny, silver fish that is critically endangered and lives in California's waterways—males and females look almost identical to the human eye, especially when they are young.

Traditionally, scientists had to catch these fish, cut them open, or use ultrasound machines to check their insides. This is stressful for the fish and can even kill them, which is a disaster for a species that is already on the brink of extinction.

This paper introduces a new, non-invasive "digital eye" called FishProtoNet. Think of it as a super-smart, transparent robot assistant that can tell male and female smelt apart just by looking at a photo, without ever touching the fish.

Here is how it works, broken down into three simple steps using everyday analogies:

1. The "Cut-Out" Artist (Removing the Clutter)

The Problem: When you take a photo of a fish in a tank, the background is messy. There are tank walls, gravel, shadows, and maybe even a label on the glass. Old computer programs often get confused by this background. They might think, "Oh, this fish is on a blue background, so it must be a male," even if the fish itself looks the same as a female.

The Solution: The researchers used two "foundation models" (which are like giant, pre-trained AI brains that have seen millions of images) called Grounding DINO and SAM2.

• Analogy: Imagine you have a child who is terrible at drawing but great at following instructions. You tell them, "Find the fish." They point to the fish. Then you tell them, "Cut it out perfectly." They use scissors to cut the fish out of the picture, leaving only the fish on a blank white background.
• Result: The AI strips away all the distracting background noise, focusing only on the fish itself. This ensures the computer isn't cheating by looking at the tank instead of the fish.

2. The "Detective's Notebook" (Finding the Tiny Clues)

The Problem: Even after isolating the fish, the differences between a male and female Delta Smelt are incredibly subtle. It's like trying to tell the difference between two identical twins who are wearing the same clothes. A standard AI might just guess.

The Solution: The system uses a powerful "feature extractor" (a ResNet model) to zoom in on microscopic details—tiny differences in body shape, fin texture, or color patterns that a human eye would miss.

• Analogy: Think of this as a detective with a magnifying glass. Instead of just looking at the whole person, the detective looks at the specific way the person holds their hands or the tiny scar on their ear. The AI does the same, scanning the fish for microscopic "clues" that indicate gender.

3. The "Transparent Teacher" (Why did you guess that?)

The Problem: Most AI models are "black boxes." You feed them a picture, and they say "Male." But they can't explain why. If a biologist asks, "Are you sure?", the AI just says, "Trust me." This is dangerous in science because if the AI is wrong, no one knows why.

The Solution: This is the most unique part of the paper. They built the AI using a Prototype Network.

• Analogy: Imagine a teacher trying to teach a student how to identify a "Cat." Instead of just saying "It's a cat," the teacher shows the student three perfect examples of cats (prototypes) and says, "If the new animal looks like this cat, it's a cat. If it looks like that cat, it's a cat."
• How FishProtoNet works: The AI learns "prototypes" (ideal examples) of male and female fish. When it sees a new fish, it doesn't just guess; it compares the new fish to its mental library of "Best Male" and "Best Female" examples.
• The Magic: Because it works this way, the AI can show you exactly which part of the fish it was looking at to make its decision. It can say, "I think this is a male because the shape of its tail matches my 'Male Prototype'." This makes the AI interpretable and trustworthy for scientists.

What Did They Find?

The team tested this system on Delta Smelt at three different life stages:

1. Subadults (Teenagers): The AI struggled here. Just like human teenagers who haven't fully developed their adult features, young fish look too much alike. The AI was only about 55% accurate (barely better than a coin flip).
2. Early Spawning (Young Adults): The AI got much better (74% accurate). As the fish matured, the subtle differences became clearer.
3. Post-Spawning (Mature Adults): The AI performed best here (81% accurate). The physical differences between males and females were most obvious.

Why Does This Matter?

This isn't just about fish; it's about saving a species.

• No Stress: Scientists can now count males and females without hurting or stressing the fish.
• Better Breeding: By knowing the sex ratio easily, they can manage breeding programs more effectively to help the population grow.
• Trust: Because the AI explains how it decided, biologists can trust the results and use them to make real-world conservation decisions.

In a nutshell: The researchers built a smart, non-invasive camera system that acts like a detective with a magnifying glass and a transparent notebook. It cuts out the background, finds tiny clues, and explains its reasoning, helping scientists save the endangered Delta Smelt without ever having to touch them.

Technical Summary

1. Problem Statement

Accurate sex identification in fish is critical for aquaculture management, particularly for endangered species like the Delta Smelt (Hypomesus transpacificus), to optimize breeding strategies and maintain genetic diversity.

• Current Limitations: Traditional methods (gonad biopsy, endoscopy, ultrasound) are invasive, stressful, and can cause mortality, posing risks to threatened populations.
• Computer Vision Challenges: Existing deep learning approaches for non-invasive sex identification face two main hurdles:
  1. Background Bias: Models often memorize background noise (e.g., tank bottoms) rather than learning intrinsic morphological differences between sexes.
  2. Lack of Interpretability: Standard "black box" models (e.g., CNNs) lack transparency, making it difficult for biologists to trust or validate the decision-making process.
  3. Subtle Morphology: In early life stages (subadults), sexual dimorphism is minimal, making visual differentiation extremely difficult.

2. Methodology: FishProtoNet

The authors propose FishProtoNet, a three-stage, non-invasive framework designed to extract fish-specific features and provide interpretable predictions.

A. ROI Extraction via Visual Foundation Models (Zero-Shot)

To eliminate background bias, the framework first isolates the fish from the image without requiring species-specific training data.

• Grounding DINO: Used with the text prompt "fish" to perform zero-shot object detection, generating bounding boxes around the fish.
• SAM2 (Segment Anything Model 2): Uses the bounding boxes from Grounding DINO as visual prompts to generate precise segmentation masks (Regions of Interest - ROIs).
• Benefit: This combination leverages pre-trained foundation models to extract clean fish ROIs, effectively removing background noise without the need for extensive annotated datasets for object detection.

B. Data Augmentation

To address overfitting and improve robustness against variations in fish posture, lighting, and incomplete masks:

• Strategies: Random rotation (±15°), horizontal flipping, color jitter (brightness/contrast/saturation), random cropping, and random erasing.
• Purpose: Random erasing specifically helps the model ignore artifacts like Visible Implant Alphanumeric (VIA) tags and handles cases where the segmentation mask is imperfect.

C. Feature Extraction

• Backbone: A ResNet-50 model (pre-trained on ImageNet) is used to extract high-dimensional semantic features from the augmented fish ROIs.
• Transfer Learning: The model is fine-tuned to capture subtle morphological differences in body shape, coloration, and texture associated with gonadal development.

D. Interpretable Sex Prediction (Prototype Learning)

Instead of a standard fully connected layer, the authors employ a Deep Neural Decision Tree based on prototype learning.

• Mechanism: The model learns "prototypes" (representative feature patterns) for each sex.
• Decision Process: New samples are classified by calculating the Euclidean distance between their features and the learned prototypes within a metric space.
• Interpretability: The decision tree structure allows for a step-by-step visualization of the decision chain. Each node represents a learned prototype, and the path taken explains why a specific sex was predicted (e.g., "Prototype A present, Prototype B absent").

3. Key Contributions

1. Zero-Shot ROI Extraction: Successfully combined Grounding DINO and SAM2 to extract fish ROIs without species-specific training, effectively mitigating background bias.
2. Interpretable AI: Developed a prototype-based neural decision tree that provides transparent, biologically plausible explanations for sex classification, moving beyond "black box" predictions.
3. Robustness via Augmentation: Systematically identified optimal data augmentation strategies (specifically the combination of rotation, flipping, cropping, and erasing) to handle small datasets and imperfect segmentation.
4. End-to-End Framework: Created a complete pipeline for non-invasive Delta Smelt sex identification across different life stages.

4. Results

The model was evaluated on the FCCL Delta Smelt dataset, divided into three sessions: Subadult (Session 1), Early Spawning (Session 2), and Post-Spawning (Session 3).

Stage	Accuracy	F1 Score	Observations
Session 1 (Subadult)	54.76%	54.62%	Performance was low, likely due to negligible morphological differences in immature fish.
Session 2 (Early Spawning)	74.40%	74.27%	Significant improvement as sexual dimorphism becomes visible.
Session 3 (Post-Spawning)	81.16%	79.43%	Highest performance; morphological differences are most pronounced.

• Comparison: FishProtoNet outperformed a standard fine-tuned ResNet-50 across all stages. While a Vision Transformer (ViT) achieved slightly higher accuracy in Sessions 2 and 3, FishProtoNet offered superior interpretability.
• Ablation Study: The combination of all augmentation techniques (Rotation + Flip + Crop + Erasing) yielded the highest accuracy (81.82% on Session 3) compared to baseline or single-augmentation methods.

5. Significance and Future Work

• Conservation Impact: Provides a non-invasive tool for managing endangered Delta Smelt populations, reducing stress and mortality associated with traditional sexing methods.
• Scientific Trust: The interpretability of the model allows biologists to validate that the AI is using actual morphological features (e.g., body shape, fin structure) rather than background artifacts, bridging the gap between AI and domain expertise.
• Limitations: The model struggles with subadult fish where sexual dimorphism is not yet visually distinct.
• Future Directions: The authors suggest integrating multimodal data (e.g., infrared or ultrasound imaging) to capture physiological cues in early life stages and potentially replacing the ResNet backbone with a ViT to improve feature representation capabilities.

Code Availability: The source code is publicly available at: https://github.com/zhengmiao1/Fish_sex_identification.