💻 computer science

How many labels do you need? A decision framework for cross-habitat marine species recognition

This paper presents a decision framework demonstrating that pairing a frozen foundation model (DINOv2) with a simple linear classifier and annotating only 10–20 images per species enables reliable, scalable cross-habitat marine species recognition, significantly reducing the labeling effort required compared to traditional fine-tuning approaches.

Original authors: Alzayat Saleh, Mostafa Rahimi Azghadi

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alzayat Saleh, Mostafa Rahimi Azghadi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a marine biologist trying to count fish and corals in the ocean. In the past, you had to hire experts to look at thousands of underwater photos one by one, a slow and expensive process. Now, we have computers that can do this automatically. But there's a catch: a computer trained to spot fish in the sunny, clear waters of the Great Barrier Reef often gets confused when it sees murky, cold water in a Danish fjord. It's like teaching a student to recognize a "dog" only by looking at Golden Retrievers in a park; when they see a Poodle in a snowstorm, they might not know what it is.

This paper asks a simple but crucial question: "How many new photos do we actually need to take and label to teach our computer to work in a new ocean location?"

Here is the breakdown of their findings, using some everyday analogies:

1. The Problem: The "Local Guide" vs. The "Universal Expert"

Think of traditional computer models (like ResNet or EfficientNet) as local guides. If you train a local guide in a tropical reef, they learn the specific colors of the sand, the shade of the water, and the background texture. They get really good at that specific spot. But if you take them to a different reef with different water color, they get lost because they were memorizing the background, not the animals.

The authors tested Foundation Models (specifically one called DINOv2). Think of these as universal experts. They were trained on hundreds of millions of images from all over the internet. They didn't just memorize backgrounds; they learned the fundamental "shape," "texture," and "structure" of things. They know what a fish looks like because of its fins and body shape, regardless of whether the water is blue, green, or murky.

2. The Experiment: Testing the "Universal Expert"

The researchers set up a massive test across five different marine datasets, ranging from tropical coral reefs to cold, temperate fjords. They tried four different ways to adapt these models to new sites:

The "Frozen Brain" (Linear Probe): They took the universal expert's brain, froze it so it couldn't learn anything new, and just added a simple "yes/no" switch on top.
The "Fine-Tuning" (Full Fine-tuning): They let the whole brain relearn everything from scratch using the new photos (the old, expensive way).
The "Light Tweaks" (LoRA & VPT): They made tiny, specific adjustments to the brain's internal wiring.

The Big Surprise:
The "Frozen Brain" approach won. By simply freezing the massive, pre-trained model and adding a tiny, simple switch (only 1,538 trainable parameters), it performed just as well as the models that were allowed to relearn everything (which required millions of parameters).

The Analogy: It's like having a master chef who has cooked every dish in the world. You don't need to send them to culinary school again to learn how to cook a specific local soup. You just need to tell them, "Okay, today we are making soup," and they can do it immediately because they already know what soup is.

3. The Answer: How Many Photos Do You Need?

This is the most practical part of the paper. The researchers wanted to know: If I go to a new reef, how many fish do I need to label for the computer to get it right?

The Old Way: You might need thousands of photos to train a model from scratch.
The New Way: You only need 10 to 20 photos per species.

The Metaphor: Imagine you are showing a friend a new type of fruit.

If you show them zero photos, they might guess wrong (Zero-shot learning).
If you show them one photo, they might get it right 50% of the time.
If you show them 10 to 20 photos, they can reliably identify that fruit in any lighting or angle.

The study found that with just 10–20 labeled images per species, the "Frozen Brain" model (DINOv2) could achieve high accuracy at a new site. This reduces the work for human experts by about 90% (an order of magnitude). Instead of spending weeks labeling, you might only need 1 to 4 hours per site.

4. Why Does This Work?

The researchers looked inside the "brain" of the models to see what they were learning.

Traditional Models: Learned to recognize "Green Water" or "Sandy Bottom." When the water changed, they failed.
Foundation Models: Learned to recognize "Fish Shape" or "Coral Structure." They ignored the water color and focused on the animal itself.

The Bottom Line

If you are running a marine monitoring program and want to expand to a new location:

Don't try to retrain a massive model from scratch.
Use a pre-trained "Universal Expert" model (DINOv2).
Freeze its brain (don't let it change).
Just show it 10 to 20 photos of the new animals you want to count.
Add a simple "switch" to tell it what to look for.

This approach is fast, cheap, and works even when moving from tropical reefs to cold, murky fjords. It turns a months-long project into a few hours of work.

Technical Summary: A Decision Framework for Cross-Habitat Marine Species Recognition

Problem Statement
Ecological monitoring programmes, particularly those involving coral reefs, generate vast quantities of imagery (e.g., >100,000 images annually for the Great Barrier Reef) that require expert annotation for species identification. Human annotation is a bottleneck, limited by throughput (50–100 images/hour) and inter-observer variability. While deep learning models have achieved high accuracy (>90%) on specific, single-site datasets, they suffer from a critical limitation: poor generalisation to new sites. Models trained on one habitat often fail when deployed elsewhere because they learn "habitat-specific shortcuts" (e.g., water colour, background texture) rather than species-diagnostic features. Consequently, ecologists lack evidence-based guidance on the labour cost (number of labelled images) required to deploy reliable automated recognition at a new, unseen site.

Methodology
The authors constructed a systematic cross-ecosystem benchmark and evaluation framework to address this gap.

Datasets: The study utilised five publicly available marine datasets spanning three oceans and three taxonomic groups (fish, corals, invertebrates):
- DeepFish: 20 distinct reef habitats along the Great Barrier Reef (source for most transfers).
- AQUA20: Mixed tropical/subtropical marine species.
- Moorea Corals: Benthic quadrats from French Polynesia.
- Coralscapes: Red Sea benthic segmentation.
- Brackish: Cold temperate fjord imagery from Denmark.
  These datasets define a "transfer difficulty gradient" ranging from within-habitat transfer (easiest) to cross-taxon, cross-ocean transfer with severe class imbalance (hardest).
Models: Four backbone architectures were evaluated:
- Foundation Models: DINOv2-Base (self-supervised ViT) and CLIP-Base (language-image ViT).
- CNN Baselines: ResNet-50 and EfficientNet-B4 (supervised ImageNet pretraining).
Adaptation Strategies: The models were tested under four adaptation regimes:
- Linear Probe: Frozen backbone with a trainable linear classifier.
- Low-Rank Adaptation (LoRA): Parameter-efficient fine-tuning of attention layers.
- Visual Prompt Tuning (VPT): Learnable tokens prepended to transformer layers.
- Full Fine-tuning: Updating all parameters (applied to CNN baselines).
Experimental Protocols:
1. Protocol A (Within-Habitat): Leave-one-habitat-out evaluation across 20 DeepFish sites to test generalisation within a single reef system.
2. Protocol B (Cross-Dataset): Open-set transfer from DeepFish to the four other datasets to measure performance degradation across geographic and ecological boundaries.
3. Protocol C (Few-Shot): Systematic variation of labelled target samples per class ( $k=0$ to $100$) to determine the minimum annotation effort required for reliable deployment.

Key Results

Superiority of Frozen Foundation Models: In within-habitat transfer (Protocol A), a frozen DINOv2-Base backbone with a linear classifier (only 1,538 trainable parameters) achieved the highest mean macro F1 (0.70). This performance was statistically comparable to fully fine-tuned CNNs (EfficientNet-B4, 17.6M parameters) and parameter-efficient methods like LoRA, despite using four orders of magnitude fewer trainable parameters.
Feature Analysis: t-SNE visualisations revealed that DINOv2 features cluster by species identity regardless of habitat, indicating the learning of habitat-invariant representations. Conversely, CNN baselines (ResNet-50) clustered by habitat, confirming they rely on background cues that fail in new environments.
Cross-Habitat Robustness: Foundation models degraded more gracefully than CNNs when transferring across ecosystems. While performance dropped along the difficulty gradient, DINOv2 maintained utility in cross-taxon transfers where CNNs approached chance levels.
Few-Shot Efficiency (Protocol C): The study quantified the "labelling effort vs. accuracy" trade-off.
- Zero-shot classification (CLIP, $k=0$ ) provided a baseline but was limited by vocabulary mismatches in taxonomic labels.
- Adding just 10–20 labelled images per species was sufficient to deploy reliable recognition at new sites.
- DINOv2 with a linear probe reached 80% of its maximum balanced accuracy with approximately 11 labelled samples per class, significantly outperforming other configurations in sample efficiency.

Significance and Claims
The paper claims to provide the first systematic, evidence-based decision framework for cross-habitat marine species recognition. Its primary contributions are:

A Benchmark: A multi-dataset evaluation spanning tropical to temperate environments and fish to coral taxa, establishing a difficulty gradient for future research.
Operational Guidance: The finding that ecologists can deploy reliable automated systems at new sites by pairing a frozen, open-source foundation model (DINOv2) with a simple linear classifier and annotating only 10–20 images per species. This reduces annotation effort by roughly an order of magnitude compared to conventional training-from-scratch approaches.
Methodological Insight: The demonstration that foundation models learn species-diagnostic, habitat-invariant representations, whereas traditional CNNs encode habitat-specific shortcuts. This suggests that for ecological monitoring, the "visual experience" acquired during large-scale self-supervised pretraining is more valuable than task-specific fine-tuning on small, site-specific datasets.

The authors conclude that this framework allows monitoring programmes to budget labelling effort against expected accuracy when expanding across sites, ecosystems, or imaging platforms, applicable to any ecological monitoring system requiring generalisation with limited labelled data.

1. The Problem: The "Local Guide" vs. The "Universal Expert"

2. The Experiment: Testing the "Universal Expert"

3. The Answer: How Many Photos Do You Need?

4. Why Does This Work?

The Bottom Line

More like this