💻 computer science

A Self-Supervised Approach for Minimal-Annotation Hydroacoustic Data Exploration

This paper presents a self-supervised pipeline using Masked AutoEncoders and clustering algorithms to efficiently explore and classify large-scale, minimally annotated hydroacoustic data, successfully identifying diverse marine and anthropogenic signals in a multi-year dataset from Mayotte Island.

Original authors: Pierre-Yves Raumer, Axel Marmoret, Dorian Cazau, Anatole Gros-Martial, Richard Dreo, Maelle Torterotot, Sara Bazin, Flore Samaran, Jean-Yves Royer

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pierre-Yves Raumer, Axel Marmoret, Dorian Cazau, Anatole Gros-Martial, Richard Dreo, Maelle Torterotot, Sara Bazin, Flore Samaran, Jean-Yves Royer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a room that never stops talking. This room is the deep ocean, and the "talking" is a massive, continuous recording of sounds made by whales, volcanoes, ships, and earthquakes. The problem? The recording is so huge and so noisy that a human detective would need to listen to it for centuries just to find the interesting parts. Usually, scientists have to manually label every single sound they hear, which is like trying to sort a mountain of mixed-up Legos by hand.

This paper introduces a new, super-smart robot detective that doesn't need a manual instruction book to start working. It uses a "self-supervised" approach, which is like giving the robot a giant box of mixed-up Lego bricks and saying, "Figure out how these pieces fit together by trying to rebuild the missing parts."

The Robot's Training: The "Fill-in-the-Blanks" Game
The robot, called a Masked AutoEncoder (or MAE), starts by playing a game of "fill-in-the-blanks." The scientists take a picture of the sound (called a spectrogram, which looks like a colorful map of frequencies over time) and cover up 50% of it with a black mask. The robot's job is to look at the visible parts and guess what the hidden parts look like. To do this, it has to learn the secret patterns of the ocean: how a whale call looks, how a ship engine hums, and how an earthquake rumbles. It practices this on a massive dataset of recordings from the Indian Ocean, spanning from 2009 to 2024, covering 34 different locations.

The Big Trick: Sorting the Messy Room
Here is where the paper makes a crucial move. Many other methods try to look at the whole room at once and say, "This whole hour sounds like a whale." But in the ocean, sounds often overlap—a whale might sing while a ship passes by. If you look at the whole room, you just get a messy mix.

This paper argues that looking at the whole room is a bad idea. Instead, their robot breaks the sound map into tiny 16x16 patches. It looks at each patch individually. If a patch is just background noise (like static), the robot ignores it. If a patch has a sound, the robot groups nearby sound-patches together into a single "event." This is like sorting the Legos by color and shape before trying to build the castle. By separating the overlapping sounds into individual events first, the robot can tell the difference between a whale and a ship even if they happen at the same time.

The Discovery: Finding Patterns in the Chaos
Once the robot has sorted all these tiny sound events from the entire multi-year dataset, it uses a special map-making tool (called UMAP) and a clustering algorithm (HDBSCAN) to group similar events together.

The results are impressive. The robot found 317 distinct clusters of sound. A human expert then spent less than one hour looking at these groups and labeling them. They found:

Known signals: Antarctic Blue Whales, Fin Whales, Minke Whales, and even the rare Omura's Whale. They also found seismic signals from the Fani Maoré volcano and ship noise.
Unknown signals: The robot found five types of sounds that no one had fully characterized before, including a mysterious "8-second pulse" and a "42 Hz signal."

The paper suggests that this method is highly effective at finding seasonal patterns. For example, it correctly spotted that Fin Whale calls disappear in certain years (2021, 2024, and 2025) and that seismic activity decreases over time, just as scientists expected.

How Good Is It? (The Reality Check)
The researchers didn't just say "it works"; they tested it. They compared their robot against two existing, specialized detectors (one based on whale call patterns and one based on a supervised AI model called YOLO).

The Verdict: The robot's performance was comparable to these existing detectors. It achieved similar accuracy in finding whales and earthquakes.
The Catch: The paper is careful to note that while the robot is great at finding patterns and seasonal trends, it isn't perfect at identifying every single sound with 100% precision. It's an "exploration tool," not a magic wand that solves everything instantly.
What They Ruled Out: The paper explicitly argues against methods that try to analyze the whole audio window at once without breaking it down into events first. They found that simply averaging all the sound patches together made the robot much worse at its job. They also found that skipping the initial "training" phase (pre-training on a huge dataset) made the robot significantly less accurate, proving that the "fill-in-the-blanks" practice is essential.

The Bottom Line
This paper suggests that we don't need to spend years manually labeling ocean sounds to understand them. By teaching a computer to play "fill-in-the-blanks" and then letting it sort the sounds into little event-piles, we can quickly uncover the secrets of the ocean. The robot found known whales, tracked volcanic eruptions, and even discovered new, mysterious sounds that humans hadn't fully understood yet—all with just one hour of human help. It's a powerful new way to listen to the ocean, turning a mountain of noise into a clear map of life and geology.

Technical Summary: A Self-Supervised Approach for Minimal-Annotation Hydroacoustic Data Exploration

Problem Statement
Passive hydroacoustic monitoring generates vast volumes of continuous low-frequency recordings, yet only a fraction is explored due to the high cost of manual annotation. Existing methodologies face a trade-off: manual annotation is time-consuming and user-dependent, while supervised automatic detectors require large labeled datasets that are rarely available for rare signals or understudied environments. Furthermore, traditional automated paradigms (both supervised and unsupervised) often rely on prior knowledge to target specific, structured signals, confining exploration to predefined targets and hindering the discovery of unknown hydroacoustic events.

Methodology
The authors propose a self-supervised exploration pipeline designed to identify recurring hydroacoustic patterns with minimal manual annotation. The workflow consists of four main stages:

Masked AutoEncoder (MAE) Pre-training:
- A Vision Transformer (ViT) is trained within a Masked AutoEncoder framework on a large, diverse dataset of low-frequency hydroacoustic spectrograms (compiled from Indian Ocean networks, 2009–2024).
- The model learns to reconstruct masked portions of spectrograms, forcing it to learn the underlying structure of acoustic environments (e.g., stereotyped marine mammal calls).
- The architecture uses a lightweight ViT encoder (16 layers) and decoder (6 layers) adapted for single-channel grayscale spectrograms.
In-Domain Specialization:
- The pre-trained model is fine-tuned on a smaller target dataset (MAHY*2, near Mayotte Island) to specialize in the specific acoustic characteristics of the target environment.
Event-Level Extraction and Disentanglement:
- Unlike methods that generate a single embedding per spectrogram window, this pipeline extracts patch-level embeddings from the MAE encoder.
- Patch Discarding: "Empty" patches (low energy or low activation) are filtered out to remove background noise.
- Spectrogram-Level Clustering: Remaining adjacent patches are merged into "events" based on Chebyshev distance (merging if separated by $\le$ 2 patches). This step disentangles overlapping sources within a single time window.
- A representative embedding vector is selected for each event (the patch closest to the mean embedding of the event).
Dataset-Level Clustering and Mapping:
- Event-level embeddings are clustered at the dataset scale. Two strategies are compared:
  - MAE-K-Means: K-Means followed by agglomerative hierarchical clustering.
  - MAE-UMAP: Dimensionality reduction via UMAP followed by HDBSCAN.
- Cluster-to-Class Mapping: A lightweight manual inspection phase (approx. 1 hour) maps the resulting clusters to semantic classes (e.g., specific whale species, seismic phases, or noise) based on visual similarity of representative spectrograms.

Key Contributions

Adaptation of MAE: Successfully adapting the Masked AutoEncoder framework to low-frequency hydroacoustic spectrograms for patch-level representation learning.
Event-Level Disentanglement: Introducing a strategy to cluster adjacent patches within individual spectrograms before dataset-level clustering, addressing the challenge of overlapping sources common in low-frequency data.
Minimal-Annotation Pipeline: Validating a practical pipeline on a multi-year dataset containing biological, geophysical, and anthropogenic signals, requiring only limited manual inspection to generate interpretable results.

Results
The pipeline was evaluated on the MAHY*2 dataset, which includes marine mammal vocalizations (e.g., Antarctic blue whales, fin whales), seismo-volcanic signals, and anthropogenic noise.

Clustering: The method identified 317 clusters. Manual mapping grouped these into 15 hydroacoustic classes (or noise) in less than one hour.
Temporal Patterns: The clusters successfully reconstructed known seasonal patterns for marine mammals (e.g., bi-annual peaks for South Western Indian Ocean pygmy blue whales) and the temporal decay of seismic sequences. It also identified patterns for previously unstudied signals (e.g., "Und. 42 Hz" and "LF 8 sec pulse").
Classification Performance: When used as a classifier (by aggregating event detections into hourly presence/absence scores), the method achieved performance comparable to two existing detectors: a periodicity-based detector (Dréo et al., 2025) and a supervised YOLO-based object detector (BioDCASE baseline).
- The MAE-K-Means approach generally outperformed MAE-UMAP and ablated variants (e.g., averaging embeddings or selecting max-norm patches).
- Pre-training was found to be critical for optimal clustering accuracy, though the model could still function without it for specific classes.

Significance and Claims
The paper claims that this self-supervised, event-level clustering strategy provides a general methodology for exploring large hydroacoustic datasets with minimal annotation effort. Its primary significance lies in its ability to:

Approximate Detection: Construct approximate detectors for known signals without requiring large labeled training sets.
Discover Unknowns: Identify and characterize recurring acoustic patterns that were previously unknown or poorly documented, without relying on prior knowledge of the signals.
Scalability: Enable the rapid construction of long-term activity time series from unexplored archives, facilitating the extraction of first-order information from passive hydroacoustic monitoring networks.

The authors emphasize that while per-window classification accuracy may be imperfect, the aggregated cluster occurrences provide reliable estimates of seasonal hydroacoustic activity, making the method a valuable tool for exploratory analysis where labeled data is scarce.

Technical Summary: A Self-Supervised Approach for Minimal-Annotation Hydroacoustic Data Exploration

More like this