🧬 biology

YAPAT — An Open-Source Platform for Active and Weakly Supervised Annotation of Passive Acoustic Monitoring Data

YAPAT is an open-source, web-based platform that streamlines the large-scale annotation of passive acoustic monitoring data by integrating ontology-grounded labeling, multi-label active learning, and dual cold-start mechanisms to reduce expert effort and produce high-quality, interoperable datasets.

Original authors: Thiago S. Gouvêa, Rida Saghir, Prathmesh P. Doddanawar, Novruz Mammadli, Pratik P. Sitapara, Hannes Kath, Daniel Sonntag

Published 2026-07-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thiago S. Gouvêa, Rida Saghir, Prathmesh P. Doddanawar, Novruz Mammadli, Pratik P. Sitapara, Hannes Kath, Daniel Sonntag

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine the natural world as a giant, 24-hour radio station that never stops broadcasting. Instead of music, it plays the songs of frogs, birds, insects, and whales. Scientists use special microphones, called Passive Acoustic Monitoring (PAM) units, to tape-record these sounds for days, weeks, or even years. This is a superpower for understanding nature because it lets researchers listen to entire ecosystems without ever stepping foot in them or scaring the animals away. However, there's a massive problem: these microphones record so much audio that the files pile up into a mountain too huge for any human to listen to and write down what they hear. It's like trying to read every single book in a library the size of a city, but the books are made of sound.

To solve this, scientists use computers to help listen. But computers are only as smart as the examples they are taught with. To teach a computer to recognize a specific frog call, a human expert first has to listen to thousands of recordings and label them, saying, "Yes, this is a frog," or "No, this is just wind." This process is slow, boring, and expensive. If the experts are too busy labeling, the computers can't learn, and the mountains of audio data sit useless. The big question in this field is: How can we get computers to learn from these massive sound archives without needing a human to listen to every single second first?

This is where a new tool called YAPAT (Yet Another PAM Annotation Tool) comes in. Think of YAPAT as a high-tech, interactive detective game for nature sounds. Instead of forcing a human to listen to every recording one by one, YAPAT uses a clever mix of computer tricks to do the heavy lifting. It starts by breaking the long audio files into tiny 3-second clips. Then, it uses a pre-trained "brain" (called BirdNET) to turn each clip into a unique digital fingerprint.

The magic happens in how YAPAT helps the human expert. Imagine you are looking at a giant map of all these sound fingerprints. YAPAT doesn't just show you a random list; it acts like a smart guide. It has three main ways to get you started, even if you have zero labeled data to begin with:

The "Look-alike" Search: If you have one clear recording of a specific animal, you can upload it. YAPAT instantly finds all the other 3-second clips in your massive archive that sound similar, letting you label a whole bunch of them at once.
The "File-Level" Guess: If you have a folder of recordings where you only know which animal was in the file (but not exactly when), YAPAT can use a technique called "weak supervision" to guess where in the file the animal is singing. It's like looking at a photo of a party and guessing who is dancing based on the crowd's energy, even if you didn't film the dance floor directly.
The "Smart Sort": Once the computer starts learning, it stops showing you random clips. Instead, it uses a special scoring system to show you the clips it is most confused about, or the ones that are the most different from what it already knows. This ensures the human expert spends their time only on the clips that will teach the computer the most, rather than wasting time on easy or boring examples.

YAPAT also connects these sound labels to official scientific dictionaries (ontologies), ensuring that when a scientist says "tree frog," it means the exact same thing to everyone else in the world, making the data reusable for future research.

The paper presents YAPAT as a web-based platform that brings all these features together in one place. The authors tested the system on a real dataset called AnuraSet, which contains over 30,000 sound clips of frogs. They found that the system works fast enough to be interactive. For example, searching for similar sounds takes less than a second, even with thousands of clips. Training the computer to recognize new patterns takes only a few seconds on a standard computer, or even faster on a powerful one. The system is designed to be open-source, meaning anyone can download it, set it up on their own server, and start organizing their own sound archives.

The authors are careful to note that while YAPAT is a powerful tool, it isn't a magic wand that solves everything instantly. It relies on the "BirdNET" brain, which was originally trained on birds, so it might not be perfect for every type of animal (like bats or whales) without some extra tweaking. Also, the system currently cuts sounds into fixed 3-second chunks, which means it can't pinpoint the exact millisecond a sound starts or stops if it happens to fall on the edge of a cut. However, by combining smart searching, weak supervision, and active learning, YAPAT successfully bridges the gap between massive, unmanageable audio archives and the high-quality, labeled data scientists need to understand our planet's biodiversity. It turns the impossible task of listening to everything into a manageable, collaborative game where humans and computers work together to decode nature's song.

Technical Summary: YAPAT — An Open-Source Platform for Active and Weakly Supervised Annotation of Passive Acoustic Monitoring Data

1. Problem Statement

Passive Acoustic Monitoring (PAM) enables non-invasive, large-scale assessment of animal biodiversity but generates audio archives that exceed the capacity of domain experts to annotate manually. While machine learning (ML) can automate species detection, it requires high-quality labeled training data, creating a bottleneck where applications are restricted to taxa and habitats with existing pre-trained models or labeled datasets.

Existing annotation tools address fragments of this challenge: desktop software (e.g., Raven Pro) offers spectrogram visualization; automated tools (e.g., BirdNET-Analyzer) provide inference; and web platforms (e.g., ARBIMON, Whombat) support collaborative workflows. However, no existing tool supports the full annotation cycle required to produce high-quality, interoperable datasets at scale. Specifically, current solutions lack the joint integration of:

Ontology-grounded label spaces.
Weakly supervised sound event detection (WSSED) for cold-start scenarios.
Active learning (AL) combined with vector similarity search.
Interactive embedding space projections for exploratory annotation.

2. Methodology and System Architecture

YAPAT (Yet Another PAM Annotation Tool) is a web-based, open-source platform designed for interactive machine learning workflows in PAM curation. The system is built on a client-server architecture:

Backend: Python (FastAPI, SQLAlchemy) with a PostgreSQL database extended by pgvector for efficient cosine-similarity retrieval. Long-running tasks (embedding, inference, retraining) are managed asynchronously via Celery and Redis.
Frontend: TypeScript (React 18, Vite, Tailwind CSS) running in the browser, utilizing WebGL for rendering 2D projections and client-side JavaScript for mel-spectrogram computation.
Embedding Model: The system utilizes frozen BirdNET embeddings (1024-dimensional vectors from 3-second, 48kHz windows) as the primary feature representation.

Core Workflow:

Ingestion: Field recordings are segmented into fixed-length (default 3s), non-overlapping snippets.
Embedding: Snippets are embedded using BirdNET and stored in the database.
Annotation Interface: Users interact with a paginated feed of snippet cards containing audio players, spectrograms, and multi-label input fields.
Feedback Loop: Annotations trigger asynchronous retraining of a linear classifier head over the frozen embeddings.

3. Key Contributions

YAPAT distinguishes itself by integrating four capabilities that are absent in combination from prior tools:

A. Ontology-Grounded Label Spaces
Labels are linked at annotation time to authoritative biological (GBIF) and environmental (ENVO) taxonomies. The system supports explicit local concepts as fallbacks, ensuring dataset interoperability and reuse.

B. Multi-Label Active Learning (AL)
YAPAT employs a composite acquisition score to rank unlabeled snippets, balancing three criteria:

Uncertainty: Mean per-label Bernoulli entropy over the active label scope.
Diversity: Distance from a snippet's embedding to its nearest labeled neighbor (encouraging coverage of underexplored regions).
Density: A k-nearest-neighbor density estimate within the unlabeled pool (down-weighting isolated outliers).
Snippets are ranked by this composite score to direct expert effort toward the most informative samples.

C. Dual Cold-Start Mechanisms
To address the "cold-start" problem (the need for labeled data to train a classifier before annotation can begin), YAPAT offers two approaches:

Upload-Based Vector Similarity: Users upload a reference clip (e.g., from a sound library). The system embeds this clip and retrieves nearest neighbors in the dataset via cosine similarity, allowing annotation to begin without any prior classifier.
Weakly Supervised Sound Event Detection (WSSED): Using Multiple Instance Learning (MIL), the system trains a linear classifier on "bags" of 3-second segments derived from file-level labeled collections (e.g., Xeno-Canto, FNJV). This bridges the domain gap between clean focal recordings and noisy PAM data, providing initial predictions to bootstrap the AL cycle.

D. Interactive 2D Embedding Projection
The platform renders an interactive scatter plot of the BirdNET embedding space using user-selectable dimensionality reduction (PCA, t-SNE, UMAP, Isomap). Points are color-coded by annotation status. Clicking a point highlights the corresponding snippet in the annotation feed, coupling spatial exploration with the annotation workflow.

4. Results and Scalability Benchmarks

The system was evaluated using AnuraSet (30,687 snippets from 1,612 recordings).

Ingestion & Embedding: Embedding extraction scales linearly ( $O(N)$ ). On a workstation with an AMD EPYC CPU, full dataset embedding takes ~150s; with a single NVIDIA RTX PRO 6000 GPU, it takes ~52s (approx. 3x speedup).
Similarity Search: Vector similarity queries exhibit near-constant latency (~0.56s) across the full dataset range, remaining within interactive response times.
Active Learning Scoring:
- Uncertainty scoring is near-constant (~0.03–0.12s).
- Density scoring scales as $O(N \log N)$ , reaching ~1.6s.
- Diversity scoring (comparing unlabeled to labeled sets) is the dominant cost, scaling as $O(N \cdot N_l)$ , reaching ~4.1s at full scale. Despite this, all operations remain within interactive limits for realistic sessions.
Retraining: Updating the linear head scales linearly with the number of labeled snippets ( $O(N_l)$ ). GPU acceleration provides a 14–23x speedup over CPU. For typical early cycles ( $N_l < 2,000$ ), CPU retraining completes in under 9 seconds without blocking the interface.
Frontend Performance: WebGL rendering of the embedding projection remains constant (~0.11s) regardless of dataset size. Initial page load for the full dataset is ~2.2s.

5. Significance and Claims

The paper claims that YAPAT is the first tool to jointly support ontology-grounded labels, weak supervision, active learning with vector similarity, and interactive embedding projection in a collaborative, browser-based environment.

Interoperability: By grounding labels in GBIF and ENVO, YAPAT produces datasets suitable for deposition in public repositories and downstream analysis.
Efficiency: The combination of cold-start mechanisms and multi-criteria active learning significantly reduces the expert effort required to produce well-labeled datasets, particularly for taxa lacking pre-existing models.
Deployability: The system is open-source, containerized (Docker), and requires no client-side installation beyond a web browser.

Limitations Acknowledged:

Embedding Dependency: The current architecture relies on BirdNET, which is trained on avian vocalizations and may not generalize equally to other taxa (though transfer to anurans was successful).
Annotation Granularity: Snippets are fixed-length and non-overlapping, preventing temporal delineation of events spanning window boundaries or frequency-specific annotations within a snippet.
Training Constraints: The classifier is a linear head over frozen embeddings; it does not support fine-tuning the encoder or advanced data augmentation.
Unlabeled Data: Unlabeled snippets do not contribute to classifier training (no semi-supervised learning), leaving a large pool of data unused.
Setup Complexity: While open-source, initial server setup requires IT assistance and is not a "point-and-click" installation for non-technical users.

The authors position YAPAT not as a replacement for all existing tools, but as a comprehensive platform that fills the gap in producing high-quality, interoperable PAM datasets at scale through an integrated, human-in-the-loop workflow.