🧬 biology

Pretraining Beyond Birds: Multi-Taxa Acoustic Representation Learning for Pantanal Biodiversity Monitoring

This paper presents a deep learning pipeline for the BirdCLEF 2026 competition that achieves top-tier multi-taxa biodiversity monitoring in the Pantanal by leveraging cross-class pretraining, revealing a counterintuitive calibration-quality paradox in pseudo-labeling, and applying taxonomic smoothing to improve species identification across 234 species.

Original authors: Arunodhayan Sampathkumar, danny kowerko

Published 2026-09-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Arunodhayan Sampathkumar, danny kowerko

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The natural world is speaking, but for the most part, we are not listening closely enough. In the vast, wet expanses of the Pantanal, the world's largest tropical wetland, thousands of species of birds, frogs, insects, and mammals communicate through sound every day. To understand the health of this ecosystem, scientists have turned to passive acoustic monitoring. This method involves placing autonomous recording units in the wild to capture the continuous chorus of life. These devices generate terabytes of audio data, far too much for human ears to sort through manually. To make sense of this deluge, researchers rely on artificial intelligence, specifically deep learning models trained to recognize the unique acoustic signatures of different species. The challenge has long been that these computer models were almost exclusively taught to listen to birds. While birds are vocal and diverse, they are only one part of the story. The wetland is also filled with the buzzing of insects, the croaking of amphibians, and the calls of mammals, all of which are vital indicators of ecological health but were largely ignored by previous listening systems.

A team of researchers from Chemnitz University of Technology set out to build a listening system that could hear the entire chorus, not just the birds. They entered the BirdCLEF 2026 competition, a global challenge to identify 234 different species in the Pantanal. This specific task was difficult because the target list included not only 162 bird species but also 35 frogs, 28 insects, 8 mammals, and 1 reptile. The team faced a significant hurdle: the training data they had was heavily skewed. Nearly 98 percent of the labeled audio clips they could use to teach their computer were bird sounds, leaving the other species with very little to learn from. To solve this, they developed a new approach that fundamentally changed how the computer learned to listen. Instead of training their model only on bird songs, they fed it a massive library of sounds from 9,257 different species, including frogs, insects, and mammals. This broader education allowed the computer's core "brain" to recognize the acoustic features of non-bird life, resulting in a system that could identify the full range of wildlife with remarkable accuracy.

The researchers discovered that this broader training was the single most important factor in their success. By teaching the model to recognize sounds from five different groups of animals, they improved its ability to identify the target species by a measurable margin compared to models trained only on birds. This was critical because nearly one-third of the species they needed to find were not birds. Without this multi-species foundation, the computer lacked the necessary vocabulary to distinguish a frog call from a bird song or an insect buzz. The team also uncovered a surprising and counterintuitive lesson about how these computer models learn from unlabeled data. In their effort to teach the computer using the vast amount of unlabeled recordings from the wetland, they found that a model which was very confident and well-calibrated actually produced worse teaching material than a slightly less confident model. The more confident model tended to be too cautious, assigning low probabilities to uncertain sounds, which resulted in weak lessons for the next stage of learning. The less confident model, by contrast, made bolder guesses that turned out to be more useful for training. This finding suggests that in the future, simply making a model more confident does not always make it a better teacher for itself.

To refine their final results, the team added a clever finishing step that required no extra training time. They realized that animals in the wild do not appear randomly; they follow patterns based on their family trees. A specific type of frog is likely to be found in the same places as other frogs of the same genus. The researchers programmed their system to gently nudge its predictions based on these known relationships. If the computer was unsure about a specific frog call, it would look at what it thought about the other frogs in that same family and adjust its answer accordingly. This small adjustment, blending the prediction for one species with the average prediction of its relatives, provided a consistent, free boost to the system's accuracy. The final result was a system that achieved a high level of precision in identifying species, ranking third among all submissions that were not selected for the top prize. The team's work demonstrates that to truly monitor biodiversity, we must teach our machines to listen to the whole ecosystem, not just the most vocal members. By expanding the training data to include the full diversity of life and understanding the quirks of how these models learn, scientists can build tools that offer a clearer, more complete picture of the natural world.

Technical Summary: Pretraining Beyond Birds for Pantanal Biodiversity Monitoring

Problem Statement

Passive acoustic monitoring (PAM) offers a scalable solution for biodiversity assessment, yet automated species identification remains difficult in multi-taxa ecosystems. The specific challenge addressed in this work is the BirdCLEF 2026 competition, which targets 234 species across five taxonomic classes (Aves, Amphibia, Insecta, Mammalia, and Reptilia) in the Pantanal wetland of Brazil.

The core difficulties identified are:

Multi-Taxa Complexity: Existing deep learning models are predominantly trained on bird vocalizations. They lack the acoustic feature representations necessary to distinguish frog calls, insect stridulations, and mammal vocalizations, despite non-bird taxa comprising 30.8% of the target species.
Severe Class Imbalance: The labeled training set is heavily skewed; birds account for 97.9% of labeled clips despite representing only 69.2% of target species. Non-bird taxa and specific insect sonotypes have extremely few or zero labeled training examples.
Semi-Supervised Constraints: A significant portion of the evaluation data (10,658 unlabelled soundscapes) must be leveraged via pseudo-labeling, but the quality of these labels is sensitive to model calibration and confidence thresholds.

Methodology

The authors propose a comprehensive deep learning pipeline involving systematic pretraining, semi-supervised learning, ensemble methods, and post-processing.

1. Data and Preprocessing

Input: 5-second audio segments resampled to 32 kHz, converted to 256×256 log-mel spectrograms.
External Pretraining Corpus: To address the lack of non-bird data, a corpus of 760,008 clips spanning 9,257 species was assembled. This includes bird data from XenoCanto/BirdCLEF and non-bird data (Amphibia, Insecta, Mammalia, Reptilia) from iNaturalist.
Augmentation: Techniques include SpecAugment, XY masking, random filtering, background noise mixing, and soft pseudo-label mixing.

2. Model Architecture

Backbone: EfficientNetV2-S, initialized with ImageNet weights and fine-tuned.
Head: A Sound Event Detection (SED) head comprising:
- GeMFreq Pooling: Generalized Mean Frequency pooling to aggregate features along the frequency axis.
- AttHead: An attention-based classification head with parallel attention and classification branches.
Alternative Branch: A ProtoSSM branch utilizing Perch embeddings and a bidirectional State Space Model (SSM) to capture long-range temporal dependencies.

3. Training Strategy

Multi-Taxa Pretraining: The backbone is first pretrained on the 9,257-species external corpus before fine-tuning on the competition data.
Semi-Supervised Learning (Pseudo-Labeling): Soft pseudo-labels are generated from unlabelled soundscapes. A critical finding involves the "calibration-quality paradox" (detailed below), where a weaker teacher model (AUROC 0.930) with higher average confidence (0.861) produced better pseudo-labels than a stronger, better-calibrated teacher (AUROC 0.946) with lower confidence (0.257).
Ensemble: The final system blends predictions from:
- Multiple EfficientNetV2-S variants (SED head).
- HGUNet-SED models.
- The ProtoSSM branch.
- Blending ratios are optimized (e.g., 65/35 for SED/ProtoSSM).

4. Inference and Post-Processing

Test-Time Augmentation (TTA): Temporal shifts (2.5s offset) are applied to generate additional segments.
Taxonomic Smoothing: A zero-cost post-processing step blends individual species predictions with the mean predictions of their taxonomic genus ( $\alpha=0.15$ ) and class ( $\alpha=0.05$ ).
Deployment: Models were converted to OpenVINO IR format for CPU-only inference to meet the 90-minute runtime constraint.

Key Contributions

The paper highlights three primary contributions derived from 288 systematic experiments:

Multi-Taxa Backbone Pretraining: Extending pretraining from bird-only data to a diverse corpus of 9,257 species (including non-birds) yields a +0.016 AUROC improvement over bird-only pretraining. This demonstrates that taxonomic diversity in pretraining is more beneficial than simply scaling the number of bird species.
The Calibration-Quality Paradox: The authors identify a counterintuitive phenomenon in pseudo-label generation: a better-calibrated teacher model produced lower-quality pseudo-labels and degraded downstream performance compared to a less calibrated but more confident teacher. This suggests that confidence re-calibration or sharpening is necessary before iterative self-training in bioacoustics.
Taxonomic Smoothing: A post-processing technique that leverages known ecological co-occurrence patterns by blending predictions with taxonomic means. This provides a consistent +0.001 AUROC gain without additional training costs.

Results

The proposed pipeline achieved the following performance on the BirdCLEF 2026 leaderboards:

Public Leaderboard: AUROC 0.960 (after taxonomic smoothing).
Private Leaderboard: AUROC 0.959 (after taxonomic smoothing).
Ranking: 19th among selected submissions and 3rd among unselected submissions.

Ablation studies confirmed that the multi-taxa pretraining provided the largest single performance gain (+0.016), followed by the ensemble of diverse architectures and the ProtoSSM branch. The system demonstrated strong generalization prior to post-processing, with a configuration using the SED head with TTA and the mixed HGUNet ensemble strategy achieving identical public and private LB scores of 0.959, indicating a lack of overfitting to the public set. However, the final taxonomic smoothing step, while boosting the public score to 0.960, slightly reduced the private score to 0.954 (from the 0.959 achieved without smoothing), suggesting mild overfitting to the species co-occurrence distribution in the public evaluation set.

Significance and Claims

The authors claim that this work provides practical guidance for multi-taxa bioacoustic monitoring in conservation applications. Specifically:

Beyond Ornithology: The results establish that for PAM applications expanding beyond bird monitoring, pretraining on taxonomically diverse data is critical for model performance.
Semi-Supervised Best Practices: The identification of the calibration-quality paradox offers a new heuristic for designing self-training pipelines, suggesting that raw confidence may be a more valuable metric for pseudo-label selection than calibration metrics in this domain.
Efficiency: The taxonomic smoothing approach offers a computationally free method to improve accuracy by leveraging ecological priors, though it requires careful consideration regarding generalization to private datasets.

The authors emphasize that their findings are supported by reproducible evidence from extensive ablation studies and that all code and model weights are publicly available to facilitate future research.