Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study

Technical Summary: Masked Autoencoders with Limited Data in Bioacoustics

Problem Statement
Bioacoustic recognition requires fine-grained acoustic understanding to distinguish between similar-sounding species. However, large-scale bioacoustic repositories (e.g., iNaturalist) are typically weakly annotated, often providing only a single positive species label per recording. This scarcity of high-quality, exhaustive annotations makes supervised learning challenging. While Masked Autoencoders (MAE) have demonstrated strong transferability on massive general audio corpora, their effectiveness in moderate-sized, fine-grained bioacoustic settings remains underexplored. The central question addressed is whether MAE pretraining can be effectively adapted for species classification when data scales are limited and annotations are noisy.

Methodology
The authors propose a two-stage training paradigm for fine-grained bioacoustic classification:

1. Self-Supervised Pretraining: An audio encoder is trained on large-scale unlabeled audio data using a masked spectrogram reconstruction objective. The input audio is converted to log-mel spectrograms, partitioned into non-overlapping patches, and a random subset of patches is masked. A lightweight decoder predicts the pixel-level values of the masked patches, minimizing the mean squared error (MSE) loss over the masked regions.
2. Downstream Adaptation: The pretrained encoder is adapted to a smaller, labeled target dataset (iNatSounds) via:
   • Linear Probing: Freezing the encoder and training only the classification head.
   • Full Fine-tuning: Updating both the encoder and classification head parameters using cross-entropy loss.

The study systematically evaluates the impact of pretraining data scale, domain specificity (general audio vs. bioacoustic), data curation (filtering noisy segments), and transfer strategies. Experiments utilize the iNatSounds dataset (5,569 species, 1.2k hours of audio) and AudioSet (2M clips) as pretraining sources.

Key Contributions and Results

• Linear Probing Ineffectiveness: MAE-pretrained representations yield extremely low performance under linear probing (e.g., ~3.5% Top-1 accuracy on iNatSounds), indicating that the masked reconstruction objective alone does not produce linearly separable features suitable for fine-grained species recognition without substantial task-specific adaptation.
• Pretraining Scale Dominates Domain Specificity:
  • Models pretrained on diverse, general-domain audio (AudioSet) or even general images (ImageNet) significantly outperform models pretrained solely on domain-specific bioacoustic data (iNatSounds).
  • Continual pretraining on the target domain (iNatSounds) provides only marginal gains (approx. 10% improvement over random initialization) compared to the ~10% gains achieved by pretraining on larger, diverse datasets like AudioSet.
  • The best performance is achieved by pretraining on AudioSet and then fine-tuning on iNatSounds, but the additional gain from an intermediate domain-specific self-supervised step is negligible.
• Data Curation Offers Limited Benefits:
  • The authors investigated filtering out "empty" or low-confidence audio segments to reduce noise. While filtering successfully preserved informative patterns (evidenced by MobileNetV3 performance on filtered subsets), it did not improve MAE pretraining outcomes.
  • In fact, using the full, noisier dataset for pretraining yielded better results than using curated, smaller subsets. This suggests that in moderate-scale settings, the volume of data outweighs the benefits of strict curation.
• Comparison with Prior Work: Unlike findings from large-scale benchmarks where domain-specific pretraining is crucial, this study finds that for moderate-sized datasets (like iNatSounds, which is 5x smaller in audio duration and 3x fewer species than prior large-scale corpora), "off-the-shelf" general domain models are superior.

Significance and Claims
The paper claims to provide practical guidance for model selection in bioacoustics under limited supervision. The primary finding is that pretraining scale dominates objective design in moderate-sized fine-grained settings. The authors conclude that:

1. Additional masked reconstruction pretraining on domain-specific data provides limited benefits and may even degrade performance relative to using off-the-shelf models pretrained on massive general corpora.
2. Selective data filtering offers negligible advantages when the overall data scale is limited.
3. Supervised fine-tuning from general-domain pretrained models is the dominant strategy for moderate-sized datasets, while the benefits of in-domain pretraining likely only emerge at much larger data scales.

These findings clarify when MAE-based pretraining is effective, suggesting that for ecological research with moderate data availability, leveraging large-scale general audio or even image pretraining is more effective than attempting to curate and pretrain on smaller, specific bioacoustic datasets.