Technical Summary: Adaptation of Speech and Bioacoustics Models

Problem Statement

Self-supervised learning (SSL) models pre-trained on human speech (e.g., HuBERT) and animal vocalizations (e.g., AVES) have demonstrated strong transferability to bioacoustic tasks. However, their frozen representations are not necessarily optimized for specific downstream datasets, which are often species-specific, limited in size, and expensive to annotate. Full fine-tuning of large SSL encoders is computationally expensive and often impractical given the scarcity of annotated animal data. While prior work has compared pre-training domains, it remains unclear how these models should be adapted for bioacoustic tasks, specifically regarding which components to update, how many layers to modify, and whether speech-pretrained and bioacoustic-pretrained models benefit from adaptation strategies in the same way. Furthermore, existing literature suggests a progressive decline in classification performance across deeper encoder layers when using frozen models; it is unknown if direct adaptation on downstream bioacoustic data alters this trend.

Methodology

This study investigates Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA) for Call-Type Identification (CTID). The authors employ two architecturally identical transformer-based models: HuBERT (pre-trained on human speech) and AVES-Bio (pre-trained on bioacoustic data). The research is structured around four systematic ablation studies to determine optimal adaptation strategies:

1. Matrix Selection (Q1): The study evaluates which subsets of transformer projection matrices yield the best performance. Configurations range from adapting only feed-forward networks ([FC1, FC2]) to adapting all self-attention projections (Q, K, V) and output/feed-forward layers ([Q, K, V, FC0, FC1, FC2]).
2. Scope of Adaptation (Q2): The authors examine whether extending LoRA adapters beyond the Transformer encoder improves performance. They compare:
   • Encoder only: Adapters in Transformer layers.
   • Projector + Encoder: Adapters in the Transformer and the feature projection layer.
   • Extractor + Projector + Encoder: Full fine-tuning of the convolutional feature extractor combined with adapters in the projector and encoder.
3. Layer Selection Strategies (Q3): Two strategies are compared to determine which encoder layers should be adapted:
   • Bottoms-up: Incrementally adapting layers from the first layer upwards ($l_1$ to $l_k$).
   • Top-down: Incrementally adapting layers from the final layer downwards ($l_L$ to $l_{L-k}$).
4. Fine-Tuning Strategies (Q4): The study compares three paradigms:
   • Linear Probing: All layers frozen; a linear classifier is trained on top.
   • LoRA + Freezing: Selected layers receive LoRA adapters and are fine-tuned; others remain frozen.
   • LoRA + Pruning: Selected layers are fine-tuned with LoRA; unselected layers are removed entirely from the model.

Experiments were conducted on two datasets: InfantMarmosetsVox (IMV) (72,920 samples, 11 call types) and Abzaliev (8,034 samples, 14 call types). Hyperparameters were optimized via grid search, identifying a rank ($r$) of 60, scaling factor ($\alpha$) of 3, and learning rate of $10^{-3}$ as optimal.

Key Results

Matrix and Scope Selection

• Matrix Selection: Performance exhibits a monotonic progression where adapting a broader set of projection matrices yields stronger results. The configuration adapting all self-attention and feed-forward projections ([Q, K, V, FC0, FC1, FC2]) achieved the highest Unweighted Average Recall (UAR).
• Feature Extractor Adaptation: Directly fine-tuning the convolutional feature extractor (instead of using LoRA) consistently and significantly degraded downstream performance. Adapting the feature projector yielded only marginal gains compared to adapting the encoder alone.

Layer Selection and Trends

• Strategy Comparison: Neither "bottoms-up" nor "top-down" layer selection strategies consistently outperformed the other. Both produced comparable results when adapters were placed on the same matrices.
• Layer-wise Performance:
  • Frozen Models: Consistent with prior literature, linear probing on frozen models showed a progressive decline in performance as deeper layers were used.
  • LoRA Adaptation: When LoRA was applied, deeper transformer layers in AVES showed a general upward trajectory in performance. This indicates that deeper layers encode abstract features that can effectively classify calls, but only when the layers are fine-tuned.

Fine-Tuning Strategies and Dataset Size

• Large Dataset (IMV): LoRA fine-tuning (with either freezing or pruning) substantially outperformed simple linear probing across nearly all layers.
• Small Dataset (Abzaliev): Simple linear probing remained more reliable and often exceeded LoRA performance, though the gap was modest. This suggests LoRA's efficacy scales with dataset size, while linear probing is a more robust baseline for low-data scenarios.
• Pruning vs. Freezing: Removing unused layers (pruning) offered no significant advantage over simply freezing them in terms of performance, though it reduces model size.

Classifier Complexity

• On the Abzaliev dataset, a deeper 4-layer MLP classifier outperformed the single linear layer used in the LoRA experiments.
• On the IMV dataset, the single linear layer outperformed the MLP. This indicates dataset-specific behavior regarding the necessity of classifier non-linearity.

Significance and Claims

The paper claims that Low-Rank Adaptation (LoRA) provides a practical and effective framework for adapting large SSL representations to bioacoustic tasks, particularly when sufficient labeled data is available.

• Optimization of Adaptation: The study highlights that the success of PEFT in bioacoustics is highly dependent on specific design choices: adapting a broader set of transformer projections is superior, while adapting the convolutional feature extractor is detrimental.
• Reversing Layer Trends: A key finding is that direct adaptation via LoRA can reverse the typical performance decline observed in deeper layers of frozen SSL models, allowing deeper layers to contribute meaningfully to classification.
• Data Dependency: The authors modestly conclude that while LoRA is a powerful tool for bioacoustic classification with ample data, a classic linear probe may remain a stronger and more stable baseline in low-data settings.
• Generalizability: The findings underscore the importance of carefully selecting adapter placements, layer strategies, and fine-tuning methods when applying large SSL models to bioacoustic tasks, rather than assuming a one-size-fits-all approach.

The work does not claim to solve all bioacoustic challenges but establishes a systematic understanding of how parameter-efficient methods interact with speech and bioacoustic pre-trained models, offering a guide for future adaptation strategies in the field.