⚡ electrical engineering

Beyond the Baseband: Adaptive Multi-Band Encoding for Full-Spectrum Bioacoustics Classification

This paper proposes and validates an adaptive multi-band encoding framework that decomposes full-spectrum bioacoustic signals into band features and fuses them, demonstrating that this approach consistently outperforms traditional baseband and time-expansion methods in animal call classification across multiple datasets.

Original authors: Eklavya Sarkar, Marius Miron, David Robinson, Gagan Narula, Milad Alizadeh, Ellen Gilsenan-McMahon, Emmanuel Chemla, Olivier Pietquin, Matthieu Geist

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eklavya Sarkar, Marius Miron, David Robinson, Gagan Narula, Milad Alizadeh, Ellen Gilsenan-McMahon, Emmanuel Chemla, Olivier Pietquin, Matthieu Geist

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Tunnel Vision" of AI

Imagine you are trying to listen to a conversation between two people, but you are wearing headphones that only let you hear the lowest, rumbling bass notes. You miss the high-pitched squeaks, the sharp whistles, and the crisp consonants.

This is exactly what happens when scientists use current AI models to study animal sounds.

The Reality: Many animals (like bats, insects, and marine mammals) talk in frequencies so high they are "ultrasonic"—far beyond what humans can hear. A bat's call might be like a high-pitched whistle at 100,000 Hz.
The AI Limit: Most powerful AI models were trained on human speech. Human speech fits comfortably in a low-frequency range (0–8 kHz). Because of this, these AI models act like a filter that chops off everything above 8,000 Hz.
The Result: When scientists feed a recording of a bat into these AIs, the AI is essentially listening to a muffled, low-quality version of the sound, throwing away all the high-frequency details that actually contain the most important information.

The Old Fix: "Slow Motion" (Time Expansion)

Previously, scientists tried to fix this by playing the recording in slow motion.

The Analogy: Imagine taking a high-speed video of a hummingbird's wings and slowing it down so you can see the details. By slowing the audio down, the high-pitched whistles drop down into the low range the AI can hear.
The Flaw: While this makes the pitch audible, it stretches the sound out. A 1-second bat call becomes a 15-second long, stretched-out drone. This makes the AI work much harder and slower to process the data, and it distorts the natural rhythm of the sound.

The New Solution: "The Multi-Band Team"

The authors of this paper propose a smarter way to listen to the full spectrum of animal sounds without slowing them down. They call this Adaptive Multi-Band Encoding.

Think of the full range of animal sounds as a massive, colorful rainbow. The old AI could only see the bottom few colors (Red and Orange). The new method breaks the whole rainbow into separate strips, processes each strip, and then puts them back together.

Here is how their system works, step-by-step:

Slicing the Spectrum: Instead of listening to the whole sound at once, the system slices the audio into different "bands" or layers.
- Band 1: The low sounds (what the AI already knows).
- Band 2: The medium-high sounds.
- Band 3: The super-high sounds (the part the AI usually ignores).
The "Heterodyning" Trick: For the high bands, the system uses a mathematical trick (like shifting a radio station) to lower the pitch of that specific slice just enough so the AI can understand it, without stretching the time. It's like translating a high-pitched alien language into a low-pitched human language instantly, rather than playing the recording in slow motion.
The Team Meeting (Fusion): Now the AI has analyzed the low slice, the medium slice, and the high slice separately. The system then uses a "fusion" strategy to combine these insights.
- Some strategies just take an average (like a group vote).
- Others use a "smart manager" (like a Gated Pool or Mixture of Experts) that decides, "Hey, for this specific bat call, the high-pitched slice is the most important, so I'll listen to that one more closely."

What They Found

The researchers tested this method on three different groups of animals: Dogs, Birds, and Bats.

For Dogs and Birds: These animals mostly speak in frequencies humans can already hear. The new method performed just as well as the standard method, proving it doesn't break things that are already working.
For Bats: This is where the magic happened. Bats scream at frequencies way above human hearing.
- The standard AI (Baseband) failed to recognize them well because it was missing the high notes.
- The "Slow Motion" method (Time Expansion) worked better but was slow and clunky.
- The Multi-Band Method was the clear winner. By keeping the high-frequency details intact and feeding them to the AI in a smart way, it identified bat calls much more accurately than the old methods.

The "Secret Sauce": Why It Works

The paper discovered something interesting about how the AI "thinks" about these different slices.

When the AI looks at the low sounds, it sees one pattern.
When it looks at the high sounds (after the pitch-shifting trick), it sees a completely different pattern.
Because these patterns are different (decorrelated), they provide unique clues. It's like solving a mystery where one witness saw the suspect's shoes, and another saw their hat. Putting those two different clues together gives you a much better picture than just looking at the shoes alone.

The Bottom Line

This paper shows that we don't need to build entirely new, expensive AI models to hear the full spectrum of animal life. Instead, we can take the AI models we already have, slice the animal sounds into manageable pieces, and combine them intelligently.

This allows us to "hear" the full, high-pitched conversations of bats and insects, unlocking a richer understanding of the natural world without needing to slow time down. The authors have even made their code open-source so other scientists can use this "multi-band team" approach immediately.

1. Problem Statement

Current computational bioacoustics systems face a critical bandwidth limitation. Most state-of-the-art foundation models (pre-trained on human speech or general audio) operate at a 16 kHz sampling rate, restricting their usable frequency range to the 0–8 kHz baseband.

The Issue: Many animal vocalizations (e.g., bats up to 200 kHz, insects to 100 kHz, marine mammals to 150 kHz) contain crucial information in the ultrasonic domain that is discarded when these recordings are resampled to 8 kHz.
Limitations of Existing Solutions:
- High-SR Models: Training new foundation models at higher sampling rates (e.g., 48 kHz or 250 kHz) is computationally expensive, and few such models exist (e.g., BirdNET at 48 kHz).
- Time-Expansion (TE): A common signal processing technique where audio is slowed down to shift high frequencies into the baseband. However, this reduces spectral resolution, stretches the signal duration (increasing computational cost), and fundamentally alters the signal representation.

2. Methodology: Adaptive Multi-Band (MB) Encoding

The authors propose a heterodyning-based multi-band encoding framework that allows existing 16 kHz models to process full-spectrum audio without retraining the base encoder. The pipeline consists of three stages:

Spectral Decomposition & Baseband Mapping:
- The raw audio signal ( $s$ ) sampled at $f_s$ is split into $B$ non-overlapping frequency bands, each with a width of $f_m/2$ (where $f_m$ is the model's native sampling rate, e.g., 16 kHz).
- The first band (0–8 kHz) is the standard baseband.
- For subsequent bands, the signal is band-pass filtered and then heterodyned (mixed) down to the baseband. This is achieved by multiplying the band signal by a cosine wave at the band's center frequency, followed by low-pass filtering.
- The result is $B$ baseband waveforms, each representing a distinct portion of the original high-frequency spectrum.
Representation Extraction:
- Each of the $B$ baseband waveforms is resampled to match the model's expected input rate.
- They are passed individually through a frozen pre-trained encoder (e.g., EfficientNet, BEATs, EATs) to generate embeddings.
- Temporal statistics (first-order) are computed to create fixed-length functionals for each band.
Fusion Strategies:
The paper evaluates five strategies to combine these band-level embeddings into a unified representation for classification:
- Mean-Pool (MP): Simple averaging of band embeddings.
- Gated-Pool (GP): A learnable linear projection assigns weights to bands based on their informativeness.
- Mixture-of-Experts (MoE): Each band is classified independently, and the resulting logits are weighted and summed.
- Hybrid (HYB): Similar to GP but incorporates handcrafted acoustic features (spectral entropy, flux) into the gating network.
- Self-Attention (SA): Treats band embeddings as a sequence processed by a Transformer encoder with a [CLS] token.

3. Experimental Setup

Datasets: Three bioacoustic datasets from the BEANS benchmark:
- Dogs: 44.1 kHz sampling, 10 classes.
- Cornell Birdcall Identification (CBI): 44.1 kHz, 264 classes.
- Bats: 250 kHz sampling, 10 classes (highly ultrasonic).
Models: Eight pre-trained audio models were tested, including EfficientNet variants (Bio, AS, All), BEATs, EATs, and BirdNET (48 kHz).
Baselines:
- Baseband (BB): Standard resampling to 8 kHz.
- Time-Expansion (TE): Slowing down audio to fit high frequencies into 8 kHz.

4. Key Results

A. Representation Analysis

Decorrelation: For certain models (e.g., EfficientNet), embeddings from higher frequency bands were found to be decorrelated from the baseband embeddings. This suggests they contain complementary information rather than redundant data.
Class Separation: Fusion methods generally produced more discriminative embeddings than the baseband alone, particularly for the CBI dataset.

B. Classification Performance

Dogs & CBI (44 kHz):
- Fusion vs. Baseband: Multi-band fusion consistently outperformed the standard baseband approach across most models.
- Fusion vs. Time-Expansion: Fusion significantly outperformed Time-Expansion, which degraded performance on these datasets.
- Comparison to BirdNET: Most 16 kHz models combined with fusion strategies matched or outperformed BirdNET (a native 48 kHz model) on Dogs and CBI tasks.
Bats (250 kHz):
- Information Gain: Both Time-Expansion and Fusion provided substantial gains over the baseband, confirming that ultrasonic content is vital for bat classification.
- TE vs. Fusion: In this specific high-frequency regime, Time-Expansion performed slightly better than fusion methods, likely because the extreme frequency shift required for 250 kHz audio makes heterodyning less effective than simple time-stretching for this specific task.
- Model Adaptability: When the MB framework was applied to BirdNET (48 kHz), it further improved performance, demonstrating the framework's ability to leverage even higher native bandwidths.

C. Best Performing Strategy

Mixture-of-Experts (MoE) emerged as the most robust fusion strategy, particularly for the CBI and Bats datasets. This suggests that classifying bands independently before aggregating results is more effective than fusing raw embeddings first.

5. Key Contributions

Framework Proposal: Introduced an adaptive multi-band encoding framework that unlocks the full-spectrum potential of existing 16 kHz foundation models without requiring expensive retraining.
Empirical Validation: Demonstrated that multi-band fusion can match or exceed the performance of models pre-trained at higher sampling rates (e.g., 48 kHz) on standard bioacoustic tasks.
Open Source Toolkit: Released the framework as an open-source toolkit to facilitate adoption by the bioacoustics community.
Insight on Band Decorrelation: Provided evidence that specific pre-trained models generate decorrelated embeddings for different frequency bands, validating the theoretical basis for fusion.

6. Significance

This work addresses a fundamental bottleneck in computational bioacoustics: the mismatch between the limited bandwidth of human-centric AI models and the wide spectral range of animal communication. By enabling existing models to utilize ultrasonic information through a lightweight, modular encoding strategy, the authors provide a practical and scalable solution. This approach allows researchers to leverage powerful, pre-trained foundation models for diverse species (from birds to bats) without the prohibitive cost of training new high-sampling-rate models from scratch. The results suggest that how we encode the spectrum is as important as the model architecture itself for decoding non-human vocalizations.