🤖 machine learning

Phylogenetic signal in marine mammal and bird vocalizations captured by audio foundation models: the limited benefit of domain-specific pretraining

This study demonstrates that general-purpose pretrained audio foundation models inherently encode deep phylogenetic relationships in marine mammal and bird vocalizations, outperforming both hand-crafted features and domain-specific models without requiring taxon-specific pretraining.

Original authors: Víctor Rincón Yepes

Published 2026-07-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Víctor Rincón Yepes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Sound of Family Trees

Imagine you are walking through a giant, noisy forest where every animal has a unique voice. Some chirp, some roar, and some sing complex songs. For a long time, scientists have wondered: if you listen closely, can you tell who is related to whom just by how they sound? This question sits at the intersection of two big ideas in science. The first is evolution, the story of how species change and split apart over millions of years, like branches growing on a giant tree. The second is bioacoustics, the study of animal sounds.

The big mystery is whether the "family tree" of an animal is written into its voice. If two species sound very similar, does that mean they are close cousins? Or could they just be neighbors who learned to shout the same way? To answer this, scientists used to measure sounds with simple tools, looking at basic features like pitch or the "texture" of the noise. But recently, computers have gotten much smarter. We now have "foundation models"—massive AI programs trained on millions of hours of human speech, music, and nature sounds. These AIs can understand sound in a very deep, complex way, almost like a human listener. The big question this paper asks is: Do these super-smart computers, which were never taught about animal families, accidentally learn to recognize evolutionary relationships just by listening to the sounds?

The Study: Listening to the Family Tree

In this study, researchers decided to test if these modern AI "ears" could hear the hidden history of animal families. They looked at two very different groups of animals: 32 species of marine mammals (like whales, dolphins, and seals) and 20 species of birds.

Think of the AI models as different types of detectives. The researchers gave them a massive library of recordings and asked them to measure how "similar" the sounds of different species were. They compared these sound-similarity scores against the actual "family tree" distances (how many millions of years ago two species shared a common ancestor).

They tested four different ways of analyzing the sound:

The Old Way (MFCC): This is like using a basic ruler to measure sound. It looks at simple, hand-made features.
The General AI (AST & CLAP): These are huge, general-purpose AI models trained on everything from car engines to pop songs. They weren't taught about animals at all.
The Specialist AI (BEATs-bio & BirdNET): These are models specifically trained on animal sounds or bird calls, hoping they would be better at spotting family connections.

The Big Surprise: The Generalists Won

The results were a bit of a shock. The "Old Way" (the basic ruler) barely found any connection between how animals sound and how they are related. It was like trying to read a book by looking at the color of the paper; it just didn't work.

However, the General AI models (the ones trained on human music and speech) did something amazing. They found a strong signal. When the researchers looked at whales and dolphins, the AI could tell that species that split apart more recently sounded more alike, and those that split apart long ago sounded very different. In fact, for the whale family, the connection was incredibly strong.

Here is the twist: The Specialist AIs (the ones trained specifically on birds or bioacoustics) did not do better than the general ones. In fact, for the birds, the general-purpose models were actually slightly better at finding the family tree than the bird-expert models. The paper suggests that training a model specifically on animals might actually make it worse at seeing the big evolutionary picture, perhaps because it gets too focused on tiny details or specific species labels rather than the deeper structure of the sound.

What the AI Actually Heard

The researchers wanted to make sure the AI wasn't just cheating by noticing simple things, like "big animals have low voices and small animals have high voices." They ran special tests to see if the AI was just listening to the pitch. The answer was no. Even when they removed the pitch from the equation, the AI still found the family connections.

This means the AI is picking up on something much more subtle: the style of the call. It's like how you can tell a family member is talking even if they are whispering or shouting, because of how they shape their words, not just how loud they are. The AI learned the "accent" of evolution.

The Mix-Up: When Sound Lies

The study also found some funny exceptions that prove the rule.

The Sound-Alike Strangers: The Bearded Seal and the Northern Right Whale are cousins who split apart 90 million years ago. They should sound totally different. But they sound almost identical! The AI noticed they sound the same, even though they are ancient strangers. This is likely because they both live in similar cold environments and need to make low, rumbling sounds to travel through water.
The Sound-Different Cousins: The Leopard Seal and the Weddell Seal are very close cousins, splitting apart only 10 million years ago. You'd expect them to sound similar. But they sound completely different! One makes complex, musical underwater songs, while the other makes low growls.

These examples show that while evolution leaves a mark on sound, the environment can sometimes force animals to change their voices so much that they sound like strangers, or so similar that they sound like family.

The Bottom Line

This paper tells us that we don't need to build special, animal-only computers to understand the evolutionary history of animal voices. The general-purpose AI models, trained on the sounds of our human world, are already incredibly good at hearing the "family tree" hidden in animal calls. They are better than the old, simple tools, and surprisingly, they are even better than the models built specifically for animals.

The researchers suggest that the secret isn't in training the AI on animals, but in the way these general models are built to understand the complex patterns of sound. They propose that future experiments should test if this is because of the AI's brain structure, the way it learns, or the huge variety of sounds it was trained on. But for now, the message is clear: if you want to listen to the history of life in a sound, you might just need to ask a general-purpose AI to listen for you.

Technical Summary: Phylogenetic Signal in Marine Mammal and Bird Vocalizations Captured by Audio Foundation Models

Problem Statement
A fundamental question in comparative bioacoustics is whether acoustic similarity among species reflects their evolutionary history (phylogenetic signal). While previous studies using hand-crafted spectral features (e.g., MFCCs, spectral entropy) have detected modest phylogenetic signals in birds, frogs, and fish, it remains unknown whether large-scale pretrained audio foundation models—which encode richer, high-dimensional acoustic representations—can recover stronger phylogenetic structure. Furthermore, it is unclear whether domain-specific pretraining (e.g., models trained specifically on bioacoustics or bird species) is necessary to recover this signal compared to general-purpose audio models.

Methodology
The study employs Mantel tests to correlate acoustic distance matrices with phylogenetic distance matrices across two independent radiations: marine mammals and birds.

Datasets:
- Marine Mammals: 1,754 recordings across 32 species (Cetacea and Pinnipedia) from the Watkins Marine Mammal Sound Database. Phylogenetic distances were derived from published divergence times (2–90 Myr).
- Birds: 607 recordings across 20 species from seven orders, sourced from xeno-canto. Phylogenetic distances were based on the time-calibrated phylogeny of Jetz et al. (2012).
Audio Representations: Four representations were evaluated:
1. MFCC Baseline: Hand-crafted features (105 dimensions) including MFCCs, chroma, and spectral statistics.
2. AST (Audio Spectrogram Transformer): A general-purpose transformer (768d) trained on AudioSet event tags.
3. CLAP (Contrastive Language-Audio Pretraining): A general-purpose model (512d) trained on audio-text pairs via contrastive learning.
4. BEATs-bio: A bioacoustic-specialized encoder (768d) fine-tuned on a large bioacoustic corpus.
5. BirdNET: A bird-specific classifier (1,024d) trained to distinguish ~6,000 bird species (evaluated only on birds).
Analysis:
- Acoustic distances were calculated as cosine distances between L2-normalized species centroids.
- Mantel tests (9,999 permutations) assessed the correlation between acoustic and phylogenetic distances.
- Controls:
  - Dimensionality: Embeddings were projected to the 105-dimensional MFCC space via PCA to rule out dimensionality as a confound.
  - Dominant Frequency: Partial Mantel tests controlled for dominant frequency distances.
  - Taxonomic Split: Analyses were repeated within the cetacean clade (26 species) to isolate fine-scale phylogenetic structure from the deep cetacean-pinniped split.
- Statistical Comparison: Paired bootstrap confidence intervals were used to compare the performance ( $\Delta r$ ) between models.

Key Results

Foundation Models vs. MFCC: General-purpose foundation models (AST, CLAP, BEATs-bio) consistently recovered significant phylogenetic signal, whereas hand-crafted MFCCs showed no significant correlation in the full marine mammal dataset ( $r = 0.040, p = 0.338$ $r = 0.040, p = 0.338$ ).
- In marine mammals, foundation models achieved $r \approx 0.40$ ( $p < 0.001$ ).
- In birds, foundation models achieved $r \approx 0.52–0.55$ ( $p < 0.001$ ).
Domain-Specific Pretraining: Contrary to the hypothesis that specialized training improves signal recovery, domain-specific models did not outperform general-purpose models.
- In birds, BirdNET ( $r = 0.361$ ) and BEATs-bio ( $r = 0.318$ ) performed worse than the general-purpose AST ( $r = 0.549$ ) and CLAP ( $r = 0.521$ ), though the differences were not statistically significant.
- All foundation models significantly outperformed MFCCs, but no two foundation models differed significantly from one another.
Signal Characteristics:
- Within-Cetaceans: The signal was exceptionally strong within the cetacean clade ( $r \approx 0.74–0.82$ ), indicating the models capture fine-grained evolutionary structure.
- Frequency Independence: The correlation persisted after controlling for dominant frequency (retaining 97% of $R^2$ in marine mammals), suggesting the signal is not merely a proxy for body size or pitch.
- Dimensionality: The advantage of foundation models over MFCCs remained even after projecting embeddings to the same dimensionality, confirming the benefit is not an artifact of higher dimensionality.
Convergence and Divergence: The models correctly identified extreme acoustic convergence (e.g., Bearded Seal and Northern Right Whale, 90 Myr divergence, low acoustic distance) and divergence (e.g., Leopard Seal and Weddell Seal, 10 Myr divergence, high acoustic distance), illustrating how ecological pressures can decouple acoustic similarity from phylogeny.

Significance and Claims
The paper claims that pretrained audio foundation models encode evolutionary information across independent radiations, capturing structural properties of vocalizations (such as call structure and temporal modulation) that hand-crafted features miss.

Crucially, the authors conclude that domain-specific pretraining is neither necessary nor sufficient for recovering strong phylogenetic signal. General-purpose models trained on diverse, non-biological audio (AudioSet, web audio-text pairs) performed at least as well as, and in the case of birds, better than, models specialized for bioacoustics or bird classification.

The paper posits that the lack of improvement from domain-specific models may be due to:

Training Objectives: Supervised species classification (as in BirdNET) may collapse the graded similarity required for phylogenetic analysis.
Architecture: Transformers may capture long-range spectro-temporal structures better than CNNs (BirdNET).
Data Breadth: Narrow, specialized training corpora may lack the representational richness found in broad, general audio datasets.

The authors explicitly state that they cannot disentangle these factors with current models and propose a controlled fine-tuning experiment (taking a general encoder and fine-tuning it on species classification) as the necessary next step to isolate the cause. The practical takeaway is that for comparative bioacoustics, general-purpose audio embeddings are a robust tool, and a "domain-specific" label does not automatically confer an advantage for phylogenetic analysis.