Can Foundation Models Hear What Made That Sound? A Tiered Benchmark of Audio-Language Models and Traditional Classifiers for Closed-Set Sound Source Identification
This paper introduces a tiered benchmark evaluating eleven audio-language models and traditional classifiers on a closed-set sound source identification task, revealing that while specialized foundation models like Gemini-3.1-Pro-Preview achieve the highest accuracy, zero-shot models can match category-level performance, and that confident errors and difficulty confounds significantly impact fine-grained classification reliability.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you're walking down a busy street. You hear a siren, a doorbell, or the splash of a faucet. Your brain instantly identifies the source without you even thinking about it. For decades, computers have struggled to do this same trick. They needed to be taught specific sounds, like a dog barking or a glass breaking, using a fixed list of labels. But recently, a new wave of "super-smart" computer brains—called Large Language Models (LLMs)—has arrived. These models can read, write, and now, listen. They can look at a sound and describe it in words, or even pick an answer from a list you give them.
The big question for anyone building audio technology is: "Which tool should I use?" Should I use a specialized sound detector trained on a fixed list, or should I ask a giant, general-purpose AI to listen and guess? The problem is that these tools play by different rules. Some are like a multiple-choice test where you pick A, B, or C. Others are like a free-form essay where you just describe what you hear. Comparing their scores directly is like comparing a sprinter's time to a swimmer's time; you can't just say who is "faster" without understanding the race they ran. This paper steps in to organize the chaos, setting up a fair, multi-level playing field to see who really hears best.
The Great Sound Detective Showdown
In this study, the researchers gathered eleven different "sound detectives" to solve a mystery: identifying the source of 2,242 short audio clips. The clips covered 23 specific types of sounds (like "Police Siren" vs. "Ambulance Siren") grouped into 11 broader categories. To make the comparison fair, they didn't just throw everything into one big leaderboard. Instead, they created four different tiers, or levels of difficulty, because the detectives were given different types of clues and had to answer in different ways.
The Four Tiers of Detection:
- The Multiple-Choice Pros (Tier A): These models (mostly Google's Gemini family and an open-source model called Kimi-Audio) were given the audio clip and a list of the 23 possible answers. They had to pick the exact right letter. This is the "easiest" setup because the model knows the answer is on the list.
- The Fixed-List Veterans (Tier B): These are older, specialized sound classifiers (like YAMNet and PANNs). They don't know your list of 23 sounds exists. They just hear the sound and spit out their own internal list of thousands of possible sounds. The researchers then had to manually translate the model's answer to see if it matched the target. This is harder because the model is guessing from a much larger pool.
- The Zero-Shot Matchers (Tier C): This model (CLAP) was given the 23 answers as text descriptions (e.g., "a police car siren") but wasn't asked to write a story. It simply measured how "similar" the sound was to the text description.
- The Free-Form Storytellers (Tier D): This model (BAT) was asked to just write down what it heard in free text. Since it didn't pick a letter, a second AI had to read its story and grade it.
The Winners and the "Good Enough"
When the dust settled, the results were a mix of "Wow" and "Hmm."
The Champion: The best performer was Gemini-3.1-Pro-Preview. When asked to pick from the list of 23 sounds, it got the broad categories right about 85.6% of the time and the specific sub-types (like distinguishing a police siren from an ambulance) right 56.7% of the time.
The Underdog: The open-source Kimi-Audio-7B-Instruct model was a surprise contender. For a model that is much smaller and open for anyone to use, it did pretty well on the broad categories (67.5%). However, it stumbled on the fine details, getting only 32.9% of the specific sub-types right. Also, it had a glitch: it refused to answer or got stuck in a loop for 1.6% of the clips, whereas the Gemini models never failed to answer.
The Surprise Contenders: The models that didn't see the list of answers (Tier B and Tier C) actually did surprisingly well on the broad categories. SSLAM and CLAP matched or even beat the best "multiple-choice" models at identifying the general category (like "Siren") without ever being told what the options were. However, once the task got harder and required distinguishing fine details (like "Police Siren" vs. "Ambulance Siren"), the models that saw the list (Tier A) pulled ahead significantly.
The Mystery of the "Confident" Wrong Answers
One of the most fascinating parts of the paper wasn't just who won, but how the models thought. The researchers looked at the "chain-of-thought"—the internal reasoning steps the models wrote down before giving their answer.
They found three big surprises:
- Longer isn't better: You might think a model that writes a long, detailed paragraph analyzing the sound would be smarter. Not so! The most accurate model, Gemini-3.1-Pro-Preview, was actually the most concise, writing short, direct answers. The models that wrote long, step-by-step essays were often less accurate.
- The "Holistic" Shortcut: When a model used a quick "gut feeling" (saying "this sounds like a siren" without analyzing the pitch), it was often right. When it tried to do a deep, detailed analysis, it was often wrong. But this wasn't because deep analysis is bad. It turned out the models only did deep analysis when the sound was really hard to identify. The "gut feeling" wins because it's usually used on easy sounds, while the "deep dive" is used on the tricky ones.
- The Confidence Trap: This is the scariest part. When the models got an answer wrong, they were 92% to 100% confident about it. They would say things like "It is definitely a doorbell" even when it was a siren. They almost never said, "I'm not sure." This means you can't trust a model's confidence to tell you when it's making a mistake.
The Hard Parts: Where Everyone Failed
Even the best models struggled with specific types of sounds. The paper found that the errors weren't random; they happened in predictable ways:
- Distance is invisible: Models could hear running water, but they couldn't tell if it was a faucet nearby or a stream far away. They almost always guessed "close source."
- The Doorbell Dilemma: Wireless doorbells sound a lot like electronic alarms. The models often confused a wireless doorbell with a generic appliance alarm.
- Siren Confusion: Distinguishing between a police, fire, or ambulance siren was tough. Some models had a "default" bias; if they weren't sure, they would just guess "Police" or "Ambulance" every time.
What Does This Mean for the Future?
If you just need to know "Is that a siren or a doorbell?", you don't need a giant, expensive AI. A simpler, specialized sound detector or a zero-shot model works just fine and might even be better.
But if you need to know exactly which kind of siren it is, the best tool right now is a "task-aware" LLM (like the Gemini models) that you give a list of options to choose from. However, you have to be careful: these models are still prone to specific mistakes, and they will confidently tell you they are right even when they are wrong.
The paper concludes that while AI is getting better at hearing, it's not a magic bullet yet. The best approach depends entirely on how specific you need the answer to be, and we still need to figure out how to make these models admit when they are unsure.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.