From sound to source: Human and model recognition of environmental sounds

This paper introduces a large-scale behavioral benchmark for human environmental sound recognition and demonstrates that artificial neural networks trained on real-world multi-source scenes achieve near-human accuracy and alignment with brain responses, outperforming traditional auditory models.

The Gist

Imagine you are walking down a busy city street. You hear a siren, the crunch of gravel underfoot, a dog barking, and a car honking all at once. Your brain instantly sorts this chaotic soup of noise, identifies each sound, and tells you, "That's a fire truck coming from the left," or "That's a dog behind me." This is environmental sound recognition.

For a long time, scientists knew humans were good at this, but they didn't have a good way to measure exactly how good, or to build computers that could do it the same way.

This paper is like a massive "driving test" for both human ears and computer ears. The researchers created a giant, rigorous test to see how well humans and different types of AI models can identify sounds in messy, real-world situations.

Here is the breakdown of their journey, using some everyday analogies:

1. The "Noise Party" (The Human Benchmark)

First, the researchers needed to know how humans perform. They didn't just play one sound at a time; they threw a "noise party."

• The Test: They played recordings where 1 to 5 different sounds (like a cough, a car, or a bird) were mixed together. Then, they asked participants: "Was a cough in that mix?"
• The Result: Just like at a loud party where it's hard to hear one person talk, humans got worse at identifying sounds as the "party" got louder (more sounds mixed together). However, humans are surprisingly resilient; even with five sounds mixed, they could still pick out the target.
• The Distortion Test: They also took single sounds and "degraded" them, like putting them through a bad phone connection, reversing parts of them, or muffling them. They found that humans are very sensitive to losing frequency (the pitch/tone) but are surprisingly tough when it comes to messing with time (speeding up or slowing down the sound).

2. The Computer Contestants (The Models)

Next, they invited three types of "computers" to take the same test to see who could mimic the human brain best.

• The Old School Models (The "Rulebook" Students): These were traditional models built by engineers who tried to copy the human ear using math formulas (like a cochlear filter).
  • The Verdict: They failed miserably. They were like students who memorized a rulebook but couldn't handle a real-life exam. They couldn't keep up with the messy, mixed-up sounds.
• The "From Scratch" Neural Networks (The "Fresh Graduates"): These were modern AI models (Deep Learning) that started with no knowledge and learned only from the specific sounds used in the test.
  • The Verdict: They did okay, much better than the old school models, but they still struggled with the trickiest parts of the test.
• The "Pre-Trained" Giants (The "World Travelers"): These were the same modern AI models, but before taking the test, they had already studied millions of hours of YouTube videos and audio clips from the internet (a dataset called AudioSet). They had seen almost every type of sound imaginable before.
  • The Verdict: They won. These models performed almost as well as humans. They didn't just get the right answers; they made the same kinds of mistakes humans did. If a sound was hard for a human to hear, it was hard for these models too. If a human could ignore a distortion, the model could too.

3. The "Brain Scan" Check

To make sure these "World Traveler" models weren't just lucky, the researchers looked at their "brains" (their internal processing layers) and compared them to actual human brain scans (fMRI).

• The Analogy: Imagine looking at the wiring inside a robot and comparing it to the wiring inside a human.
• The Finding: The models that performed best on the sound test also had the "brain wiring" that looked most like human brains. The better the model acted like a human, the more its internal structure resembled a human brain.

The Big Takeaway

The main lesson here is about experience.

Think of learning to recognize sounds like learning to drive.

• If you only practice in a quiet, empty parking lot (small datasets), you might pass a basic test, but you'll crash in a busy city.
• If you practice driving in every weather condition, on every type of road, and in every city imaginable (massive, diverse datasets), you become a master driver who handles chaos naturally.

The paper shows that to build a computer that "hears" like a human, we don't need to program it with complex rules about how ears work. Instead, we just need to let it "listen" to the world as much as possible. When AI is optimized to solve the real-world problem of recognizing sounds in a noisy world, it naturally develops human-like hearing abilities.

In short: The best way to teach a computer to hear is to let it listen to the whole world, not just a textbook.

Technical Summary

1. Problem Statement

Environmental sound recognition (identifying everyday sounds like footsteps, rainfall, or animal calls) is critical for human survival and daily life. However, unlike speech recognition, this domain lacks:

• Large-scale behavioral benchmarks: Previous studies used small stimulus sets (e.g., 70–168 sounds) and lacked standardized paradigms.
• Computational models: While machine hearing models (e.g., CNNs, Transformers) have improved automated recognition, they have not been systematically evaluated against human behavioral data to determine if they replicate human perceptual patterns.
• Understanding of robustness: It is unclear how human recognition degrades under concurrent sound sources (scene size) or various audio distortions (filtering, reverberation, time manipulation).

The authors aim to bridge this gap by creating a comprehensive benchmark to compare human performance with state-of-the-art computational models.

2. Methodology

A. The EnvAudioEval Benchmark (Human Experiments)

The authors conducted two large-scale online experiments (via Prolific) involving over 350 participants to establish a behavioral baseline.

• Task: A sound category detection task. Participants heard an auditory scene and judged whether a specific target category was present ("Yes"/"No").
• Experiment 1: Scene Size Effects:
  • Stimuli: Auditory scenes containing 1 to 5 concurrent sound sources (superimposed).
  • Goal: Measure how recognition accuracy ($d'$) declines as the number of sources increases.
  • Categories: 51 distinct environmental sound categories (subset of GISE-51).
• Experiment 2: Distortion Effects:
  • Stimuli: Single-source sounds subjected to 68 different types of distortions at varying levels.
  • Distortions: Included spectral filtering (high/low/band-pass), temporal manipulations (time dilation/compression, local time reversal), reverberation (varying DRR and RT60), clipping, noise vocoding, and modulation filtering.
  • Goal: Establish a "fingerprint" of human robustness to specific acoustic degradations.

B. Computational Models

The authors evaluated three classes of models on the same benchmark tasks:

1. Baseline Models (Biologically Inspired):
   • Cochleagram (Coch): Cochlear filterbank + Linear Classifier.
   • Spectrotemporal (ST): Cochlear stage + Spectrotemporal modulation filterbank (approximating auditory cortex) + Linear Classifier.
2. In-House CNN Models:
   • CochCNN: Fixed cochlear front-end + Convolutional Neural Network (CNN).
   • CochSTVGGish: Spectrotemporal front-end + CNN (VGGish architecture).
3. Pretrained/External Models:
   • VGGishPretrained: VGGish CNN pretrained on AudioSet (2M+ clips), fine-tuned on the benchmark.
   • SSASTPretrained: Self-supervised Audio Spectrogram Transformer (SSAST) pretrained on AudioSet/LibriSpeech, fine-tuned on the benchmark.
   • CochCNNPretrained: In-house CNN pretrained on AudioSet, fine-tuned.

Training Data: Models were trained/fine-tuned on the EnvAudioScene dataset (1.5 million synthesized scenes derived from the GISE-51 dataset, with realistic spatialization and reverberation).

C. Evaluation Metrics

• Behavioral Alignment: Comparison of model $d'$ scores against human $d'$ scores across categories, scene sizes, and distortions using correlation and Root Mean Squared Error (RMSE).
• Brain Alignment: Comparison of model representations against human auditory cortex fMRI data (from a previous study, Tuckute et al.) using:
  • Voxel-wise Predictivity: Linear regression of model features to fMRI voxel responses.
  • Representational Similarity Analysis (RSA): Correlation of Representational Dissimilarity Matrices (RDMs) between models and brain data.

3. Key Results

A. Human Performance Patterns

• Scene Size: Recognition accuracy ($d'$) declined reliably as the number of concurrent sources increased (from 1 to 5), though performance remained above chance even at 5 sources.
• Category Variability: Recognizability varied significantly by category (e.g., "coughing" was highly recognizable; "car" was less so). This pattern was highly reliable across participants.
• Distortion Robustness: Humans were highly robust to reverberation and temporal manipulations (time dilation) but significantly impaired by spectral filtering (loss of frequency information).

B. Model vs. Human Alignment

• Baseline Failure: Traditional biologically inspired models (Coch and ST) failed to match human performance levels or patterns, significantly underperforming humans.
• CNN Success: CNN-based models (CochCNN, VGGish) qualitatively replicated the decline in performance with increasing scene size.
• The Power of Pretraining: Models pretrained on large-scale datasets (AudioSet) and then fine-tuned (VGGishPretrained, SSASTPretrained, CochCNNPretrained) achieved the best results:
  • They reached near-human accuracy levels.
  • They showed the highest correlation with human performance across sound categories ($\rho \approx 0.88$) and distortions.
  • They were the only models that did not significantly underperform humans in category-wise comparisons.
• Limitations: Even the best models struggled with audio filtering distortions, suggesting they rely more heavily on spectral information than humans do, likely due to the specific distribution of training data.

C. Model-Brain Alignment

• Models that better replicated human behavior also exhibited better alignment with human fMRI data in the auditory cortex.
• Pretrained models showed superior brain predictivity (variance explained) and representational similarity compared to baseline and non-pretrained models.
• This confirms a positive correlation between task performance, human behavioral similarity, and neural similarity.

4. Key Contributions

1. EnvAudioEval Benchmark: The creation of a large-scale, standardized benchmark for environmental sound recognition involving 51 categories, 68 distortions, and multi-source scenes (up to 5 sources). This is significantly larger than previous studies (e.g., 2,176 stimuli vs. ~168 in prior work).
2. Systematic Model Evaluation: The first comprehensive comparison of diverse model classes (biological baselines, CNNs, Transformers) against human behavioral data in environmental sound recognition.
3. Evidence for Data-Driven Optimization: Demonstrated that optimizing models for real-world recognition tasks (via large-scale pretraining) yields systems that not only perform well but also mimic human perceptual limitations and neural representations.
4. Neural Validation: Established that models with high behavioral similarity to humans also possess high neural similarity, validating the use of these models as proxies for human auditory processing.

5. Significance and Future Directions

• Paradigm Shift: The results suggest that human-like auditory perception emerges naturally in systems optimized for real-world classification, rather than requiring explicit biological constraints in the architecture.
• Data is Key: The superior performance of pretrained models highlights that the scale and diversity of training data are more critical than specific architectural choices for achieving human-like robustness.
• Future Work:
  • Self-Supervised Learning: Exploring self-supervised training on even larger datasets to further close the gap with human performance.
  • Attention and Salience: Using the benchmark to study how attention influences source separation and salience in auditory scenes.
  • Spatial Audio: Extending the benchmark to binaural/spatialized scenes to test models on spatial cues.
  • Hierarchical Representation: Moving beyond fixed categorical labels to capture the hierarchical and physical nature of human sound understanding.

In conclusion, this paper provides a foundational framework for understanding environmental sound recognition, demonstrating that modern deep learning models, when trained on sufficient data, can effectively model both human behavior and the underlying neural mechanisms of auditory perception.