Embedding-Space Diffusion for Zero-Shot Environmental Sound Classification

Technical Summary: Embedding-Space Diffusion for Zero-Shot Environmental Sound Classification

Problem Statement
Zero-shot learning (ZSL) aims to enable models to classify unseen classes by leveraging semantic information, bridging the gap between training and testing sets with disjoint class sets. While ZSL has seen significant research in computer vision, its application to environmental audio remains underexplored and yields poor performance in existing studies. Current environmental audio ZSL methods largely rely on non-generative approaches (e.g., compatibility networks) or fail to utilize the success of generative models observed in vision tasks. Furthermore, existing generative approaches in ZSL often depend on large, pre-trained models (like Stable Diffusion) tied to specific modalities (images and text), limiting their applicability to audio domains where such pre-trained foundations may not exist or where feature data must remain unseen during pre-training.

Methodology
The paper proposes ZeroDiffusion, a novel generative method for zero-shot environmental sound classification that operates entirely within the embedding space, avoiding the need for pre-trained generative models. The methodology consists of three primary stages:

1. Embedding Generation:

   • Class Embeddings ($\sigma(z)$): Class labels are converted into 300-dimensional vectors using a pre-trained Word2Vec model. To enrich these embeddings, synonyms are added for most datasets (except ARCA23K-FSD and GTZAN due to size or genre constraints).
   • Audio Embeddings ($\tau(w)$): A modified YAMNet model is trained on seen classes to extract 128-dimensional audio embeddings from log Mel spectrograms. This ensures the feature extractor does not inadvertently "see" the unseen classes during pre-training.

2. Diffusion Model Training:

   • A lightweight, fully connected neural network is trained as a diffusion model conditioned on class auxiliary data.
   • Input: The model takes a noise-injected audio embedding ($\tilde{\tau}(w)$) concatenated with a jittered class embedding ($\tilde{\sigma}(z)$).
   • Objective: The model learns to reconstruct the original audio embedding from the noisy input, conditioned on the class embedding.
   • Loss Function: The training objective combines five components: Mean Squared Error (reconstruction), Maximum Mean Discrepancy (distribution alignment), variance loss (diversity), global centroid alignment, and cosine similarity loss (directional alignment).
   • Generation: Once trained, the model generates synthetic audio embeddings for unseen classes by feeding them random noise and their corresponding class embeddings.

3. Classifier Training:

   • A non-linear compatibility network (based on Attribute Label Embedding, ALE) is trained using a combination of real audio embeddings from seen classes and the synthetic embeddings generated for unseen classes.
   • The classifier learns a similarity function $f(w, z)$ to map feature data to class labels without ever having seen the actual audio samples of the unseen classes.

Experimental Setup
The study evaluates four methods across six datasets: ESC-50, ARCA23K-FSD, FSC22, UrbanSound8k, TAU Urban Acoustics 2019, and GTZAN.

• Baselines:
  • Non-linear ALE: A state-of-the-art non-generative baseline for environmental audio.
  • CADA-VAE: A cross-aligned and distribution-aligned Variational Autoencoder adapted from computer vision.
  • LisGAN: A Leveraging Invariant Side Generative Adversarial Network, adapted to the inductive setting (removing transductive components).
• Protocol: Experiments were repeated ten times with different random seeds. Standard non-linear ALE partitions were used for ESC-50, while new random partitions were created for the other datasets.

Key Results

• Performance: ZeroDiffusion achieved the highest average accuracy across all six datasets (34.86%), outperforming the non-linear ALE baseline (32.27%), CADA-VAE (31.37%), and LisGAN (29.36%).
• Dataset Specifics:
  • ZeroDiffusion showed significant improvements on ARCA23K-FSD and TAU Urban Acoustics 2019.
  • It performed competitively with baselines on UrbanSound8k and GTZAN.
  • On ESC-50 and FSC22, results were mixed; ZeroDiffusion did not consistently outperform all methods on every fold, and performance was notably unstable on FSC22 test sets.
• Stability: Generative methods (including ZeroDiffusion) exhibited higher standard deviations across runs compared to the non-generative ALE baseline. For instance, CADA-VAE showed up to 8.08% standard deviation on UrbanSound8k, while ALE remained around 1%.
• Data Dependency: The authors observed that ZeroDiffusion's performance correlated with dataset size. Datasets with more samples per class (e.g., TAU 2019 with 1,440 samples/class) yielded better results, whereas smaller datasets (ESC-50 with 40 samples/class) showed instability.

Contributions and Significance
The paper claims the following contributions:

1. First Embedding-Space Diffusion for ZSL: Introduces the first diffusion-based method for zero-shot learning that operates in the embedding space without relying on pre-trained generative models (like Stable Diffusion), making it modality-agnostic.
2. New State-of-the-Art: Establishes a new state-of-the-art for zero-shot environmental sound classification, outperforming or matching existing methods across six diverse audio datasets.
3. New Benchmarks: Provides the first zero-shot learning benchmarks for FSC22, ARCA23K-FSD, and TAU Urban Acoustics 2019.
4. Comprehensive Comparison: Offers the first direct comparison of GANs (LisGAN), VAEs (CADA-VAE), and Diffusion models for zero-shot environmental sound classification.

Significance and Limitations
The authors position this work as establishing diffusion models as a promising approach for zero-shot learning in audio, providing a foundation for future research. However, the paper is modest regarding the reliability of generative methods. It explicitly acknowledges that while ZeroDiffusion improves accuracy, it suffers from higher variance and instability compared to non-generative baselines, particularly on smaller datasets. The authors suggest that the lack of theoretical guarantees in zero-shot learning and the dependency on dataset size are critical areas for future understanding. The work does not claim to solve the instability of generative models but rather demonstrates their potential for higher accuracy when sufficient data is available.