⚡ electrical engineering

Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss

This paper proposes a semi-supervised sound event detection framework that integrates conditional mixup with embedding-level contrastive loss to effectively leverage abundant unlabeled data, achieving state-of-the-art performance on the DESED dataset.

Original authors: Nian Shao, Xian Li, Xiaofei Li

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nian Shao, Xian Li, Xiaofei Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different sounds in a busy kitchen: a blender whirring, a dog barking, or water running. This task is called Sound Event Detection (SED).

The problem is that teaching a robot this way is like trying to teach a child to speak by only showing them a few flashcards. You have plenty of recordings of kitchen sounds (unlabeled data), but you only have a tiny number of recordings where someone has carefully written down exactly when the blender started and stopped (labeled data). Without those precise notes, the robot gets confused.

This paper presents a new way to train the robot using both the few "note-taking" examples and the mountain of "no-notes" examples. Here is how they did it, explained simply:

1. The Starting Point: The "Teacher" and the "Student"

The researchers started with a smart robot that had already learned a lot about sound from a massive library of audio (a pre-trained model called ATST-Frame). Think of this robot as a Student who is already good at hearing, but needs to learn to spot specific events.

To teach the Student using the "no-notes" recordings, they used a Teacher. The Teacher is a copy of the Student that is slightly more stable. The Teacher guesses what sounds are in the unlabeled recordings, and the Student tries to match those guesses. This is called "pseudo-labeling."

2. The Problem: Mixing Ingredients Wrong

To make the training even better, the researchers used a technique called Mixup. Imagine you have two audio clips: one of a dog barking and one of a blender.

The Old Way: They would mix these two sounds together.
- If they mixed them 50/50, they told the robot: "This is a new sound where a dog and a blender are happening at the same time." (This is Composition).
- If they mixed them 90/10 (mostly dog, tiny bit of blender), they told the robot: "This is just a dog, but the blender is a little bit of background noise." (This is Perturbation).

The problem was that the old training system used the same mixing rule for everything. But the "Teacher" (pseudo-labeling) needed the 50/50 mix to learn about co-occurring events, while the "Self-Check" (contrastive learning) needed the 90/10 mix to learn that the sound shouldn't change too much. Using the wrong mix confused the robot.

3. The Solution: "Conditional Mixup"

The authors invented a smart switch called Conditional Mixup.

The Switch: They look at the mixing ratio (the "lambda" value).
If the mix is balanced (50/50): The system treats it as Composition. It tells the robot, "Hey, both sounds are real here! Learn to detect them together."
If the mix is unbalanced (90/10): The system treats it as Perturbation. It tells the robot, "The main sound is still the dog; the blender is just a tiny glitch. Don't get confused."

By using this switch, the robot gets the right instruction for every single mixed-up audio clip.

4. The Secret Sauce: "Embedding-Level Contrastive Loss"

Usually, robots are taught to match their final answer (e.g., "Is this a dog? Yes/No"). This paper adds a second layer of training that happens inside the robot's brain (in the "embeddings").

Think of it like this:

Standard Training: The teacher says, "That's a dog."
New Contrastive Training: The teacher says, "Even if you add a little bit of blender noise to the dog bark, the feeling of the sound in your brain should still feel like a dog."

This forces the robot to build a very strong internal understanding of what a "dog sound" really is, regardless of minor background noise. This helps the robot use the unlabeled data much more effectively.

The Results

By combining these two ideas—Conditional Mixup (the smart switch) and Embedding Contrastive Learning (the internal feeling check)—the robot became much better at its job.

On the standard test (the DESED validation set), this new system (called ATST-SEDv2) achieved the highest scores ever recorded for this specific task:

0.645 on a score called PSDS1 (measuring how stable and accurate the timing is).
0.822 on a score called PSDS2 (measuring how well it identifies the types of sounds).

In short, the paper didn't just give the robot more data; it taught the robot how to interpret that data more intelligently by using the right "mixing recipe" for every situation.

Technical Summary: Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss

Problem Statement

Sound Event Detection (SED) is a critical component of acoustic environmental analysis but faces significant challenges due to the scarcity of frame-level labeled data. While recent advancements leverage large-scale pretrained audio foundation models (e.g., ATST-Frame, BEATs, AST) to provide robust representations, fine-tuning these models effectively remains difficult. Existing semi-supervised approaches often rely on consistency regularization (e.g., Mean Teacher) and pseudo-labeling. However, these methods may be insufficient for large pretrained encoders, as consistency objectives are often weaker than the original self-supervised pretraining tasks. Furthermore, a specific technical mismatch exists in the application of Mixup:

Pseudo-label learning treats Mixup as composition, where mixed samples preserve the content of both source clips.
Contrastive learning treats Mixup as perturbation, where the auxiliary sample serves only to perturb the input.
Current frameworks struggle to unify these conflicting views within a single semi-supervised pipeline.

Methodology

The proposed work, ATST-SEDv2, extends the ATST-SED baseline by integrating an embedding-level self-supervised contrastive objective into the semi-supervised fine-tuning framework. The architecture consists of an ATST-SED encoder (combining a CRNN and a pretrained ATST-Frame model) and a non-linear SED classifier.

1. Conditional Mixup

To resolve the semantic mismatch between composition and perturbation views of Mixup, the authors propose Conditional Mixup. The interpolation coefficient $\lambda$ (drawn from a Beta distribution) determines the mixup mode based on a threshold $\tau$ :

Composition Mode ( $0.5 \le \lambda \le \tau$ ): The auxiliary signal is treated as co-occurring with the input. The mixed sample is formed by linear interpolation, but the target label is the logical OR of the source labels (for supervised) or the sum of pseudo-labels (clamped at 1 for semi-supervised).
Perturbation Mode ( $\tau < \lambda \le 1$ ): The auxiliary signal is treated as a perturbation. The mixed sample is formed similarly, but the target label remains that of the original input sample.

2. Loss Functions

The training objective combines three components:

Supervised Loss: Binary Cross-Entropy (BCE) on labeled data.
Pseudo-Label Loss: A unified loss ( $L_{cp-psd}$ ) that applies the appropriate loss function (composition or perturbation) based on the conditional Mixup mode. This includes the Mean Teacher loss for un-augmented data.
Embedding-Level Contrastive Loss ( $L_{con}$ ): Inspired by ATST-Frame pretraining, this loss aligns embeddings of augmented views.
- For Perturbation pairs, the objective maximizes similarity between the input and the perturbed view.
- For Composition pairs, the objective maximizes similarity between the mixed view and the average of the teacher embeddings of both source samples.
- A projector and predictor are attached to the encoder during training to facilitate this contrastive learning, preventing representation collapse.

The total loss is a weighted sum of BCE, pseudo-label losses, and the contrastive loss.

Key Contributions

Conditional Mixup: A novel strategy that unifies composition and perturbation Mixup within a single semi-supervised framework by dynamically selecting the semantic view based on the interpolation coefficient.
Unified Loss Design: The design of corresponding pseudo-label and embedding-level contrastive losses that allow decision-level supervision and embedding-level supervision to complement each other during fine-tuning.
State-of-the-Art Performance: The method achieves new best results on the DESED validation set without altering the inference architecture.

Experimental Results

Experiments were conducted on the DESED dataset, utilizing strongly labeled, weakly labeled, and unlabeled data.

Baseline Comparison: The proposed method (ATST-SEDv2) improved the baseline ATST-SED (without the proposed method) from 0.583 to 0.607 in PSDS1 and 0.810 to 0.817 in PSDS2.
Lightweight Models: When applied to a CRNN baseline, the method also yielded improvements (PSDS1: 0.384 $\to$ 0.403), demonstrating its effectiveness across different model capacities.
Ablation Studies:
- Using only the contrastive loss improved performance over the frozen stage-1 model but was less effective than combining it with pseudo-label losses.
- Both the composition-only and perturbation-only branches were effective individually, but combining all components yielded the best results.
State-of-the-Art (SOTA): With cSEBB post-processing, ATST-SEDv2 achieved 0.645 PSDS1 and 0.822 PSDS2, surpassing previous SOTA methods including ATST-SED, MAT-SED, and PMAMiter2.

Significance

The paper claims that the proposed approach effectively leverages unlabeled data by addressing the semantic mismatch in Mixup strategies. By introducing an embedding-level contrastive objective alongside pseudo-labeling, the method enhances the semi-supervised fine-tuning of both lightweight and pretrained-based SED models. The results establish a new state of the art on the DESED benchmark, demonstrating that decision-level and embedding-level supervision can be successfully integrated to improve temporal stability and tagging accuracy in complex polyphonic audio scenes.