🤖 AI

A Multi-Branch Hierarchy-Aware Framework for Heterogeneous Audio Classification

This paper presents a multi-branch hierarchy-aware framework for the DCASE 2026 Challenge that leverages CLAP-based representations, augmented training data, and KNN-based post-processing to achieve state-of-the-art hierarchical F1 scores on heterogeneous audio classification tasks.

Original authors: Beile Ning, Jiayi Yu, Zitong Wang, Yufei Hu, Wenjun Xu, Yuanhang Qian, Zhongxin Bai, Gongping Huang

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Beile Ning, Jiayi Yu, Zitong Wang, Yufei Hu, Wenjun Xu, Yuanhang Qian, Zhongxin Bai, Gongping Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a bustling city. Your ears are bombarded with a chaotic mix of sounds: a siren, a dog barking, rain hitting a window, and a crowd cheering. Your brain doesn't just hear "noise"; it instantly sorts these sounds into a mental filing cabinet. It knows a siren is an "Emergency Vehicle" (Top Level) and specifically a "Police Car" (Second Level).

The DCASE 2026 Challenge asked computers to do exactly this: listen to messy, real-world audio recordings and sort them into a specific "Broad Sound Taxonomy" (BST). The catch? The computer had to get the specific sound right and make sure it fit the correct big category. If it guessed "Police Car," it couldn't accidentally call it a "Bird."

Here is how the team from Wuhan University built their "super-listener" to win this challenge, explained through simple analogies.

1. The Core Brain: The "CLAP" Translator

At the heart of their system is a pre-trained AI called CLAP. Think of CLAP as a polyglot translator who has read millions of books and listened to millions of songs. It understands the meaning of sound (e.g., "this sounds like a party") by connecting audio to text.

However, CLAP is like a generalist; it's great at big ideas but sometimes misses the tiny, specific details needed to distinguish between two very similar sounds. The team needed to give this generalist a set of specialized tools.

2. The Specialized Tools: Three Different "Ears"

To help CLAP hear the details, the team added three specialized "acoustic branches." Imagine giving your main brain three extra pairs of ears, each tuned to a different frequency:

The Log-Mel Ear: This ear is like a musician listening to the pitch and timbre of an instrument. It's great at recognizing the "shape" of the sound.
The MFCC Ear: This ear is like a linguist analyzing the structure of speech. It breaks the sound down into its building blocks.
The Log-STFT Ear: This ear is like a high-speed camera taking snapshots of the sound wave. It captures the rapid changes in the sound over time.

The Magic: The team didn't just pick one. They let all three "ears" listen to the sound, then combined their opinions with CLAP's "brain." The Log-STFT ear turned out to be the most sensitive listener on its own, acting as the star player.

3. The Training Ground: A Bigger, Cleaner Library

To teach the system, they needed a massive library of sound examples.

The Problem: Their original library (BSD10k) was good, but a bit small. They found a second, larger library (BSD35k), but it was messy—like a library where some books had the wrong titles or were written by unreliable authors.
The Solution: They created a new library called BSD-Grand. They acted like strict librarians:
- They checked the "author" (uploader) to ensure they weren't spamming the same sound 1,000 times.
- They used a "teacher model" to double-check the labels, throwing out the books with wrong titles.
- Result: A cleaner, bigger, and more diverse training set that helped the AI learn without getting confused by bad data.

4. The Rulebook: Hierarchy-Aware Thinking

The challenge required the AI to respect the "family tree" of sounds.

The Flat Approach: Imagine a student who just memorizes 23 specific answers without knowing the categories. They might guess "Police Car" correctly but fail to realize it belongs to "Emergency Vehicles."
The Hierarchical Approach: The team built a "Rulebook" into the AI. They used two strategies:
1. Global Classifier: The AI learns the big categories (5 groups) and the small categories (23 groups) at the same time, like a student studying both the chapter titles and the specific paragraphs.
2. Local Classifier (LCL): The AI first decides the big category, then zooms in to pick the specific sound. If it thinks the big category is "Nature," it won't even consider "Traffic" sounds. This prevents silly mistakes.

5. The Final Polish: The "KNN" Neighborhood Watch

Even after training, the AI sometimes hesitated. To fix this, the team added a KNN (K-Nearest Neighbors) post-processing step.

The Analogy: Imagine the AI is unsure if a sound is a "Cat" or a "Dog." Instead of guessing blindly, it looks at its "memory bank" of training sounds. It finds the 10 sounds that sound most like the current one. If 9 of those neighbors were "Cats," the AI updates its guess to "Cat."
Knowledge Distillation: They also used this "neighborhood watch" logic to teach the AI during training, essentially saying, "Look at what your neighbors think, and try to align with them."

The Results: How Did They Do?

The team submitted four different versions of their system, ranging from a single model to a "committee" of models voting together.

The Baseline: The starting point (without their improvements) scored 78.45%.
The Best Single Model: Using the "Log-STFT ear" and the "Neighborhood Watch," they jumped to 80.84%.
The Ensemble (The Committee): Their best system combined the Log-STFT ear, the Log-Mel ear, the Flat classifier, and the Local Classifier into one super-team. By averaging their votes, they achieved a score of 81.25%.

In Summary:
The team didn't invent a new type of hearing; they built a smarter system by giving a general AI (CLAP) specialized ears, cleaning up its training library, forcing it to follow a logical family tree of sounds, and letting it consult its "neighbors" before making a final decision. This combination allowed them to sort the chaotic sounds of the world with high accuracy.

Technical Summary: A Multi-Branch Hierarchy-Aware Framework for Heterogeneous Audio Classification

1. Problem Definition

This work addresses Task 1 of the DCASE 2026 Challenge, which focuses on the detection and classification of heterogeneous audio recordings according to the Broad Sound Taxonomy (BST). The challenge involves recognizing sound events and acoustic scenes from real-world recordings characterized by diverse content, recording conditions, and metadata quality.

The classification task is structured hierarchically:

Top-level: 5 broad classes.
Second-level: 23 specific categories grouped under the top-level classes.

The primary evaluation metric is the Hierarchical F1 score (Hier. F1). Consequently, a successful system must not only accurately predict the specific second-level class but also maintain consistency with the corresponding top-level category.

2. Methodology

The proposed system builds upon CLAP (Contrastive Language-Audio Pretraining) audio-text representations, enhancing them through three core strategies: dataset expansion, multi-branch acoustic modeling, and hierarchy-aware prediction refinement.

2.1. Dataset Expansion: BSD-Grand

To improve data diversity and mitigate label noise, the authors constructed an expanded training set named BSD-Grand. This dataset combines the official BSD10k-v1.2 training data with a filtered subset of BSD35k.

Filtering Strategy: The BSD35k subset underwent category-aware metadata cleaning (removing uploader-specific templates and noisy descriptions), teacher-model filtering using BSD10k-v1.2 models, and uploader-level constraints (capping samples per uploader to 200) to reduce bias.
Result: The final BSD-Grand training manifest contains 20,529 samples (10,956 from BSD10k-v1.2 and 9,573 from BSD35k).

2.2. Multi-Branch Acoustic Framework

The architecture extends the standard CLAP model by introducing feature-specific acoustic branches to complement the high-level semantic information captured by CLAP.

Acoustic Branches: Three distinct branches process different acoustic features, each encoding the input into a compact embedding fused with CLAP audio embeddings via a gated residual fusion module:
- log-Mel Branch: Uses a Multi-Dilated Frequency Dynamic Convolution (MDFD) encoder with attention pooling.
- MFCC Branch: Uses a residual Time Delay Neural Network (TDNN) with mean-max pooling.
- log-STFT Branch: Uses a 6-layer Transformer with temporal pooling.
Text Adaptation: A Highway Gate Adapter is applied specifically to the log-STFT branch. It generates a task-specific residual representation ( $\Delta t$ ) from the CLAP text embedding, controlled by a learnable highway gate to adaptively refine the text representation while preserving pre-trained semantics.

2.3. Hierarchy-Aware Prediction Heads

To exploit the hierarchical relationship between the 5 top-level and 23 second-level categories, the system investigates three classification head variants:

Flat Head: Projects fused representations directly to 23 classes, ignoring the top-level taxonomy.
Global Classifier (GC) Head: A multi-task learning approach outputting predictions for both top-level and second-level categories simultaneously, optimized via a weighted sum of cross-entropy losses.
Local Classifier per Level (LCL) Head: Trains separate classifiers for each level but employs a top-down correction mechanism during inference (e.g., probability multiplication) to enforce hierarchical consistency and prevent top-level errors from propagating to lower levels.

2.4. KNN-Based Post-Processing and Distillation

The system employs a K-Nearest Neighbors (KNN) strategy to refine predictions:

Post-Processing: A training embedding bank is constructed using concatenated feature representations from the trained log-STFT model and original CLAP embeddings. For evaluation samples, the final prediction is a weighted combination of the model's output and a neighbor prior derived from the $k$ -nearest neighbors in the bank.
Knowledge Distillation: The KNN-based neighbor prior is also used as soft supervision during training. A student model is optimized using both ground-truth labels and the KNN-based target distribution generated by a teacher model.

3. Experimental Results

Experiments were conducted using 5-fold cross-validation on the BSD10k-v1.2 dataset.

3.1. Component Analysis

Dataset Impact: Moving from the baseline (BSD10k-v1.2) to BSD-Grand improved the Hier. F1 from 78.45% to 79.64%.
Feature Branches: Among the acoustic branches, the log-STFT branch yielded the strongest single-model performance (80.54% Hier. F1), outperforming log-Mel (79.95%) and MFCC (80.13%).
Prediction Heads: The Flat head achieved the highest performance among single-head variants (80.57%), followed by LCL (80.20%) and GC (80.01%).
Post-Processing: Applying KNN-based post-processing to the log-STFT model further increased performance to 80.84% Hier. F1.

3.2. Submitted Systems Performance

The authors submitted four systems, with the best results summarized below:

System 1 (Single Model): log-STFT architecture with KNN post-processing. Achieved 80.84% Hier. F1.
System 3 (Ensemble, 5-fold CV): An ensemble of KD-log-STFT, log-Mel, Flat, and LCL models trained with 5-fold cross-validation. This system achieved the best overall performance of 81.25% Hier. F1.
System 4 (Extended Ensemble): An extended ensemble incorporating MFCC and GC heads. Achieved 81.18% Hier. F1.

All submitted systems significantly surpassed the baseline of 78.45%.

4. Significance and Claims

The paper claims that the proposed framework demonstrates consistent improvements over the CLAP-based baseline through:

Dataset Expansion: The construction of BSD-Grand effectively reduces label noise and uploader bias while increasing data diversity.
Feature Diversification: The integration of feature-specific acoustic branches (particularly log-STFT) complements the semantic richness of CLAP, addressing the limitations of relying solely on text-audio alignment.
Hierarchical Consistency: The use of hierarchy-aware heads (specifically LCL with top-down correction) and KNN-based refinement ensures that predictions respect the BST taxonomy structure.
Ensemble Synergy: The best results are achieved by combining models with complementary acoustic features and classification strategies, highlighting the benefits of ensemble learning for heterogeneous audio classification.

The authors conclude that their best ensemble system (System 3) achieves a Hier. F1 of 81.25%, validating the effectiveness of multi-branch acoustic modeling and hierarchy-aware refinement for the DCASE 2026 Task 1 challenge.