⚡ electrical engineering

Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition

The paper proposes Sparse MERIT, a multi-task learning framework utilizing a sparse mixture-of-experts architecture with dynamic frame-wise gating to effectively resolve task conflicts and significantly improve both speech enhancement and robust emotion recognition performance under challenging noisy conditions.

Original authors: Jing-Tong Tzeng, Carlos Busso, Chi-Chun Lee

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jing-Tong Tzeng, Carlos Busso, Chi-Chun Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Noisy Room" Dilemma

Imagine you are trying to listen to a friend tell a joke in a crowded, noisy bar.

The Goal: You want to understand the joke (Speech Emotion Recognition).
The Obstacle: The loud music and chatter are drowning out your friend's voice (Noise).

Traditionally, engineers tried to solve this in two separate steps:

Step 1: Hire a "Sound Cleaner" (Speech Enhancement) to turn down the music and make the voice clear.
Step 2: Send that cleaned voice to a "Joke Detector" (Emotion Recognition) to figure out if the friend is happy, angry, or sad.

The Catch: The "Sound Cleaner" is trained to make speech sound natural to human ears. Sometimes, in trying to clean up the noise, it accidentally scrubs away the tiny, subtle vocal cracks or pitch changes that tell us if someone is angry or sad. It's like a photo editor who removes the "noise" (grain) from a picture but accidentally smooths out the wrinkles on a face, making the person look emotionless.

The Old Solution: The "Shared Backpack"

Researchers tried to combine these two jobs into one team using Multi-Task Learning (MTL). They gave the team a single "backpack" (a shared neural network) to carry both the Sound Cleaner and the Joke Detector.

The Problem: The Sound Cleaner and the Joke Detector often want different things.

The Cleaner wants to fix the low-level sound waves.
The Detector wants to understand high-level feelings.
When they share one backpack, they start bumping into each other. One pulls the strap left, the other pulls right. This causes gradient interference—a fancy way of saying they get confused and stop learning effectively.

The New Solution: Sparse MERIT (The "Specialized Team")

The authors propose a new system called Sparse MERIT. Instead of one shared backpack, imagine a Swiss Army Knife or a Team of Specialized Chefs.

Here is how it works:

1. The Shared Library (Self-Supervised Backbone)

First, the system reads the noisy audio through a massive, pre-trained library (called WavLM). Think of this as a super-smart translator that has read millions of books and knows how language sounds, even when it's messy. It doesn't try to fix the noise yet; it just understands the raw material.

2. The "Expert" Kitchen (Mixture of Experts)

Instead of one chef cooking the whole meal, Sparse MERIT has a kitchen with three specialized chefs (called "Experts").

Chef A is great at fixing low, rumbling noises.
Chef B is great at preserving high-pitched emotional squeaks.
Chef C is great at general cleanup.

3. The Smart Waiter (The Gating Network)

This is the magic part. For every tiny slice of sound (a "frame"), a Smart Waiter (the Gating Network) looks at the audio and decides: "Who is the best chef for this specific slice?"

If the audio is a burst of angry shouting, the Waiter might send it to Chef B to preserve the anger.
If the audio is a low rumble of background noise, the Waiter sends it to Chef A to clean it up.

"Sparse" means: The Waiter only picks one chef for each slice of sound (Top-1 routing). It doesn't ask all three chefs to cook the same dish. This keeps the system fast and prevents the chefs from arguing with each other.

Why This is Better

The paper tested this system in a virtual "bar" with different levels of noise (from a quiet room to a hurricane of sound).

The Result: Sparse MERIT won the competition.
The Emotion Score: In the loudest conditions (-5 dB), it was 12% better at guessing emotions than the old "Clean then Detect" method. It was also 3.4% better than the old "Shared Backpack" method.
The Sound Quality: It also cleaned the audio better than the old methods, especially when the noise was something the system had never heard before (like a new type of crowd noise).

The "Aha!" Moment

The researchers found that by letting the system dynamically choose the right "expert" for every tiny moment of sound, they avoided the confusion of the old shared-backpack method.

Analogy: Imagine a classroom where one teacher tries to teach both advanced calculus and kindergarten art. They struggle to do both well.
Sparse MERIT: Instead, you have one head teacher who directs the students to three different specialist teachers. If a student needs math, they go to the math teacher. If they need art, they go to the art teacher. The students learn faster because the instruction is perfectly tailored to the moment.

What They Didn't Do (Important Limits)

They did not test this on real-world emergency calls or medical diagnoses. They only tested it on a dataset of podcast recordings (MSP-Podcast).
They did not claim this works on every type of noise (like a room with a massive echo), though they tested it on unseen noise types.
They noted a downside: This system requires a bit more computer memory (about 5GB extra) to train, like needing a bigger kitchen to store the extra chefs.

Summary

Sparse MERIT is a smart system that doesn't just "clean" noise and "guess" emotions separately. Instead, it uses a dynamic team of specialists that switch roles instantly, frame-by-frame, to ensure the voice stays clear and the emotion stays intact, even in the noisiest environments.

1. Problem Statement

Speech Emotion Recognition (SER) is critical for human-computer interaction but suffers significant performance degradation in real-world noisy environments. While Speech Enhancement (SE) can improve robustness, traditional approaches face two main limitations:

Two-Stage Pipeline Issues: Using SE as a pre-processing step introduces artifacts that obscure emotional cues and adds computational overhead. Furthermore, SE models optimize for perceptual intelligibility, which may not align with preserving emotion-discriminative features.
Multi-Task Learning (MTL) Conflicts: Jointly training SE and SER using a single shared backbone often leads to gradient interference and representational conflicts. The tasks differ significantly in complexity (SE requires fine-grained signal reconstruction, while SER requires high-level semantic abstraction), causing suboptimal convergence and biased feature representations in standard MTL models.

2. Methodology: Sparse MERIT

The authors propose Sparse Mixture-of-Experts Representation Integration Technique (Sparse MERIT), a flexible MTL framework designed to integrate self-supervised speech representations for both tasks.

Core Architecture

Backbone: Utilizes a pre-trained WavLM Large model (24-layer transformer) as a shared self-supervised backbone to extract robust speech representations.
Layer-Wise Representation: Instead of using a single layer, the model concatenates hidden states from the input and all 24 transformer layers to capture multi-scale contextual information.
Frame-Wise Sparse Routing:
- Expert Pool: A shared pool of $N=3$ expert networks (two-layer feedforward networks) processes the concatenated features.
- Task-Specific Gating: Two independent gating networks ( $G_{\phi}^{SER}$ and $G_{\phi}^{SE}$ ) dynamically route each frame to a specific expert.
- Top-K Selection: The framework employs a Top-1 routing strategy (Sparse routing), where each frame is processed by exactly one selected expert. This reduces computational overhead compared to dense routing and encourages specialized expert behavior.
Task Heads:
- SER Head: Uses attentive statistics pooling on the routed features followed by a classification head to predict emotion labels.
- SE Head: Concatenates the routed features with the log-compressed spectrogram of the noisy input and reconstructs the enhanced spectrogram using a decoder (based on BSSE-SE architecture).
Objective Function: Jointly optimizes a weighted cross-entropy loss (for SER, handling class imbalance) and an L1 loss (for SE, minimizing spectral reconstruction error).

3. Key Contributions

Novel MTL Framework: Introduction of Sparse MERIT, which moves beyond simple shared backbones by incorporating a Mixture-of-Experts (MoE) structure with frame-wise, task-specific routing. This mitigates gradient interference between SE and SER.
Efficient Representation Integration: The method leverages multi-layer self-supervised representations and sparse routing to enable parameter-efficient, task-adaptive learning without increasing inference costs significantly.
Comprehensive Validation: Extensive experiments demonstrate that Sparse MERIT outperforms state-of-the-art baselines, including SE pre-processing pipelines, naive MTL, and advanced gradient manipulation techniques (PCGrad, Uncertainty weighting).
Insight into Gating Dynamics: The paper provides an analysis of gating behavior, showing that expert selection is driven more by emotional content than noise levels, and that Top-1 routing yields better generalization than dense routing.

4. Experimental Results

Experiments were conducted on the MSP-Podcast corpus with noise augmentation (CRSS-4ENGLISH-14, Freesound, and ICASSP 2023 DNS Challenge).

Speech Emotion Recognition (SER)

Performance: Under the most challenging condition (-5 dB SNR) with unseen noise (Freesound), Sparse MERIT improved F1-macro by 12.4% over the SE pre-processing baseline and 3.8% over the naive MTL baseline.
Generalization: The model maintained robust performance across both seen and unseen noise conditions, whereas SE pre-processing baselines degraded significantly on unseen noise types.
Comparison: It outperformed other MTL strategies like Uncertainty-based weighting and PCGrad, and also surpassed a "Dense MERIT" variant (where all experts contribute to every frame), confirming the efficacy of sparse Top-1 routing.

Speech Enhancement (SE)

Metrics: Evaluated using PESQ, CSIG, CBAK, COVL, SSNR, and STOI.
Performance: Sparse MERIT consistently outperformed baselines on unseen noise conditions. At -5 dB SNR on the DNS dataset, it achieved a 44.4% improvement in SSNR and 19.6% in STOI compared to the SE-only fine-tuned baseline.
Trade-off: While it showed a slight drop in PESQ compared to some specialized SE models (due to lack of PESQ-specific discriminators), it significantly improved intelligibility (STOI) and overall quality (COVL) in joint learning scenarios.

Ablation Studies

Expert Count: SER performance peaked with 3 experts, while SE performance continued to improve slightly with more experts. The authors selected 3 to balance SER robustness and model complexity.
Balancing Loss: Contrary to standard MoE practices (e.g., Switch Transformer), adding an expert balancing loss degraded performance on unseen noise, suggesting that forcing uniform expert usage hinders task-specific specialization in this context.

5. Significance

Robustness in Real-World Scenarios: Sparse MERIT addresses the critical gap between clean-lab performance and real-world deployment by jointly optimizing for intelligibility and emotion recognition without the artifacts of two-stage pipelines.
Solving MTL Conflicts: It provides a structural solution (sparse MoE) to the gradient interference problem, proving that task-specific routing is more effective than gradient manipulation (like PCGrad) for heterogeneous tasks like SE and SER.
Practical Deployment: The framework offers a computationally efficient alternative to cascaded systems, making it suitable for resource-constrained environments where both human-audible output and automated emotion analysis are required.

In conclusion, Sparse MERIT demonstrates that integrating self-supervised representations with sparse, frame-wise expert routing creates a robust, generalizable system capable of handling complex, noisy acoustic environments for both speech enhancement and emotion recognition.