⚡ electrical engineering

Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder

This paper introduces a unified multi-speaker encoder (UME) that jointly learns speaker diarization, speech separation, and multi-speaker ASR through a shared foundational architecture and residual weighted-sum encoding, achieving state-of-the-art performance on overlapping speech datasets by effectively leveraging inter-task dependencies.

Original authors: Muhammad Shakeel, Yui Sudo, Yifan Peng, Chyi-Jiunn Lin, Shinji Watanabe

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Shakeel, Yui Sudo, Yifan Peng, Chyi-Jiunn Lin, Shinji Watanabe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a busy dinner party where three people are talking over each other, and there's a loud radio playing in the background. Your goal is to do three things at once:

Speaker Diarization: Figure out who is speaking and when.
Speech Separation: Physically separate the voices so you can hear each person clearly, like turning down the volume on the others.
Automatic Speech Recognition (ASR): Write down exactly what each person said.

Usually, computers try to solve these problems one by one, like having three different specialists working in isolation. But the authors of this paper, Muhammad Shakeel and his team, asked: "What if we built one super-smart brain that learns to do all three jobs simultaneously?"

Here is how they did it, using simple analogies:

The "Universal Translator" Brain (The Encoder)

The team started with a pre-trained "Speech Foundation Model" called OWSM. Think of this model as a highly educated student who has read millions of books and listened to thousands of hours of clear, single-person speech. It's great at understanding language, but it's never been in a noisy, crowded room with people talking over each other.

The researchers took this "student" and gave it a new job: The Unified Multi-Speaker Encoder (UME). Instead of just listening to one voice, this new system has to handle the chaos of the dinner party.

The Secret Sauce: The "Residual Weighted-Sum" (RWSE)

This is the paper's most creative innovation.

When a deep learning model processes speech, it goes through many layers, like floors in a skyscraper.

The bottom floors hear the raw sounds (beeps, buzzes, pitch).
The middle floors start to understand syllables and words.
The top floors understand the full meaning and context.

Previous models usually just took the answer from the very top floor. But the authors realized that for a chaotic room, you need information from every floor.

They created a technique called Residual Weighted-Sum Encoding (RWSE). Imagine a team of managers (the different layers of the model) passing notes to a final decision-maker.

Instead of just listening to the CEO (the top layer), the decision-maker listens to a weighted mix of notes from the CEO, the middle managers, and the entry-level staff.
The system learns to decide how much to listen to each floor. Sometimes the raw sound matters most; sometimes the context matters most.
This creates a "bottom-up alignment," ensuring that the system understands the who, the what, and the when all at the same time, because they are all talking to each other constantly.

The Three-Way Race

The system is trained using a "multi-task learning" approach. Imagine a student taking a final exam where they get points for:

Correctly identifying who is talking.
Correctly separating the voices.
Correctly transcribing the text.

If the student gets stuck on one part, the other parts help them figure it out. For example, if the system isn't sure who is speaking, the fact that it can separate the voices helps it guess the speaker. If it can't separate the voices, the text transcription might give it a clue. They all help each other, reducing the chance of error.

The Results: A New Record

The team tested their system on simulated noisy conversations (LibriMix datasets) where two or three people talked over each other with background noise.

The Outcome: Their "Unified Brain" (UME) significantly outperformed previous models that tried to do these tasks separately.
The Highlight: For the task of figuring out who spoke when (Speaker Diarization), they achieved a near-perfect score, making fewer than 2% errors even in the messiest three-person conversations. This is better than previous state-of-the-art models, even though their starting model (OWSM) was only trained on clean, single-person speech.

Why It Matters (According to the Paper)

The paper claims that by unifying these tasks, the system learns a "shared representation space." In plain English: the computer has built a single, robust understanding of human speech that handles the chaos of real-life conversations better than any specialized tool could on its own.

They didn't just build a better tool; they proved that teaching a computer to do three related jobs at once makes it smarter at all of them, especially when dealing with overlapping speech.

Technical Summary: Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder

Problem Statement
Speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) are critical tasks for understanding "who spoke what and when." Traditionally, these tasks have been addressed by independent single-task models that operate on acoustic information in isolation. This approach fails to leverage the inherent inter-dependencies among the tasks. While recent frameworks have attempted to unify these tasks, they often follow a fixed, optimal order (e.g., SD $\to$ SS $\to$ ASR) or rely on pre-trained speech foundation models (SFMs) that struggle with multi-speaker conversations. Furthermore, existing studies suggest that different layers of an encoder capture different semantic information (e.g., initial layers for SD, final layers for ASR), yet most unified models do not effectively utilize this hierarchical information across all tasks simultaneously.

Methodology: Unified Multi-Speaker Encoder (UME)
The authors propose the Unified Multi-Speaker Encoder (UME), an end-to-end (E2E) framework designed to jointly optimize SD, SS, and multi-speaker ASR using a shared speech foundation model (SFM) encoder.

Shared Encoder: The framework utilizes the OWSMv3.1 (Open Whisper Style Speech Model) encoder as a shared feature extractor. Unlike previous SFMs trained solely on single-speaker tasks, OWSMv3.1 is adapted here for multi-speaker scenarios.
Residual Weighted-Sum Encoding (RWSE): A core innovation is the extraction of hidden representations from multiple layers of the encoder. Instead of relying on a single layer or a simple weighted sum, UME employs a residual weighted-sum encoding. This involves:
1. Computing a weighted sum ( $H_{ws}$ ) of all encoder layers using task-specific learnable weights ( $\omega^{(l)}_{task}$ ).
2. Adding this weighted sum to the final encoder layer ( $H^{(L)}$ ) to amplify the influence of the final layer while integrating information from intermediate semantic levels.
3. This mechanism facilitates bottom-up alignment and information exchange between tasks.
Task-Specific Branches:
- Speaker Diarization (SD): Uses an End-to-End Neural Diarization (EEND) model with Permutation Invariant Training (PIT). It predicts binary speaker activity labels using the RWSE features.
- Speech Separation (SS): Incorporates a Conv-TasNet architecture. The RWSE features are upsampled and concatenated with the output of a 1-D convolutional encoder. These combined features pass through Temporal Convolutional Networks (TCNs) to generate masks for separating source signals.
- Multi-speaker ASR: Extends a joint CTC/attention-based framework. The RWSE features are processed through speaker-differentiating encoder blocks ( $SpeakerEnc^{c}_{SD}$ ) to extract speaker-specific features, which are then decoded using a transformer-based decoder with PIT to handle speaker permutation.
Optimization: The model is trained using a weighted sum of three loss functions: diarization loss (BCE), separation loss (SI-SDR), and ASR loss (CTC + Cross-Entropy). The weights ( $\lambda_{diar}, \lambda_{sep}, \lambda_{asr}$ ) are empirically tuned.

Key Contributions

Unified Framework: The proposal of a single network that jointly optimizes SD, SS, and multi-speaker ASR, moving away from sequential or independent task processing.
RWSE Mechanism: The introduction of Residual Weighted-Sum Encoding to leverage hidden representations from all encoder layers, hypothesized to improve bottom-up alignment and task inter-dependency.
Performance Gains: Demonstration of substantial performance improvements over single-task baselines and previous unified approaches on overlapping speech data.

Experimental Results
The framework was evaluated on the Libri2Mix (2 speakers) and Libri3Mix (3 speakers) datasets, including both noisy ("mixboth") and clean ("mixclean") conditions with 100% overlap.

Speaker Diarization: UME achieved state-of-the-art results. On the Libri2Mix clean evaluation set, it achieved a Diarization Error Rate (DER) of 1.37%, and 2.29% on Libri3Mix. These results significantly outperform previous studies, including those using WavLM Large, despite OWSMv3.1 being pre-trained on clean speech only.
Speech Separation: The unified approach showed consistent improvements over separation-only baselines (Conv-TasNet). On Libri2Mix clean data, the SI-SNR improved to 17.06 dB compared to the baseline's 10.93 dB.
Multi-speaker ASR: The framework demonstrated robustness across different overlap settings. On the Libri2Mix clean set, it achieved a Word Error Rate (WER) of 6.4%, outperforming strong baselines like SOT-Conformer and Whisper-medium-SS-TTI.
Ablation Studies: The results confirmed that initializing the ASR branch with a pre-trained model is crucial for training stability, particularly in the three-speaker scenario where training without initialization led to divergence. The RWSE mechanism was also shown to be effective compared to simple weighted sums or single-layer inputs.

Significance and Claims
The paper claims that the UME framework successfully creates a shared representation space for SD, SS, and multi-speaker ASR, allowing the tasks to benefit from each other's inter-dependencies. By utilizing the hierarchical information within the SFM encoder via RWSE, the model achieves superior generalization and performance without requiring task-specific pre-training for the encoder. The authors note that this approach effectively minimizes error accumulation common in cascaded systems. While the current work focuses on English datasets (LibriMix), the authors express interest in extending the framework to multilingual scenarios and more challenging real-world datasets like CHiME-6 in future work.