⚡ electrical engineering

Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio

This survey provides a comprehensive review and systematic taxonomy of end-to-end neural approaches for monaural multi-speaker automatic speech recognition, analyzing architectural paradigms, recent algorithmic improvements, long-form speech extensions, and benchmark performance while outlining future research directions.

Original authors: Xinlu He, Jacob Whitehill

Published 2026-05-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinlu He, Jacob Whitehill

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a crowded dinner party where three friends are talking all at once. You want to write down exactly what each person said, but their voices are blending together into a messy soup of sound. This is the challenge of Multi-Speaker Automatic Speech Recognition (ASR).

This paper is a "map" or a "survey" of the newest and smartest ways computers are learning to solve this problem, specifically when there is only one microphone (monaural audio) recording the chaos.

Here is a breakdown of the paper's main ideas using simple analogies:

1. The Old Way: The Assembly Line (Cascade Systems)

Before the new methods, computers tried to solve this like a factory assembly line.

Step 1: A machine listens and tries to figure out who is speaking and when (Speaker Diarization).
Step 2: It cuts the audio into separate clips based on those guesses.
Step 3: A different machine reads the text for each clip.

The Problem: If the first machine makes a mistake (e.g., it thinks two people are one person, or it misses a word), that error gets passed down the line. It's like a game of "Telephone" where the message gets garbled at every step. Also, if two people talk at the exact same time, the first machine often gets confused and drops the audio entirely.

2. The New Way: The End-to-End (E2E) Chef

The paper focuses on End-to-End (E2E) systems. Instead of an assembly line, imagine a single master chef who looks at the whole pot of soup (the mixed audio) and instantly knows how to separate the ingredients and write down the recipe for each person. This system learns to do the "who" and the "what" simultaneously, so mistakes in one area don't ruin the other.

The paper categorizes these new "chefs" into two main styles:

Style A: The Parallel Team (SIMO - Single Input, Multiple Output)

How it works: Imagine a team of three secretaries sitting at one desk. They all listen to the same noisy phone call. Each secretary is assigned a specific person (Secretary 1 listens only to Bob, Secretary 2 to Alice, etc.).
The Catch: You have to tell the team in advance exactly how many people are talking. If you say "There are 3 people," but a 4th person walks in, the system gets confused.
The Paper's Take: This style is good because it's modular (easy to swap out parts), but it struggles when the number of speakers changes or when the "separation" part isn't perfect.

Style B: The Single Scribe (SISO - Single Input, Single Output)

How it works: Imagine one very fast scribe who writes down everything that is said in one long, continuous stream. To keep track of who said what, they use special codes (like <sc> for "Speaker Change") inserted into the text.
The Advantage: This scribe doesn't need to know how many people are there beforehand. If a 4th person joins, they just keep writing. Because they hear the whole conversation at once, they can use the context (e.g., "Bob usually interrupts Alice") to figure out who is speaking.
The Paper's Take: This is very flexible and handles changing numbers of speakers well, but it requires the computer to be very smart about ordering the words correctly.

3. The "Secret Ingredients" (Improvements)

The paper notes that just having these two styles isn't enough. Researchers are adding "secret ingredients" to make them better:

The "Sidecar" (Pre-trained Models): Imagine taking a super-smart chef who already knows how to cook for one person (a model trained on massive amounts of single-speaker data) and giving them a small, lightweight assistant (a "Sidecar") to help them separate voices. This solves the problem of not having enough training data for multi-speaker scenarios.
Visual Cues (Audio-Visual): Sometimes the audio is too messy. The paper discusses systems that also look at a video of the speakers' faces. It's like having a human who can see who is moving their lips to help figure out who is speaking, even if the audio is noisy.
LLMs (The "Context" Brain): Using Large Language Models (like the AI behind chatbots) to help. These models can read instructions like, "Only write down what the woman says," or "Find the word 'meeting'." They act as a smart filter for the transcription.

4. The Long Story (Long-Form Audio)

Most tests use short clips (like a 10-second sentence). But real life is a 1-hour meeting.

The Challenge: How do you chop up a 1-hour recording without cutting a sentence in half?
The Solution: The paper discusses "Hypothesis Stitching." Imagine recording a long meeting in small chunks, transcribing each chunk, and then using a smart algorithm to glue them back together, making sure the speaker names stay consistent (so "Bob" in minute 1 is the same "Bob" in minute 50).

5. The Reality Check (What the Paper Found)

The authors looked at the scores (how accurate these systems are) on standard tests.

No Clear Winner: There isn't one "best" system. Sometimes the "Parallel Team" (SIMO) wins; sometimes the "Single Scribe" (SISO) wins. It depends on the specific situation.
Stagnation: Surprisingly, despite all the new fancy tech, the accuracy on real-world tests (like actual meeting recordings) hasn't improved much in the last few years.
The Bottleneck: The biggest problem isn't the algorithm; it's the data. We don't have enough high-quality recordings of people talking over each other to train these systems perfectly. Also, many researchers don't share their code, making it hard to compare who is actually doing the best.

Summary

This paper is a guidebook for the current state of "listening to a crowd." It tells us that while we have moved from clumsy assembly lines to smart, all-in-one systems, we are still struggling with the sheer chaos of overlapping voices. The future lies in combining these new smart systems with massive pre-trained models and perhaps using video or text context to help the computer "see" and "understand" the conversation better.

Technical Summary: Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio

1. Problem Statement

Monaural multi-speaker Automatic Speech Recognition (ASR) aims to transcribe audio containing multiple speakers, including overlapping speech, while attributing words to the correct individuals. This task extends the "cocktail party problem" by requiring the system to determine "who says what" rather than just "what was said."

The field faces three primary challenges:

Data Scarcity: There is a lack of large, well-annotated datasets for multi-speaker scenarios compared to single-speaker ASR.
Intrinsic Difficulty: Recognizing and attributing words in overlapping speech is inherently complex, involving not only recognition but also speaker diarization, overlap detection, and turn-taking analysis.
Limitations of Cascade Systems: Traditional modular approaches (e.g., Speaker Diarization $\to$ Segmentation $\to$ Single-Speaker ASR or Speech Separation $\to$ Single-Speaker ASR) suffer from error propagation. Diarization often assumes a single active speaker per frame, and separation modules optimize signal-level objectives rather than ASR performance, leading to compounded errors.

2. Methodology and Taxonomy

The paper provides a systematic taxonomy of End-to-End (E2E) neural approaches, categorizing them primarily by their output framework: Single-Input Multiple-Output (SIMO) and Single-Input Single-Output (SISO).

2.1. Single-Input Multiple-Output (SIMO)

SIMO frameworks process mixed audio and produce $S$ separate transcriptions (one per speaker) through parallel branches.

Architecture: Typically involves a shared mixture encoder followed by $S$ parallel branches, each containing a speaker-differentiating encoder and an ASR decoder.
Training: Relies on Permutation Invariant Training (PIT) to align output branches with reference transcriptions, as the order of speakers is unknown.
Limitations: Requires a fixed number of speakers ( $S$ ) a priori, limiting flexibility in real-world scenarios. Performance is heavily dependent on the quality of the separation module; poor separation leads to redundant or omitted transcriptions.
Recent Improvements:
- Separation Enhancement: Introducing auxiliary losses (e.g., KL divergence) to enforce distinctiveness between branches and cross-speaker encoders to share contextual information.
- Dynamic Handling: Iterative separation approaches that extract speakers one by one to handle variable speaker counts.
- Pre-training: Adapting large foundation models (e.g., Whisper, Wav2Vec) by inserting lightweight separation modules (e.g., "Sidecar" separators) between layers.

2.2. Single-Input Single-Output (SISO)

SISO frameworks process mixed audio to produce a single, unified transcription sequence containing all speakers' words.

Architecture: Generates a serialized output where transcriptions from different speakers are concatenated.
Training: Utilizes Serialized Output Training (SOT). Two main strategies exist:
- Sentence-based SOT: Concatenates sentences with speaker-change tokens (e.g., <sc>).
- Token-based SOT: Concatenates tokens based on timestamps with channel-change tokens (e.g., <cc>), offering finer granularity for overlaps.
Advantages: Naturally handles variable speaker counts, captures inter-speaker dependencies, and reduces computational overhead by avoiding permutation matching.
Improvements:
- Auxiliary Losses: Using modified CTC losses (e.g., Speaker-Aware CTC, SD-CTC) to enforce temporal consistency and speaker differentiation.
- External Speaker Modules: Integrating speaker embeddings via Frame-level or Token-level conditioning to inject speaker identity into the decoding process.
- Multi-Task Learning: Jointly optimizing ASR with auxiliary tasks like overlap detection, speaker change detection, or diarization.
- Foundation Model Adaptation: Directly fine-tuning or adapting large single-speaker foundation models (e.g., Whisper, WavLM) to the SISO format, often using lightweight adapters (e.g., LoRA).

2.3. Multi-Modal and Long-Form Extensions

Multi-Modal ASR: Incorporates visual cues (face tracks, mouth movements) or textual context (LLM instructions, diarization metadata). Visual cues help resolve speaker identity in noisy/overlapping conditions, while LLMs enable instruction-based control (e.g., "transcribe only the female speaker"). Most multi-modal systems adopt the SISO framework to fuse context directly into the joint modeling.
Long-Form Processing: Addresses continuous audio by introducing segmentation strategies (beyond simple VAD, such as adaptive sliding windows with semantic cues) and hypothesis stitching methods to merge local transcriptions into a globally consistent output with stable speaker identities.

3. Key Contributions

Comprehensive Taxonomy: The paper establishes a structured review of E2E multi-speaker ASR, distinguishing between SIMO and SISO paradigms and analyzing their architectural trade-offs.
Analysis of Improvements: It details recent algorithmic advancements, including separation enhancements, dynamic speaker handling, multi-task learning, and the adaptation of large foundation models.
Benchmark Evaluation: The authors provide a comparative analysis of methods across standard benchmarks (AMI, LibriSpeechMix, LibriMix), highlighting that no single architecture (SIMO vs. SISO) consistently outperforms the other across all scenarios.
Identification of Open Challenges: The survey identifies critical gaps, including the lack of open-source implementations for fair comparison, the stagnation of real-world benchmark progress despite methodological proposals, and the need for robust handling of overlapping speech.

4. Results and Performance

Benchmark Performance: On standard datasets like AMI and LibriMix, performance varies significantly based on training data (simulated vs. real) and speaker enrollment.
- SISO methods have shown strong performance on AMI (e.g., Conformer-based SISO models).
- SIMO methods have achieved competitive results on simulated datasets like LibriSpeechMix, particularly when leveraging pre-trained foundation models with separation modules.
Metric Evolution: The paper notes a shift in evaluation metrics from standard Word Error Rate (WER) to Speaker-Attributed WER (SA-WER) and concatenated minimum-permutation WER (cpWER) to account for speaker assignment accuracy.
Data Efficiency: Recent trends show that adapting pre-trained foundation models (often training only 8–10% of parameters) can achieve state-of-the-art performance, mitigating data scarcity issues.

5. Significance and Future Directions

The paper argues that the field is moving away from marginal WER improvements on specific benchmarks toward more robust, adaptive, and scalable systems. Its significance lies in:

Bridging the Gap: Providing the first comprehensive review of E2E multi-speaker ASR, filling a void in the literature.
Highlighting Trends: Identifying the shift toward foundation model adaptation and multi-modal integration as key drivers for future progress.
Call to Action: Emphasizing the need for standardized open-source benchmark suites and reproducible models to accelerate community progress, as current comparisons are hindered by inconsistent settings and limited code availability.

Future Directions identified by the authors include:

Hybrid Architectures: Combining the separation precision of SIMO with the comprehensive modeling of SISO.
Foundation Model Enhancement: Investigating how foundation-adapted models can better leverage cross-speaker context and multi-task learning.
Unified Frameworks: Integrating multi-speaker ASR into multi-modal Large Language Model (LLM) frameworks for more practical, instruction-based applications.
Robustness: Developing systems that can handle overlapping speech and variable speaker counts without performance degradation, potentially through optional speaker enrollment.