Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing

This paper proposes an efficient encoder-only multi-talker ASR framework that distills semantic priors from large language models into the encoder via a talker-aware teacher signal and utilizes a talker-count routing mechanism to achieve competitive performance with significantly lower inference latency compared to autoregressive LLM-based systems.

The Gist

Imagine you are at a very noisy party where three different people are talking over each other at the same time. Your goal is to write down exactly what each person said, in the order they started speaking. This is the challenge of Multi-Talker Automatic Speech Recognition (MT-ASR).

For a long time, computers struggled with this. If you asked a computer to listen to three people talking at once, it would get confused, mix up the words, or give up entirely.

Here is a simple breakdown of how the researchers in this paper solved that problem, using some fun analogies.

1. The Old Way: The "Super-Brain" Decoder

Previously, the best way to solve this was to use a Large Language Model (LLM) (like a super-smart AI chatbot) as the "decoder."

• The Analogy: Imagine you have a brilliant translator sitting at the party. They listen to the messy noise, pause to think deeply about the context, and then write down the sentences.
• The Problem: This translator is incredibly smart, but they are also slow and expensive. They need a lot of computing power to think. Also, if the noise is too heavy (three people shouting at once), even the smartest translator gets overwhelmed and makes mistakes. They are great at two people talking, but they struggle with three.

2. The New Idea: The "Smart Teacher" and the "Fast Student"

The authors of this paper came up with a clever trick. They wanted the speed of a simple system but the brainpower of the smart translator.

• The Setup: They built a "Student" (a fast, lightweight computer model) that only has an Encoder (the ears). It doesn't have a slow, thinking brain (decoder).
• The Teacher: They used the "Super-Brain" LLM as a Teacher, but only during training.
• The Process:
  1. Distillation (The Lesson): The Teacher listens to the messy party noise and figures out what was said. It then whispers the "secret meaning" and "context" to the Student. The Student learns to understand the vibe and logic of the conversation without needing the Teacher's slow brain.
  2. The Result: Once the Student has learned the lesson, the Teacher is fired. The Student can now listen to the party and write down the transcript instantly, using a fast method called CTC (which is like a rapid-fire typing system).

Why is this cool? You get the speed of a sprinter with the intelligence of a marathon runner.

3. The "Talker-Count" Problem: The Magic Switch

There was one big catch with previous fast systems: they were rigid. You had to tell the computer beforehand, "Okay, there are exactly two people talking." If you said "three," the system would break.

The authors added a Talker-Count Head (TCH).

• The Analogy: Imagine a bouncer at the door of the party. Before the music starts, the bouncer quickly counts the heads.
  • If they see two people, they open the "Two-Person Door" and send the audio to a specialized team trained for duos.
  • If they see three people, they open the "Three-Person Door" and send it to a team trained for trios.
• The Benefit: The system doesn't need you to guess the number of speakers. It figures it out on the fly and routes the audio to the right "brain" to handle it.

4. The Results: Speed vs. Smarts

The researchers tested this on a dataset called LibriMix (which is basically a library of recorded conversations mixed together).

• Two Talkers: The new system performed just as well as the slow, expensive "Super-Brain" systems.
• Three Talkers: This is where the magic happened. The old "Super-Brain" systems got confused and failed. The new "Fast Student" system, having learned the secrets from the teacher, actually did better than the slow systems.
• Speed: The new system is 10 to 20 times faster than the old LLM-based systems. It's like switching from a slow, heavy tank to a nimble sports car.

Summary

The paper is about teaching a fast, simple computer to listen to overlapping voices by letting a slow, smart AI teach it during practice. Once the lesson is learned, the smart AI leaves, and the fast computer handles the job alone. They also added a smart switch that automatically detects how many people are talking, so the system never gets confused by the crowd size.

In short: They made a speech-to-text system that is fast, smart, and flexible, capable of untangling the messiest conversations at a crowded party.

Technical Summary

Here is a detailed technical summary of the paper "Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing."

1. Problem Statement

Multi-Talker Automatic Speech Recognition (MT-ASR) aims to transcribe overlapping speech from multiple speakers. Current state-of-the-art approaches face a trade-off between performance and efficiency:

• LLM-Based Decoders: Using Large Language Models (LLMs) as autoregressive decoders (typically in an Encoder-Decoder architecture with Serialized Output Training, or SOT) provides strong semantic priors that help disentangle overlapping speech. However, these systems are computationally expensive, slow at inference, and often fragile when handling heavy overlap (e.g., three-talker mixtures).
• Encoder-Only Models: Models using only an encoder with Connectionist Temporal Classification (CTC) are fast and efficient. However, they often struggle with severe overlap because the encoder lacks the semantic guidance provided by LLMs, and training serialized CTC under heavy overlap can be unstable.
• Fixed Talker Count: Most existing CTC-based MT-ASR methods require the number of speakers to be fixed a priori, limiting their practicality in real-world scenarios where the number of speakers varies.

2. Methodology

The authors propose an Encoder-Only MT-ASR framework that distills the semantic knowledge of an LLM into the encoder during training while maintaining fast CTC-style decoding at inference. The architecture consists of three main components:

A. Architecture Overview

• Backbone: Uses WavLM-Large as the pretrained speech encoder.
• Branching: The model features a shared encoder followed by two specialized Transformer branches: one for 2-talker scenarios and one for 3-talker scenarios.
• Post-Encoder Separator: An LSTM-based module disentangles the mixed speech representation into speaker-specific streams ordered by speaking onset time.
• Serialized CTC: Each stream is decoded independently using CTC to produce talker-ordered transcripts.

B. LLM Adaptation and Knowledge Distillation

Instead of using the LLM as a decoder during inference, it serves as a train-time teacher:

1. Phase 1 (LLM Adaptation): A pretrained LLaMA decoder is adapted to multi-talker conditions using lightweight LoRA adapters and special token embeddings. This stage optimizes the SOT objective to align the LLM with overlapping speech cues.
2. Phase 2 (Distillation): The adapted LLM is frozen and used as a teacher signal. The encoder is trained with a hybrid loss function:
   $$L_{EncSep} = \alpha L_{Serialized-CTC} + (1 - \alpha) L_{SOT}$$
   Where $L_{SOT}$ is the cross-entropy loss between the encoder's output and the LLM's target sequence. This forces the encoder to learn semantic representations capable of handling overlap, even though the final inference relies solely on CTC.

C. Talker-Count Head (TCH)

To address the limitation of fixed talker counts, the authors introduce a Talker-Count Head:

• Function: Predicts the number of speakers (2 vs. 3) from the shared encoder output.
• Mechanism: Uses additive attention to compute mean and dispersion statistics of the encoder features, followed by a lightweight MLP to output class probabilities.
• Routing: During inference, the TCH dynamically routes the input to the appropriate 2-talker or 3-talker branch, eliminating the need for pre-specifying the speaker count.

3. Key Contributions

1. Semantic Distillation for Encoder-Only ASR: Successfully transfers the strong semantic priors of an LLM into an encoder-only model via a teacher-student distillation framework, achieving high performance without the inference cost of an autoregressive decoder.
2. Talker-Count Routing: Introduces a dynamic routing mechanism (TCH) that allows a single model to handle variable numbers of speakers (2 or 3), overcoming the rigidity of prior CTC-based methods.
3. Efficiency vs. Performance: Demonstrates that the proposed model achieves performance comparable to LLM-based systems in 2-talker scenarios and significantly outperforms them in challenging 3-talker scenarios, all while maintaining a drastically lower Real-Time Factor (RTF).

4. Experimental Results

Experiments were conducted on the LibriMix dataset (Libri2Mix and Libri3Mix).

• Performance (WER):

  • 2-Talker Condition: The proposed encoder-only model (ID-8/9) achieved Word Error Rates (WER) comparable to LLM-based SOT systems (e.g., ~4.0% WER on clean test data vs. 4.6% for LLM baselines).
  • 3-Talker Condition: The proposed model showed significant improvements over LLM-based systems. For example, on the noisy Libri3Mix evaluation set, the proposed model achieved 13.7% WER, outperforming the best LLM baseline (20.8% WER).
  • Stability: Training serialized CTC without the LLM distillation (ID-5) resulted in failure to converge effectively, proving the necessity of the semantic regularization.

• Efficiency (RTF):

  • The proposed CTC-based model achieved an RTF of 0.0043 (2-talker) and 0.0106 (3-talker).
  • In contrast, the LLM-based baseline had an RTF of 0.1150 and 0.0981, respectively. The proposed method is roughly 10x faster.

• Talker-Count Accuracy:

  • The TCH achieved high accuracy for 2-talker mixtures (>99%).
  • Accuracy for 3-talker mixtures was lower (~90-95%) but sufficient to provide consistent routing gains, leading to overall system improvements.

5. Significance

This work bridges the gap between the semantic robustness of large language models and the computational efficiency of encoder-only architectures.

• Practical Deployment: By removing the autoregressive decoder at inference, the system becomes viable for real-time, low-latency applications (e.g., live meeting transcription) where LLM decoders are too slow.
• Handling Complexity: The results suggest that the bottleneck in multi-talker ASR is not just the decoder's ability to generate text, but the encoder's ability to represent mixed speech. Distilling LLM priors directly into the encoder solves this representation bottleneck more effectively than relying on the decoder to disentangle the signal.
• Scalability: The Talker-Count Head offers a pathway toward truly variable-talker ASR systems that do not require manual configuration of speaker counts.