Technical Summary: CALM – Joint Contextual Acoustic-Linguistic Modeling for Multi-Speaker ASR

1. Problem Statement

Single-speaker automatic speech recognition (ASR) systems have achieved state-of-the-art performance, yet their efficacy degrades significantly in multi-speaker environments characterized by overlapping speech and conversation-specific vocabulary. This degradation stems from two primary sources:

1. Acoustic Challenges: Suboptimal speaker attribution due to interference from non-target talkers in overlapping segments.
2. Linguistic Challenges: Limited adaptability to domain-specific language, including proper nouns, acronyms, and technical jargon that are sparse or unseen in training data.

Prior approaches have largely treated these issues separately. Acoustic errors are typically mitigated via target-speaker (TS) conditioning (e.g., speaker embeddings), while linguistic adaptability is handled through contextual biasing (CB) or shallow fusion. However, existing work often evaluates these methods under single-speaker or non-overlapping conditions, or relies on serialized output training (SOT) frameworks that struggle in highly overlapping scenarios. Crucially, prior methods neglect the inter-dependencies between acoustic and linguistic contexts, addressing acoustic errors alone proving insufficient when linguistic errors (e.g., from unseen vocabularies) remain orthogonal sources of degradation.

2. Methodology: The CALM Framework

The authors propose CALM (Joint Contextual Acoustic-Linguistic Modeling), an end-to-end framework that integrates target-speaker conditioning with dynamic vocabulary-based contextual biasing. The architecture is designed to ensure robust contextual adaptation in overlapping speech through the following components:

2.1. Acoustic and Speaker Modeling

• Feature Extraction: The system employs WavLM-Large as an upstream model to extract frame-level features, which are projected into the encoder space.
• Speaker Encoder: Target-speaker cues (enrollment utterances) are processed by an ECAPA-TDNN with attentive pooling, followed by a RawNet3 projector, to generate a speaker embedding ($E_s$).
• Audio Encoder: A Conformer encoder maps the projected features to hidden states.
• Speaker Adaptation: The speaker embedding $E_s$ modulates the hidden representations of the intermediate and final audio encoder states using a FiLM (Feature-wise Linear Modulation) mechanism. This allows the acoustic model to selectively amplify features relevant to the enrolled speaker.
• Auxiliary VAD: A Voice Activity Detection (VAD) head is attached to the final adapted encoder states to compute frame-level target-speaker activity posteriors, regularizing the acoustic modeling.

2.2. Dynamic Vocabulary and Contextual Biasing

• Bias Encoder: A Transformer-based encoder (with mean pooling) encodes a biasing list $B$ (a set of phrases) into phrase-level vector representations.
• Dynamic Vocabulary Expansion: The framework extends the static vocabulary ($V_{stat}$) with a dynamic vocabulary ($V_{d-vocab}$) derived from the biasing list. Each biasing phrase is treated as a single dynamic token.
• Joint Output Layers: The model computes alignment scores for both static and dynamic vocabularies. These are concatenated and passed through a weighted softmax (adapted from prior work) controlled by a biasing weight hyperparameter ($\mu$). This mechanism prevents over-biasing or under-biasing during inference.
• Self-Conditioning: Similar to prior work, the posterior probabilities from intermediate layers are added back to the encoder input, allowing dynamic vocabulary information to propagate throughout the encoder.

2.3. Training Objective

The model is optimized using a weighted multitask loss function combining:

1. CTC Loss: Computed on the final layer and selected intermediate layers (interCTC).
2. Attention Loss: Computed via a dynamic vocabulary-based attention decoder that predicts tokens autoregressively.
3. VAD Loss: Binary cross-entropy between predicted and ground-truth activity labels.

3. Key Contributions

The paper outlines three primary contributions:

1. Unified Framework: A novel architecture that integrates target-speaker conditioning with dynamic vocabulary expansion, enabling joint acoustic-linguistic adaptation.
2. Scalable Personalization: A mechanism for personalized contextual adaptation that remains effective across varying biasing list sizes and dynamic multi-speaker conversational scenarios.
3. Extensive Evaluation: Comprehensive experiments on English (LibriSpeechMix, AMI) and Japanese (CSJMix) datasets, demonstrating substantial gains over state-of-the-art baselines.

4. Experimental Results

The authors evaluated CALM on simulated English (LibriSpeechMix), simulated Japanese (CSJMix), and standardized real-world mixtures (AMI-IHM-Mix).

4.1. English (LibriSpeechMix)

• Performance: On two-speaker mixtures (LibriSpeech2Mix), CALM reduced the Biased Word Error Rate (B-WER) from 12.7 to 4.7 (with a biasing list of 2000 entries) compared to a baseline TS-ASR model.
• Generalization: Unlike conventional contextual biasing methods where performance degrades as the biasing list grows, CALM showed consistent improvements in both B-WER and Unbiased WER (U-WER) as the list size increased.
• Comparison: CALM outperformed reproduced baselines and prior target-speaker ASR approaches (e.g., Whisper-large-SS-TTI, Transformer SA-ASR), achieving the lowest overall WER (3.56 on LibriSpeech2Mix).

4.2. Japanese (CSJMix)

• Cross-Linguistic Validation: CALM demonstrated robustness in Japanese, a character-based language. On CSJMix2, the Biased Character Error Rate (B-CER) dropped from 16.6 to 8.4 (eval3) with a biasing list of 100.
• Consistency: The framework maintained stable overall CER while consistently improving B-CER across two- and three-speaker mixtures.

4.3. Standardized Mixtures (AMI)

• Real-World Validation: On the AMI corpus (IHM-mix condition), CALM reduced B-WER from 34.7 to 22.1 with a biasing list of 1000.
• Observations: While overall WER saw a slight increase (attributed to insertion errors in short utterances and fillers common in conversational data), the significant reduction in B-WER validated the model's ability to recognize biasing terms in standardized, real-world mixtures.

5. Significance and Claims

The paper claims that CALM represents a significant step forward by moving beyond the limitations of prior target-speaker and contextual biasing approaches, which have historically been evaluated in isolation or under non-overlapping conditions.

• Joint Modeling: The primary significance lies in the demonstration that jointly leveraging linguistic and acoustic adaptation is more effective than addressing them separately. The speaker embeddings directly modulate the dynamic vocabulary layer, aligning acoustic and linguistic information.
• Robustness: The framework proves effective not only in simulated environments but also in standardized, real-world conversational datasets (AMI), handling challenges such as fillers and speaker variability.
• Scalability: The results confirm that the integration of dynamic vocabulary expansion with speaker-aware adaptation enables scalable and personalized multi-speaker ASR, effectively addressing both overlap-induced acoustic errors and unseen vocabulary challenges.

The authors conclude that this joint acoustic-linguistic integration advances the field beyond the constraints of previous methods, providing a robust solution for personalized multi-speaker ASR in real-world applications like meeting minutes and group discussions.