Technical Summary: Speaker-Aware Simulation Improves Conversational Speech Recognition

Problem Statement

Automatic Speech Recognition (ASR) for conversational speech faces significant challenges due to the scarcity of large-scale, well-annotated multi-speaker dialogue data and the complex temporal dynamics of natural interactions, such as interruptions, simultaneous speech, and hesitation. While synthetic conversation simulation offers a data augmentation strategy by transforming single-speaker recordings into multi-speaker dialogues, prior work has been limited in two key areas:

1. Language Scope: Previous frameworks, such as Speaker-Aware Simulated Conversations (SASC), have primarily focused on English, leaving their applicability to lower-resource languages like Hungarian unverified.
2. Temporal Modeling: Existing simulation methods, including SASC, often rely on independence assumptions where the length of a speaker's next utterance does not influence the pause preceding or following it. This contradicts natural human interaction patterns where longer planned turns are often preceded by longer gaps, and short reactive responses follow with minimal delay.

Methodology

1. Speaker-Aware Simulated Conversations (SASC) Adaptation

The authors adapt the SASC framework for Hungarian conversational ASR. The core mechanism involves:

• Speaker-Specific Timing: Unlike previous methods using global distributions, SASC models turn-taking via a first-order Markov chain and learns per-speaker timing distributions. It distinguishes between same-speaker transitions ($X_n = X_{n-1}$) and different-speaker transitions ($X_n \neq X_{n-1}$).
• Pause Generation: Pauses (or overlaps, denoted by $\delta < 0$) are generated by sampling a speaker-specific base value ($\mu$) from a distribution estimated via Kernel Density Estimation (KDE) and adding a deviation term ($v$).
• Acoustic Environment: All speakers in a simulated dialogue are placed in a shared acoustic environment using Room Impulse Responses (RIRs) to simulate a common physical space.

2. C-SASC: Duration-Conditioned Extension

To address the limitation of independence assumptions, the authors propose C-SASC (Conditioned SASC). This variant introduces a dependency between utterance duration and pause behavior:

• Conditional Residuals: While the base pause value ($\mu$) remains speaker-specific and fixed per dialogue, the residual term ($r$) is explicitly conditioned on the duration of the upcoming utterance ($d$).
• Nadaraya-Watson Estimator: The conditional residual density $p(r|d)$ is modeled using a kernel conditional density estimator. This allows the pause deviation to vary systematically based on the length of the surrounding utterances, capturing local temporal dependencies observed in natural dialogue.
• Bandwidth Selection: The framework employs Silverman's rule of thumb for speaker-specific baseline pauses and Scott's rule for the conditional residual distributions to ensure adaptive bandwidth selection.

3. Data Generation and Statistics Extraction

• Corpora: The framework utilizes the BEA-Large corpus (Hungarian, single-speaker) for generating synthetic dialogues. Statistics for pause modeling are extracted from three sources: CallHome (English), BEA-Dialogue (Hungarian), and GRASS (Austrian German).
• Preprocessing: Due to inaccuracies in public annotations (e.g., lack of temporal precision in CallHome or touching timestamps in BEA-Dialogue), the authors re-annotated data using automatic Voice Activity Detection (VAD) and forced alignment with punctuation restoration to derive accurate utterance durations and inter-utterance gaps.
• Synthesis: Synthetic dialogues are constructed by filtering BEA-Large recordings (2–10 seconds) and iteratively determining speaker turns via the Markov chain. The generated data is combined with real conversational data (BEA-Dialogue) for ASR training.

Key Contributions

1. Cross-Lingual Application: This work represents the first application of the SASC framework beyond English, demonstrating its effectiveness on Hungarian conversational ASR.
2. C-SASC Framework: Introduction of an extended variant that incorporates utterance-duration-conditioned pause modeling, preserving the efficiency of the original SASC while better reflecting temporal dependencies.
3. Systematic Evaluation: A comprehensive investigation comparing statistics derived from three distinct corpora (CallHome, BEA-Dialogue, GRASS) to assess cross-corpus and cross-language transferability.
4. Scaling Analysis: An analysis of the effect of simulated dataset size (varying speaker pairings) on downstream ASR performance, identifying points of diminishing returns.
5. RIR Impact Assessment: An examination of Room Impulse Response (RIR) augmentation within speaker-aware simulations, identifying conditions where it may be detrimental.

Experimental Results

The authors evaluated SASC and C-SASC against various baselines (including naive concatenation and previous simulation methods) on the Hungarian BEA-Dialogue corpus using metrics such as Word Error Rate (WER), Character Error Rate (CER), and their permutation-corrected variants (cpWER, cpCER).

• Simulation vs. Baselines: Speaker-aware simulation consistently outperformed naive concatenation-based augmentation and models trained without simulated data. Even simple concatenation strategies showed improvement, highlighting the value of exposing models to conversational structure.
• SASC vs. C-SASC:
  • Modest Gains: The additional duration conditioning in C-SASC yielded modest but systematic gains, most notably in character-level error rates (cpCER).
  • Domain Dependency: The effectiveness of C-SASC depended heavily on the match between source conversational statistics and the target domain.
    • BEA-based (Hungarian): C-SASC consistently outperformed SASC across all metrics.
    • GRASS-based (Austrian German): Results were mixed, with C-SASC performing better on some metrics and SASC on others.
    • CallHome-based (English): C-SASC underperformed compared to SASC, particularly on the evaluation set. This was attributed to a significant mismatch in utterance duration distributions between the English source and the Hungarian target simulation material.
• RIR Augmentation: Contrary to expectations, adding RIR augmentation generally degraded performance in this specific setup, likely due to a mismatch between the simulated reverberation and the high-quality studio recordings used for evaluation.

Significance and Claims

The paper claims that speaker-aware conversational simulation is a robust method for improving Hungarian ASR, confirming its utility beyond English. The primary significance lies in demonstrating that:

1. Realistic Timing Matters: Incorporating corpus-derived conversational statistics (speaker individuality and turn-taking patterns) yields consistent gains over naive data augmentation.
2. Conditional Modeling has Limits: While C-SASC offers a more faithful representation of temporal dependencies, the downstream benefits for standard ASR objectives are modest. The authors suggest that more detailed temporal conditioning may primarily benefit fine-grained tasks (like character boundary alignment) rather than high-level lexical recognition.
3. Corpus Mismatch Sensitivity: The study highlights that conditional modeling increases sensitivity to corpus mismatch. Transferring statistics across languages or domains (e.g., English to Hungarian) without alignment can negate the benefits of advanced temporal modeling.

The authors conclude that while increasingly detailed temporal modeling offers diminishing returns for standard ASR, the proposed framework remains valuable for its interpretability, low computational overhead, and flexibility, serving as a foundation for future work in speaker diarization and overlap detection.