On the Role of Conversational Timing in Synthetic Training Data for ASR

Technical Summary: On the Role of Conversational Timing in Synthetic Training Data for ASR

Problem Statement
Synthetic multi-speaker conversations are widely used to train conversational Automatic Speech Recognition (ASR) systems when natural data is limited. Current simulation pipelines typically prioritize "realism," estimating timing statistics (pause duration, overlap frequency) from real corpora and generating synthetic data to reproduce these distributions. However, the authors argue that realism alone does not guarantee utility for downstream ASR performance. A distribution that faithfully matches a corpus may not be the most effective for minimizing recognition error. The central problem addressed is identifying which specific timing properties of simulated data actually contribute to better ASR performance, rather than simply imitating a target corpus.

Methodology
The paper proposes a framework treating conversational timing as a controllable variable for systematic analysis rather than a fixed statistic.

1. Exponential-Tilting Parameterization:

   • The authors define a base timing density ($p_0$) as a weighted mixture of kernel density estimates (KDE) from three corpora: BEA-Dialogue (Hungarian), CallHome (English), and GRASS (Austrian German).
   • They generate a parametric family of timing distributions using exponential tilting: $p_\theta(\delta) = p_0(\delta) \exp(\theta^\top T(\delta)) / Z(\theta)$.
   • The statistic vector $T(\delta)$ includes indicators for overlap, global delay, positive gaps, and negative overlaps. This creates a four-dimensional parameter space ($\theta \in \mathbb{R}^4$) that allows for smooth, controlled deviations from the base distribution.
   • The authors note that while the density is identifiable only through a compressed three-dimensional representation ($\phi$), they retain the four-dimensional overcomplete representation for experimental design to distinguish specific simulator interventions (e.g., overlap mass vs. gap-side modulation).

2. Optimization Strategy:

   • Latin Hypercube Sampling (LHS): Used to provide broad initial coverage of the parameter space.
   • Multi-Objective Bayesian Optimization (BO): Employed to adaptively refine the search. The objective is to minimize both concatenated-permutation Word Error Rate (cpWER) and Character Error Rate (cpCER) on a Hungarian dialogue benchmark.
   • The optimization uses Gaussian Process surrogates and Expected Hypervolume Improvement (EHVI) to select the next evaluation point, balancing exploitation and exploration without collapsing the two metrics into a single scalar.

3. Evaluation Protocol:

   • For each sampled $\theta$, simulated training conversations are generated, an ASR system (based on the NVIDIA NeMo FastConformer Large CTC model) is trained, and performance is evaluated on the BEA-Dialogue development and evaluation splits.
   • Intrinsic timing statistics (overlap rate, mean overlap, gap tail mass, delay variability) are computed for every generated dataset to correlate with ASR errors.

Key Contributions

• Controllable Timing Formulation: Introduction of an exponential-tilting family that yields a compact, interpretable set of timing distributions anchored in multi-corpus data, allowing for controlled deviations from corpus-like regimes.
• Structured Analysis Framework: Framing conversational simulation as an analysis problem to identify intrinsic timing properties associated with ASR behavior, rather than merely seeking a single "best" configuration.
• Measurement Protocol: Definition of a comprehensive set of metrics including overlap/gap statistics, tail mass, distributional distance (KL divergence), and tilt-vector components.
• Empirical Insights: A case study on the BEA-Dialogue benchmark linking simulation design choices to specific error patterns (cpWER/cpCER).

Results
The study evaluated 25 timing configurations (10 LHS, 15 BO). Key findings include:

• Timing Statistics vs. Raw Coordinates: Downstream ASR behavior is explained more directly by induced timing statistics than by the raw simulator coordinates ($\theta$) or the proximity to a specific corpus. Intrinsic timing features (e.g., overlap rate, gap tail mass) showed significantly higher predictive power ($R^2 \approx 0.53$) for cpWER than the raw $\theta$ parameters ($R^2 \approx 0.28$).
• The Overlap–Gap Trade-off:
  • Overlap: Higher overlap exposure (overlap rate and overlap tail mass) is associated with lower cpWER.
  • Gaps: Longer and more variable gaps (mean gap, gap tail mass, delay standard deviation) are associated with higher cpWER.
  • This trend holds for cpCER as well, though with weaker statistical support.
• Optimization Efficacy: Bayesian optimization yielded modest aggregate improvements on the evaluation set (median cpWER decreased from 17.72% to 17.61%). However, the development set did not show consistent improvement, highlighting that the search process is better viewed as an experimental design mechanism than a method for finding a single optimal setting.
• Corpus Similarity: High-performing settings were not necessarily those closest to the reference corpus embeddings in parameter space. While extreme deviations from real-corpus distributions led to higher errors, the best settings were found within the same broad family of distributions, simply shifting the balance toward more overlap and fewer long gaps.
• Parameter Redundancy: Principal Component Analysis (PCA) revealed that while the four-dimensional $\theta$ space has active degrees of freedom, the observable timing effects largely collapse onto a dominant overlap–gap axis.

Significance and Claims
The paper claims that realistic simulation should be complemented by task-relevant diagnostics of overlap, gap, and timing-variability profiles. The primary value of the proposed framework is analytical: it reveals that the utility of synthetic data is driven by specific timing regimes (specifically, increased overlap exposure and reduced long-gap dominance) rather than strict corpus fidelity.

The authors modestly conclude that while Bayesian optimization can find slightly better configurations than corpus-derived baselines, the improvement is marginal. The true contribution is the demonstration that conversational timing should be treated as a controllable training variable. By monitoring the induced statistics (overlap rate, gap tail mass) rather than just the simulator coordinates, practitioners can generate training data that is better aligned with the specific needs of the ASR task, even if the resulting distribution is not a perfect replica of a specific real-world corpus.