Technical Summary: Leveraging Synthetic Speech for LLM-Based ASR Systems

Problem Statement

In regulated domains such as banking and healthcare, the collection and retention of real speech data are constrained by privacy regulations (e.g., GDPR) and the sensitivity of biometric information. While modern Text-to-Speech (TTS) systems offer a potential alternative for training Automatic Speech Recognition (ASR) models without exposing real customer recordings, a persistent distributional gap exists between synthetic and real data. Current LLM-based ASR models can distinguish between real and synthetic signals with high accuracy, limiting the efficacy of synthetic data as a direct substitute for real recordings. Prior work has largely treated this gap as a "black box," engineering around it via data filtering or model merging without understanding the internal mechanisms of the discrimination.

Methodology

The authors investigate this problem using the SLAM-ASR framework, which couples a frozen WavLM-Large encoder with a frozen Llama-3.2-3B-Instruct backbone via a trainable projector. The study employs a two-stage training protocol: (1) projector pre-training on mixed-domain data, and (2) domain adaptation using LoRA adapters on banking-specific data.

The research is structured around four key inquiries:

1. Localization of Discrimination: Where within the LLM backbone is the real/synthetic distinction encoded?
2. Filter Efficacy: Do signal-level perturbations that disrupt this distinction improve downstream ASR performance?
3. Data Substitution vs. Augmentation: What proportions of synthetic data can replace or augment real data?
4. Architectural Modification: Can layer-wise weighted pooling (LWP) extract more value from imperfect synthetic data?

Key Experimental Components

• Dataset: DefinedAI corpus of English customer-agent calls. Synthetic counterparts were generated using Qwen3-TTS (VoiceDesign variant) conditioned on persona attributes, totaling ~51 hours.
• Augmentation: Synthetic audio was convolved with Room Impulse Responses (RIRs) from the BUT Speech@FIT Reverb Database to simulate real-world acoustic irregularities.
• Perturbations: Six signal-level filters were tested (Add Noise, Band Pass, High Pass, Low Pass, Pitch Shift, Time Stretch).
• Interpretability Analysis: The authors probed the 28 layers of the Llama backbone using overlap metrics (Silhouette scores, Wasserstein-1, PCA-2D KDE overlap) to quantify the separation between real and synthetic representations.
• Layer-wise Weighted Pooling (LWP): An architectural modification where a learnable softmax-weighted combination of all LLM hidden states is used for the language modeling head, rather than relying solely on the final layer.

Key Findings

1. Localization of the Synthetic/Real Gap

The analysis reveals that the discrimination between real and synthetic speech is concentrated in the early-to-middle layers (0 to 14) of the LLM.

• Layer Behavior: Discriminative signals are strong in early layers but collapse by layer 28, where the model converges on semantic content regardless of the audio source.
• Perturbation Impact: Temporal and prosodic perturbations (specifically Time Stretch and Pitch Shift) are most effective at disrupting the separation in early layers. High Pass filtering alone was found to be the least effective.

2. The Role of RIR Augmentation

Convolution with RIRs significantly narrows the performance gap, but not by improving perceptual quality.

• Quality Metrics: RIR convolution actually degrades naturalness (UTMOS drops from 4.36 to 1.34) and only marginally improves PESQ.
• Mechanism: The gain comes from reproducing the acoustic irregularities (reverberation, channel variability) of real telephone recordings, thereby masking the "clean" signature of raw TTS output.

3. Data Substitution and Augmentation Results

• Substitution: In a fixed-budget setting, a mix of 10% to 30% real data combined with RIR-augmented synthetic data matches or slightly exceeds the performance of a 100% real-data baseline (8.68% WER). Replacing more than 30% of real data leads to performance degradation.
• Augmentation: Adding RIR-augmented synthetic data to a full real corpus consistently improves performance, with the best result (8.01% WER) achieved using 100% real + 100% synthetic (RIR-augmented). Raw synthetic data shows diminishing returns.

4. Interpretability vs. Downstream Gains

The study highlights a crucial distinction: representation-level separability does not directly predict downstream ASR gains.

• While early layers (0–14) encode the domain gap, the final layer (28) is most critical for decoding accuracy.
• Filters that disrupt early-layer separation (like Time Stretch) do not always yield ASR improvements if they introduce mismatches that outweigh the masking benefit.

5. Layer-wise Weighted Pooling (LWP)

Implementing LWP allows the model to learn which layers to prioritize.

• Effectiveness: LWP is most beneficial in low-resource substitution settings.
• Performance: A system using 25% real data (13.6 hours) combined with RIR-augmented synthetic data and LWP achieves 8.70% WER, nearly matching the 100% real baseline (8.68%).
• Mechanism: The learned weights concentrate heavily on the final layer (Layer 28) for speech tokens, indicating that while early layers encode the domain gap, the final layer encodes the linguistically rich information necessary for transcription. The LWP mechanism learns domain-agnostic preferences once the acoustic gap is narrowed by RIRs.

Significance and Claims

The paper claims to provide one of the first systematic answers regarding where inside a speech LLM the synthetic/real gap is encoded and how to leverage this for training.

• Practical Recipe: The authors propose a training strategy combining RIR augmentation with interpretability-guided filters (specifically Pitch Shift and High Pass + Pitch Shift) and Layer-wise Weighted Pooling.
• Efficiency: This approach allows an ASR system to match a fully real-data baseline using only 25% of the real speech and surpass it at higher proportions of real data.
• Theoretical Insight: The work demonstrates that the "domain gap" and "semantic utility" are encoded in different parts of the network. Effective training requires disrupting the gap in early layers (via RIRs and perturbations) while preserving the semantic refinement in the final layers (via LWP).

The authors conclude that while synthetic data cannot fully replace real data without penalty, it can significantly reduce the data burden in regulated domains when the acoustic distribution is aligned via RIRs and the model architecture is adapted to handle the specific characteristics of the mixed data. Future work is suggested to scale these findings to larger models, multilingual settings, and reinforcement learning approaches.