⚡ electrical engineering

Efficient ASR Training with Conversations that Never Happened

This paper demonstrates that augmenting limited real conversational data with LLM-generated, TTS-synthesized dialogues significantly improves ASR performance for lower-resource languages, outperforming a model trained on substantially larger amounts of real speech data.

Original authors: Máté Gedeon, Péter Mihajlik

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Máté Gedeon, Péter Mihajlik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human conversation. The robot is great at listening to clear, single-person speeches (like a news anchor), but it gets confused when people interrupt each other, talk over one another, or switch topics quickly. This is especially hard if you don't have thousands of hours of real recordings of people chatting in your specific language or niche topic.

This paper presents a clever solution: teaching the robot using "fake" conversations that never actually happened.

Here is how they did it, broken down into simple steps:

1. The Scriptwriter (The LLM)

First, the researchers used powerful AI text generators (like GPT, Claude, and others) to write scripts for imaginary conversations.

The Analogy: Think of this as hiring a team of creative writers. Instead of just giving them a topic, they told the writers: "Create a scene where a 45-year-old female doctor talks to a 20-year-old student about their weekend."
The AI generated the dialogue, the characters' ages, genders, and jobs, making the conversations sound natural and varied.

2. The Voice Actors (The TTS)

Next, they needed to turn those text scripts into actual audio.

The Analogy: They used a "digital voice actor" system. They had a library of real human voice samples. When the script called for a "young male," the system picked the real voice sample that sounded most like a young male and read the AI's script.
This ensured the robot heard different voices, not just one robotic monotone.

3. The Director (The Simulation)

Finally, they had to make the audio sound like a real, messy conversation, not just people reading sentences one by one.

The Analogy: In real life, people pause, interrupt, and sometimes talk over each other. The researchers used a "director" to arrange the audio clips. They added realistic pauses and even overlapped some voices so the robot could learn how to handle the chaos of a real group chat.

The Big Experiment

The researchers tested this method using the Hungarian language. They wanted to see if training the robot with these "fake" conversations would help it understand real Hungarian conversations better than just training it on real data alone.

They tried five different "AI Writers" (LLMs) to see which one wrote the best scripts:

GPT
Claude
Gemini
Grok
Qwen

What They Found

Fake is Better than Nothing: Adding these AI-generated conversations to the training data made the robot significantly smarter at understanding real Hungarian chats.
Not All Writers Are Equal: Some AI writers produced better scripts than others. GPT and Claude were the top performers.
Mixing is Tricky: They thought mixing scripts from all five writers would be the best idea. Surprisingly, it wasn't. Mixing too many different styles actually made the robot slightly worse off, like mixing too many different spices and ruining the flavor. The best mix was just the top two writers (GPT and Claude).
The "Magic" Combination: The absolute best result came from combining the AI-generated conversations with a small amount of "simulated" conversations based on real recordings.

The Knockout Result

Here is the most impressive part:

They trained a robot using only 67 hours of real Hungarian conversation data.
They added 636 hours of these AI-generated "fake" conversations.
The Result: This small team (67 hours real + 636 hours fake) beat a "super-robot" that had been trained on 2,700 hours of real Hungarian speech but had never seen the specific type of conversation they were testing on.

The Takeaway

You don't always need to wait years to collect thousands of hours of real recordings. By using AI to write scripts, digital voices to read them, and smart editing to make them sound real, you can create a massive, high-quality training library very quickly. This is a practical way to teach speech recognition systems to understand difficult, real-world conversations, especially for languages or topics where data is scarce.

Technical Summary: Efficient ASR Training with Conversations that Never Happened

Problem Statement
Conversational Automatic Speech Recognition (ASR) for lower-resource languages and niche domains faces a critical bottleneck: the scarcity of domain-matched, multi-speaker training data. While deep acoustic modeling and self-supervised pretraining have advanced ASR significantly, these gains are heavily data-dependent. Existing corpora often fail to capture the full spectrum of speaker diversity, discourse phenomena, and topic variation found in real-world interactions. Traditional data augmentation methods, such as speed perturbation or masking, improve robustness but cannot introduce new lexical content, speaker roles, or complex conversational structures. Furthermore, while neural Text-to-Speech (TTS) allows for scalable speech generation, standard pipelines often lack control over speaker attributes and multi-party interaction dynamics, failing to simulate the temporal structure of natural dialogue (e.g., turn-taking, overlaps, and interruptions).

Methodology
The authors propose a unified, three-stage pipeline to generate synthetic multi-speaker conversational audio for ASR augmentation. This approach integrates Large Language Models (LLMs), TTS, and conversation simulation:

LLM-Based Scenario and Dialogue Generation: The pipeline prompts an LLM to generate a scenario consisting of a topic ( $t_k$ ) and participant metadata ( $M_k$ ), including age, gender, occupation, and conversational role. Conditioned on this metadata, the LLM generates a structured, turn-level dialogue where each utterance is assigned to a specific speaker index.
Metadata-Conditioned TTS Synthesis: Each generated utterance is synthesized using a neural TTS model (xTTS-v2). Crucially, the system maps the generated speaker metadata (age and gender) to the most suitable reference voice profile from a pre-existing bank of real speech segments. The selection algorithm chooses the reference with the matching gender and the closest age, ensuring speaker consistency and diversity.
Speaker-Aware Conversation Simulation: The synthesized utterances are assembled into a single conversational waveform. Unlike simple concatenation, this stage uses a speaker-aware simulation method to model temporal dynamics. It applies sampled inter-turn offsets to simulate pauses and overlaps based on whether the speaker changes or remains the same, thereby reproducing the interactional organization of spontaneous speech without requiring a dedicated conversational TTS model.

Key Contributions

Unified Framework: A pipeline combining LLM scenario generation, metadata-conditioned TTS, and speaker-aware simulation to create synthetic conversational data for ASR.
Metadata-Conditioned Voice Selection: A procedure that explicitly maps generated speaker attributes (age/gender) to reference TTS profiles, bridging the gap between text generation and acoustic realism.
Comparative Evaluation: A rigorous assessment of five contemporary LLM families (GPT, Claude Haiku, Gemini, Grok, and Qwen) under a shared augmentation protocol, evaluating single-generator performance, fixed-budget mixtures, and scale-up configurations.
Empirical Validation: Evidence demonstrating that LLM-driven, speaker-aware synthetic conversations improve Hungarian conversational ASR performance compared to non-augmented baselines and prior augmentation methods.

Results
The experiments were conducted on the Hungarian BEA-Dialogue benchmark using a FastConformer-Large training recipe.

Single-Generator Performance: Synthetic conversations consistently improved performance over training on real data alone. GPT-5.4 mini emerged as the strongest single generator, achieving a character error rate (cpCER) of 8.20 and word error rate (cpWER) of 17.75 at a scale of 500 conversations.
Generator Mixtures: Under a fixed budget of 500 conversations, mixing generators yielded only marginal or negative returns if weaker generators diluted the contribution of the strongest ones. The best fixed-budget mixture (GPT + Haiku) slightly outperformed the best single generator (17.56 cpWER vs. 17.75 cpWER).
Scale-Up: Removing the fixed budget constraint revealed that scaling the amount of synthetic data from high-performing generators yielded further gains. The best configuration combined GPT, Haiku, Qwen, and Grok, reaching 16.65 cpWER.
Combined Augmentation: The most effective system combined the best LLM-generated scale-up (4 generators) with simulated conversations derived from real utterances (BEA-Large). This hybrid approach achieved a cpWER of 15.40 and cpCER of 7.57.
Comparison to Large-Scale Baselines: The combined system, trained on only 67 hours of real conversation plus 636 hours of simulated data, outperformed a zero-shot model trained on 2,700 hours of Hungarian speech (which achieved 16.27 cpWER).

Significance and Claims
The paper claims that LLM-generated conversational data, when synthesized with TTS and structured via speaker-aware simulation, serves as a practical and efficient complement to real conversational corpora. The findings suggest that for low-resource languages, generating scenario-based dialogues with controlled speaker attributes can provide more relevant training context than large-scale generic acoustic pretraining. The authors emphasize that the choice of generator and the composition of the data are critical; simply increasing data volume or mixing arbitrary generators does not guarantee improvement. Instead, the most effective strategy involves scaling high-performing generators and combining them with simulations derived from real utterances to capture both linguistic diversity and natural acoustic properties. This approach offers a resource-efficient pathway to training robust conversational ASR systems where domain-matched labeled data is scarce.