💬 NLP

When Synthetic Speech Is All You Have: Better Call GRPO

This paper demonstrates that Group Relative Policy Optimization (GRPO) significantly outperforms supervised fine-tuning in adapting large language model-based automatic speech recognition to regulated domains using only synthetic speech, achieving a 45% relative reduction in word error rate by improving behavioral aspects like stopping calibration and attention alignment rather than altering early-layer representations.

Original authors: Shashi Kumar, Yanis Labrak, Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello, Kadri Hacioğlu, Petr Motlicek, Andreas Stolcke

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shashi Kumar, Yanis Labrak, Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello, Kadri Hacioğlu, Petr Motlicek, Andreas Stolcke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart robot how to listen to phone calls in a bank. The problem? Real bank calls are like secret vaults; you can't just grab them to practice because of strict privacy laws. So, scientists decided to use "fake" voices made by computers (Text-to-Speech) to train the robot instead.

But here's the twist: the robot was getting confused. When trained with these fake voices using the standard method (called Supervised Fine-Tuning or SFT), the robot started hallucinating. It would listen to a short sentence and then just keep talking, inventing long, confident paragraphs of nonsense that never happened. It was like a student who, after hearing the teacher say "The capital is Paris," would confidently write a whole essay about French history that wasn't asked for.

The paper's main finding is that a new training trick called GRPO (Group Relative Policy Optimization) fixes this mess. Instead of just copying the fake voice word-for-word, GRPO acts like a strict coach who says, "Okay, try saying that sentence five different ways. I'll only give you a high score if the whole sentence makes sense and stops at the right time."

The Big Showdown: SFT vs. GRPO

When the researchers tested this on a banking task, the results were wild:

The Old Way (SFT): The robot made a huge mess, with a 36.71% error rate. It was mostly guilty of "insertion errors"—adding words that weren't there.
The New Way (GRPO): The robot cleaned up its act, dropping the error rate to 22.09%. That's a 40% relative improvement!
The Power Combo: If they first taught the robot with the old way and then polished it with GRPO, the error rate dropped even further to 20.21%.

The paper explicitly argues against the idea that you need to perfectly mimic the fake voice's sound quirks. The standard method tries to copy every little glitch in the fake voice, which actually makes the robot worse. GRPO ignores those tiny, weird glitches and focuses on the big picture: "Did you get the right words, and did you stop talking when the audio stopped?"

How Much Real Data Do You Actually Need?

You might think, "If fake voices are so tricky, maybe we need tons of real human recordings to fix it." The paper says: Not really.

They found that if you have a huge pool of fake voices, you only need a tiny bit of real human speech to make the magic work.

With 0 hours of real speech, the error rate was 22.09%.
Adding just 1 hour of real speech dropped the error to 20.32%.
Adding 5 hours got it down to 15.51%.
But here's the kicker: adding 54 hours of real speech only lowered it to 12.61%.

The paper suggests that the first 5 to 10 hours of real data give you about two-thirds of the total possible improvement. After that, collecting more real recordings gives you very little extra bang for your buck. It's like tuning a radio: you need a little bit of real signal to lock in the station, but once it's locked, turning the dial more doesn't make the music much clearer.

Why Does GRPO Work? (The "Why" Behind the Magic)

The researchers dug deep to see why GRPO is better. They found it's not because the robot learned a new "language" inside its brain. Instead, it changed its behavior.

Stopping on a Dime: The old method (SFT) kept talking even after the audio ended, like a singer who keeps hitting high notes after the song is over. GRPO learned to stop exactly when the audio stops. It became much better at knowing when to be quiet.
Listening, Not Guessing: When the robot used SFT, it would start "hallucinating" words because it wasn't sure what it heard, so it just guessed based on what it thought should be said. GRPO forced the robot to stick to the audio evidence. If the audio was fuzzy, GRPO made the robot pause rather than invent a story.
No Overhaul Needed: The paper measured the robot's internal "brain waves" (representations) and found that GRPO didn't rewrite the robot's early memories. It just tweaked the final decision-making layer. It's like taking a car that runs well but has a shaky driver; GRPO didn't rebuild the engine, it just taught the driver to brake better.

What Didn't Work?

The paper also tested some fancy ideas that turned out to be unnecessary:

Picking the "Best" Fake Voices: They tried to use complex math to pick the most diverse or interesting fake voices to train on. Surprisingly, just picking fake voices randomly worked just as well, or even better, once they had a decent amount of data (around 25 hours).
Complex Scoring: They tried giving the robot extra points for matching the exact length of the sentence or counting letters. The simple score—just counting how many words were wrong (WER)—was the best and simplest tool.

The Bottom Line

When you are stuck with only fake voices to train a speech robot, don't just copy them word-for-word. Use GRPO. It teaches the robot to listen to the whole sentence, stop when the sound stops, and ignore the weird glitches in the fake voice.

The authors are confident that this approach is the way to go for privacy-heavy jobs like banking. They showed that with just a little bit of real data (5–10 hours) and a lot of fake data, you can get a robot that listens almost as well as one trained on a mountain of real recordings, without ever breaking privacy laws. It's a win for privacy, a win for cost, and a win for robots that finally know when to shut up.

Technical Summary: When Synthetic Speech Is All You Have: Better Call GRPO

1. Problem Statement

The adoption of Large Language Model (LLM)-based Automatic Speech Recognition (ASR) in regulated domains, such as banking, is severely bottlenecked by data privacy constraints. Collecting real customer-agent telephone calls is costly and legally restricted due to regulations like GDPR and the EU AI Act, which classify such biometric and financial data as sensitive. Consequently, the community has turned to synthetic speech generated via Text-to-Speech (TTS) systems from available transcripts as a substitute.

However, a significant acoustic mismatch exists between synthetic speech and real recordings. Previous work attempting to close this gap has relied almost exclusively on Supervised Fine-Tuning (SFT). The authors hypothesize that SFT is suboptimal for this specific regime because its token-level cross-entropy objective forces the model to reproduce reference tokens one-by-one. This causes the model to absorb local synthetic artifacts (e.g., unnatural prosody, phonetic glitches, imperfect room impulse response simulations) into fluent but acoustically unsupported continuations, leading to high rates of insertion errors (hallucinations).

2. Methodology

The paper proposes shifting from SFT to Reinforcement Learning (RL), specifically Group Relative Policy Optimization (GRPO), to adapt LLM-based ASR models using synthetic speech.

Model Architecture

The experiments utilize a modular LLM-based ASR backbone:

Encoder: Frozen WavLM-Large.
Projector: Trainable linear layers with ReLU activation, downsampling acoustic sequences by a factor of 5.
LLM: Frozen Llama-3.2-1B-Instruct with trainable LoRA adapters ( $r=16, \alpha=32$ ).
Adaptation Scope: Only the projector and LoRA parameters are updated; the encoder and LLM backbone remain frozen to isolate the effects of the adaptation objective.

Training Paradigms

Supervised Fine-Tuning (SFT): Minimizes token-level cross-entropy loss against ground-truth transcripts.
Group Relative Policy Optimization (GRPO):
- Mechanism: For each input utterance, the policy samples a group of $G$ hypotheses (decoded transcripts).
- Reward: Hypotheses are scored using a reward function (default: $1 - \text{WER}$ ).
- Advantage Calculation: A group-relative advantage is computed by standardizing rewards within the group ( $A_i = \frac{r_i - \mu}{\sigma + \epsilon}$ ), eliminating the need for a learned value function (critic).
- Objective: The policy is updated to maximize the clipped surrogate objective, reinforcing hypotheses that outperform the group average while suppressing lower-reward ones. This optimizes a sequence-level objective rather than token-level likelihood.

Reward Functions

The study evaluates several reward functions: Word Error Rate (WER), Character Error Rate (CER), a length-consistency term (LEN), and various additive combinations.

Data Setup

Real Data: DefinedAI banking corpus (54 hours training, 6.55 hours test).
Synthetic Data: Transcripts from the banking corpus synthesized using Qwen3-TTS with VoiceDesign for persona matching. The audio is convolved with measured Room Impulse Responses (RIRs) to simulate channel acoustics.
Pre-training: Models are initialized from an LS 960 (LibriSpeech) SFT checkpoint.

3. Key Contributions and Findings

A. Superiority of GRPO over SFT on Synthetic Data

Performance Gap: When adapting solely on synthetic speech, SFT yields a WER of 36.71%, whereas GRPO reduces this to 22.09% (a 40% relative reduction).
Combined Approach: An SFT-then-GRPO pipeline achieves 20.21% WER, a 45% relative improvement over SFT alone.
Real Data: On abundant real data, GRPO provides marginal gains over SFT (10.27% $\to$ 9.49% WER), primarily driven by reduced insertions.

B. Data Efficiency and Scaling

Real Data Diminishing Returns: Mixing a small amount of real data with synthetic speech yields the highest returns. The first 5–10 hours of real data account for roughly two-thirds of the achievable WER reduction. Beyond 25 hours, returns diminish significantly.
Synthetic Data Saturation: Increasing synthetic data is not always beneficial. With a fixed 1-hour real budget, performance peaks at 25 hours of synthetic data (WER 16.77%) and degrades when using the full 54-hour pool (WER 20.32%), suggesting that excessive synthetic data can dilute the anchoring effect of limited real data.

C. Reward and Selection Strategies

Reward Function: Optimizing directly for WER is the most effective strategy. Adding length (LEN) or CER terms generally hurts WER performance or increases insertion errors.
Data Selection: For synthetic speech selection, random sampling outperforms complex diversity-based strategies (speaker, semantic, or acoustic embeddings) and GRPO-specific selectors (based on rollout statistics) when the synthetic pool is moderately large (25h). Selection strategies only show marginal benefits at very small budgets (1h).

D. Mechanism of Improvement: Behavioral vs. Representational

The paper traces the performance gain to behavioral changes rather than fundamental representation shifts:

Insertion Reduction: SFT on synthetic data suffers from "insertion tails," where the model generates hundreds of extra tokens after the speech ends. GRPO effectively eliminates this tail.
Stopping Calibration: GRPO significantly improves the model's ability to stop generating text when acoustic evidence ends. This is evidenced by a sharp drop in token probability after the final reference token and lower Expected Calibration Error (ECE).
Attention Anchoring: GRPO maintains lower attention entropy over audio tokens compared to SFT. While SFT's attention becomes diffuse (relying on language priors), GRPO keeps attention sharp and concentrated on stable audio evidence, preventing hallucinations.
Representation Preservation: Layer-wise CKA similarity analysis shows that GRPO preserves the early-to-middle layer representations of the base model. In contrast, SFT significantly alters these representations to fit synthetic artifacts. GRPO acts as a "lightweight policy correction" that fixes stopping behavior without rewriting the underlying acoustic representations.

4. Significance and Claims

The paper claims that when synthetic speech is the primary or only resource for domain adaptation in LLM-based ASR, Reinforcement Learning (specifically GRPO) should be preferred over Supervised Fine-Tuning.

The significance lies in demonstrating that:

Synthetic speech, often considered too acoustically mismatched for high-performance ASR, can be effectively utilized if the training objective is sequence-level (GRPO) rather than token-level (SFT).
The primary failure mode of SFT in this context is hallucinated insertions caused by the model's inability to distinguish between synthetic artifacts and real speech boundaries.
GRPO solves this by optimizing for the evaluation metric (WER) directly, learning to suppress hypotheses that contain long insertion chains, thereby making synthetic data a viable substitute for real data in privacy-constrained domains.

The authors conclude that a combination of a small amount of real data (5–10 hours), a large pool of synthetic data (25 hours), and a simple WER-based GRPO reward function offers the most practical and effective path forward for regulated domain adaptation.