💬 NLP

Elderly-Contextual Data Augmentation via Speech Synthesis for Elderly ASR

This paper proposes a data augmentation pipeline that combines LLM-based transcript paraphrasing with TTS synthesis using elderly reference speakers to generate synthetic elderly-contextual data, which significantly improves the performance of the Whisper ASR model on low-resource elderly speech datasets.

Original authors: Minsik Lee, Seoi Hong, Chongmin Lee, Sieun Choi, Jian Kim, Jua Han, Jihie Kim

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Minsik Lee, Seoi Hong, Chongmin Lee, Sieun Choi, Jian Kim, Jua Han, Jihie Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a smart robot how to understand the world. You give it a massive library of books (data) to study. But here's the problem: the library is full of stories written by young, energetic people speaking quickly and clearly. It has very few stories from elderly people, whose voices might be a bit shaky, slower, or use different words.

Because the robot hasn't practiced enough with these specific voices, it gets confused and makes mistakes when an older person speaks to it. This is the core problem the paper tackles: how to teach a speech-recognition robot to understand elderly people when there aren't enough real recordings of them to study.

Here is how the authors solved it, using a "two-step magic trick":

1. The Problem: A Missing Library

The authors point out that most speech datasets are like a library missing the "Senior Section." Real recordings of people over 70 are hard to find because it's difficult to get permission to record them, and many existing recordings are just people reading scripts (like a news anchor) rather than chatting naturally. Without enough practice data, the robot (specifically a model called Whisper) stumbles over elderly speech.

2. The Solution: A "Ghost Writer" and a "Voice Actor"

Instead of waiting for more real recordings, the team built a pipeline to create new practice data using two tools:

Step A: The "Ghost Writer" (Large Language Model)
First, they take an existing sentence and ask a super-smart AI writer (an LLM) to rewrite it. But they don't just ask for a synonym swap. They give the AI a specific instruction: "Rewrite this sentence as if an elderly person were saying it."
- The Analogy: Imagine you have a sentence like "The meeting is at 3 PM." The AI rewrites it to sound like a grandparent might say it, perhaps adding a bit more context or changing the phrasing to match how older generations often speak. This creates a "script" that feels authentic to an elderly speaker.
Step B: The "Voice Actor" (Text-to-Speech)
Once the AI has written these "elderly-style" scripts, they feed them into a Text-to-Speech (TTS) system. However, they don't use a generic robot voice. They use a special "voice bank" containing recordings of real elderly people.
- The Analogy: Think of this as hiring a voice actor who sounds exactly like a 75-year-old to read the new script. The result is a brand-new audio file that sounds like a real elderly person speaking, even though the words were generated by a computer.

3. The Result: A Better Robot

The team mixed these newly created "fake" but realistic audio clips with the few "real" clips they had. They then retrained the robot (Whisper) on this massive, mixed pile of data.

What happened?
The robot got significantly better at understanding elderly speech.

The Score: On their tests, the robot made 58.2% fewer mistakes compared to when it was trained without this special trick.
The Secret Sauce: They found that the more "fake" elderly data they added (up to doubling the size of their training set), the better the robot got. They also discovered that using a mix of male and female elderly voices for the "Voice Actors" worked better than using just one gender.

4. Why It Matters (According to the Paper)

The paper shows that you don't need to wait for the world to magically produce more recordings of elderly people. You can use AI to simulate the missing data. By combining a smart writer (to change the words) and a smart voice actor (to change the sound), they filled the gaps in the robot's library.

In short: They taught the robot to understand the elderly by creating a "practice gym" full of synthetic elderly voices, allowing the robot to learn the unique rhythm and style of aging speech without needing thousands of new real-world recordings.

Note: The paper strictly focuses on improving the accuracy of speech-to-text software for people over 70. It does not claim this technology is currently being used in hospitals, for medical diagnosis, or in specific clinical treatments.

1. Problem Statement

Automatic Speech Recognition (ASR) systems suffer significant performance degradation when applied to elderly speakers (Elderly ASR or EASR). This gap is driven by two primary factors:

Data Scarcity: Public corpora (e.g., Common Voice) contain very few speakers in the "late elderly" category (aged 70+). Existing datasets are often dominated by scripted or read speech, failing to capture natural conversational patterns.
Distinct Acoustic/Linguistic Characteristics: Elderly speech exhibits unique physiological and linguistic traits, including slower speaking rates, increased pauses, unstable articulation, hoarseness, and age-related disorders (e.g., dysarthria). Additionally, there are population-level shifts in lexical diversity and discourse organization that standard ASR models fail to capture.

Traditional data augmentation techniques (like speed perturbation or SpecAugment) address acoustic variability but do not resolve the contextual and lexical mismatch between general-domain training data and the specific linguistic patterns of the elderly.

2. Methodology

The authors propose a novel data augmentation pipeline that combines Large Language Models (LLMs) for text generation with Text-to-Speech (TTS) synthesis to create synthetic elderly speech data.

A. Pipeline Overview

The framework operates in three sequential stages:

Elderly-Contextual Paraphrasing (LLM):
- Given an original transcript $t_i$ , an LLM generates a paraphrased version $t'_i$ specifically tailored to elderly contexts.
- Prompt Engineering: The LLM is instructed to:
  - Rewrite the sentence to fit a CSV format (for structured generation).
  - Adhere to length constraints (3–20 words).
  - Crucially: Adjust tone and content to reflect speech frequently used by elderly people (e.g., changing "child" to "grandchild," adding retrospective or didactic tones).
Speech Synthesis (TTS):
- The generated elderly-contextual transcript $t'_i$ is converted into synthetic audio $a'_i$ using a TTS model (OpenVoice2).
- Speaker Conditioning: The synthesis is conditioned on a curated pool of elderly reference speakers ( $S_{ref}$ ). The system balances speaker assignment to ensure diverse acoustic characteristics (gender and regional accents) are represented.
Model Fine-Tuning:
- The synthetic audio-text pairs are merged with the original dataset to form a training set.
- This combined dataset is used to fine-tune the Whisper ASR model without modifying its architecture.

B. Experimental Setup

Datasets:
- English: Common Voice 18.0 (CV18), filtered for speakers aged 70+ (12 hours of data).
- Korean: VOTE400, a dataset of speakers aged 75+ collected from senior welfare institutes (10 hours of data).
Models: Whisper (Small, Medium, Large-v2, Large-v3) as the backbone; GPT-5, GPT-4o, and Gemini 3 Flash for text generation.
Metrics: Word Error Rate (WER) for English; Character Error Rate (CER) for Korean.

3. Key Contributions

Novel Framework: Proposes the first EASR augmentation pipeline that integrates LLM-based elderly-contextual text generation with TTS, addressing both acoustic and linguistic gaps simultaneously.
Performance Gains: Demonstrates that this method significantly outperforms conventional augmentation baselines (speed perturbation, SpecAugment) and TTS-only approaches.
Comprehensive Analysis: Provides empirical guidance on critical generation factors:
- Augmentation Ratio: Performance improves monotonically with higher ratios, with 100% augmentation yielding the best results.
- Speaker Composition: A balanced mix of male and female reference speakers (4F+4M) yields superior results compared to gender-skewed pools.
- LLM Selection: Larger/more advanced LLMs (GPT-5) generate higher-quality paraphrases, leading to better ASR performance.

4. Experimental Results

Main Results:
- On the CV18 (English) dataset, the proposed method reduced WER from 4.1% (Baseline) to 2.2%, a 46.3% relative reduction.
- On the VOTE400 (Korean) dataset, WER dropped from 11.6% to 5.2%, a 55.2% relative reduction.
- Compared to the Whisper baseline, the method achieved up to a 58.2% relative reduction in error rates.
Comparison with Baselines:
- The proposed method (LLM+TTS) significantly outperformed TTS-only augmentation and standard signal-level augmentations (Speed Perturbation, SpecAugment).
- Combining the proposed method with SpecAugment yielded the absolute best performance (2.1% WER on CV18).
Model Size Efficiency: The augmentation was particularly effective for smaller models. Whisper-Small augmented with this method surpassed the baseline performance of much larger models (e.g., Whisper-Large-v2) on the English dataset.
Qualitative Analysis:
- The method successfully corrected phonetic ambiguities (e.g., distinguishing "grandchild" from "and child") and domain-specific vocabulary errors.
- The LLM successfully introduced diverse elderly-contextual variations (e.g., shifting from neutral statements to didactic or worry-based tones), though some outputs occasionally relied on stereotypes.

5. Significance and Conclusion

This work addresses a critical bottleneck in human-computer interaction for the aging population: the lack of representative training data for EASR. By leveraging LLMs to simulate the linguistic and contextual nuances of elderly speech (not just the acoustic voice), the authors demonstrate that synthetic data can effectively bridge the performance gap.

The study establishes that:

Context matters: Simply synthesizing speech is insufficient; the text must reflect the specific discourse patterns of the elderly.
Scalability: The approach is highly effective across different model sizes and languages (English and Korean).
Practicality: It offers a viable path to improve ASR reliability for the elderly without requiring massive new data collection efforts, which are often hindered by privacy and logistical constraints.

Future work suggested by the authors includes expanding to more languages, refining prompt engineering to reduce stereotypical outputs, and exploring tighter integration between generative models and ASR training loops.