💬 NLP

LLM-to-Speech: A Synthetic Data Pipeline for Training Dialectal Text-to-Speech Models

This paper introduces NileTTS, the first publicly available Egyptian Arabic text-to-speech dataset and a reproducible synthetic data pipeline, to address the scarcity of resources for this widely spoken dialect by generating 38 hours of diverse speech via large language models and fine-tuning the XTTS v2 model.

Original authors: Ahmed Khaled Khamis, Hesham Ali

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmed Khaled Khamis, Hesham Ali

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to tell stories, but not in the formal, stiff language of textbooks (like Modern Standard Arabic), but in the lively, slang-filled, everyday language of Cairo, Egypt. This is the challenge the authors of this paper tackled.

Here is the story of NileTTS, explained simply:

The Problem: The "Missing Voice"

Think of the world of AI voice assistants (like Siri or Alexa) as a giant library. For languages like English or Spanish, the library is overflowing with books. But for Arabic, the library is mostly filled with books written in "Modern Standard Arabic"—the formal language used in news and schools.

However, over 100 million people speak Egyptian Arabic in their daily lives. It's the most understood dialect in the Arab world, kind of like how American English is understood globally. Yet, in our AI library, the "Egyptian Arabic" section is almost empty. Existing tools either don't understand the local slang or sound like a robot reading a dictionary.

The Solution: Building a "Virtual Studio"

Instead of hiring 50 actors to record hours of audio (which is expensive and slow), the researchers built a digital factory to create the data they needed. They called their project NileTTS.

Here is how their "factory" works, step-by-step:

The Scriptwriter (The LLM):
Imagine a super-smart AI writer (a Large Language Model) that knows Egyptian slang perfectly. The researchers asked this AI to write scripts about three things: Medical advice, Sales pitches, and Casual chatting. The AI wrote these scripts entirely in authentic Egyptian dialect, avoiding formal language.
The Actors (The Audio Synthesizer):
Next, they took those scripts and fed them into a tool called NotebookLM. Think of this as a virtual recording studio with two "ghost actors"—one male and one female. These actors read the scripts and have a natural, podcast-style conversation. Crucially, they sound like real Egyptians, complete with the right accent and rhythm.
The Transcriber and Editor (Whisper & Diarization):
Now they had hours of audio, but no text to match it with (because the AI generated the audio from the text, but they needed to verify it). They used a tool called Whisper to listen to the audio and type it out automatically.
- The Twist: Since the audio had two people talking, they needed to know who said what. They used a "voice fingerprint" tool (Speaker Diarization) to tag every sentence: "This part was said by the Male Ghost," and "This part was said by the Female Ghost."
The Quality Control:
Humans stepped in to double-check the work, making sure the AI didn't make any silly mistakes with the slang or mix up the voices.

The Result: A New "Voice" for AI

Once they had this massive collection of 38 hours of "fake but perfect" Egyptian conversations, they used it to teach a powerful existing AI model (called XTTS v2) how to speak Egyptian.

Think of XTTS v2 as a talented actor who already knows how to speak 16 languages, but only knows the formal version of Arabic. The researchers gave them this new "NileTTS" script and said, "Here, practice this dialect."

The outcome was impressive:

Before: The AI sounded a bit robotic and misunderstood Egyptian slang (like a foreigner trying to speak with a dictionary).
After: The AI became much clearer, understood the local words, and sounded much more like a real person from Cairo.

Why This Matters

This paper is a game-changer because it proves you don't need to spend thousands of dollars recording real humans to build a great voice AI for a specific dialect. You can build a reproducible pipeline (a recipe) that uses other AIs to generate the data for you.

The Big Takeaway:
Just like you can teach a child a new accent by listening to a podcast, the researchers taught a robot a new accent by feeding it a synthetic podcast. They have now released their "recipe," their "scripts," and the "trained robot" to the public, so anyone can build better Egyptian voice assistants in the future.

In short: They built a virtual voice factory to give the world's most spoken Arabic dialect the high-quality voice it deserves.

1. Problem Statement

Despite significant advancements in neural Text-to-Speech (TTS) for high-resource languages, Arabic dialects remain severely under-resourced. While Modern Standard Arabic (MSA) and Gulf dialects have received attention, Egyptian Arabic—the most widely understood dialect with over 100 million native speakers—is critically lacking in high-quality TTS resources. Existing datasets are either too small, limited to a single speaker, restricted in domain coverage, or not publicly available. Consequently, Egyptian Arabic speakers lack access to state-of-the-art voice assistants and audiobook tools.

2. Methodology

The authors propose NileTTS, a novel solution that combines a large-scale synthetic dataset with a fine-tuned TTS model. The core innovation is a fully automated, reproducible synthetic data generation pipeline that eliminates the need for costly manual recording.

A. Synthetic Data Generation Pipeline

The pipeline consists of four sequential stages:

Content Generation (LLMs): Large Language Models (Gemini and Claude) are prompted to generate authentic Egyptian Arabic reports across three diverse domains: Medical, Sales/Customer Service, and General Conversations. The prompts explicitly instruct the models to avoid MSA and use natural, colloquial dialect.
Audio Synthesis: The generated text is converted into speech using NotebookLM, a neural audio synthesis tool. This tool generates podcast-style conversations between two distinct virtual hosts (one male, one female) with high-quality, natural-sounding Egyptian Arabic pronunciation.
Transcription & Segmentation: The generated audio is processed using OpenAI's Whisper Large model for automatic transcription and word-level timestamping. The continuous audio is segmented into utterance-level chunks (discarding segments <1s or silence) to create (audio, text) pairs.
Speaker Diarization: Since the source audio contains two speakers, ECAPA-TDNN embeddings are extracted from the audio segments. Using K-Means clustering ( $k=2$ ), the system identifies two centroids representing the male and female voices. Each segment is assigned to the nearest centroid to ensure consistent speaker labeling.
Quality Control: A manual review process verifies transcription accuracy (specifically for dialectal vocabulary), speaker consistency, and audio quality (checking for artifacts), correcting or removing erroneous segments.

B. Dataset Statistics (NileTTS)

Total Duration: 38.1 hours of transcribed speech.
Utterances: 9,521 total (8,571 for training, 950 for evaluation).
Speakers: 2 (1 Male, 1 Female) with balanced speaking time.
Domains: Sales (21.0 hrs), General (11.2 hrs), Medical (5.9 hrs).
Format: WAV files paired with metadata (transcription and speaker ID) compatible with XTTS v2.

C. Model Fine-Tuning

The authors fine-tuned XTTS v2, a state-of-the-art multilingual TTS model with zero-shot voice cloning capabilities.

Architecture: The GPT-style autoregressive component of XTTS v2 was fine-tuned, while the DVAE (audio tokenizer) was kept frozen.
Training: 30 epochs were planned, but training was stopped at 13 epochs (55,719 steps) as metrics stabilized earlier. The best checkpoint was selected at Step 34,289 (Epoch 8).
Hyperparameters: AdamW optimizer, learning rate of $5e^{-6}$ , effective batch size of 16.

3. Key Contributions

NileTTS Dataset: The first publicly available, large-scale (38 hours) Egyptian Arabic TTS dataset covering multiple domains and two speakers.
Synthetic Pipeline: A reproducible, end-to-end pipeline for generating dialectal TTS data using LLMs and neural audio synthesis, offering a scalable solution for other low-resource dialects.
Open-Source Model: A fine-tuned XTTS v2 model specifically adapted for Egyptian Arabic, released to serve as a baseline for future research.

4. Results and Evaluation

The fine-tuned NileTTS model was evaluated against the baseline XTTS v2 (trained on MSA and Gulf dialects) using a held-out evaluation set.

Metric	Baseline XTTS v2	NileTTS (Fine-tuned)	Improvement
Word Error Rate (WER)	26.8%	18.8%	29.9% relative reduction
Character Error Rate (CER)	8.1%	4.1%	49.4% relative reduction
Speaker Similarity	0.713	0.755	+5.9% increase

Intelligibility: The significant drop in WER and CER indicates that the model produces much more accurate and intelligible Egyptian Arabic speech.
Voice Cloning: The increase in speaker similarity demonstrates better preservation of the target voice characteristics.
Qualitative: Manual listening confirmed natural prosody, accurate dialectal pronunciation, and consistent speaker identity.

5. Significance and Limitations

Significance:

Bridging the Gap: This work directly addresses the resource scarcity for the most widely spoken Arabic dialect, enabling better voice technology for millions of users.
Scalability: The synthetic pipeline proves that high-quality dialectal datasets can be created without expensive human recording sessions, making it a viable strategy for other under-resourced languages.
Performance: The results demonstrate that dialect-specific fine-tuning yields substantial gains even when the base model already supports the broader language family.

Limitations:

Speaker Diversity: The dataset currently contains only two speakers, which may limit generalization to diverse voice characteristics.
Synthetic Nature: The data is generated by AI, not recorded from humans. While effective, it may lack the subtle nuances of natural human speech.
Domain Coverage: Specialized domains like news broadcasting or poetry are not included.
Evaluation Bias: The use of Whisper for both dataset transcription and evaluation metrics (WER/CER) may introduce self-consistency bias. Future work should include formal human evaluation (MOS) and alternative ASR models.

In conclusion, NileTTS establishes a new standard for Egyptian Arabic TTS and provides a blueprint for leveraging generative AI to democratize speech technology for low-resource dialects.