JaiTTS: A Thai Voice Cloning Model

1. Problem Statement

The paper addresses three critical gaps in current Text-to-Speech (TTS) technology, specifically regarding the Thai language:

• English-Centric Bias: Most state-of-the-art open-source and commercial TTS models are heavily optimized for English. While multilingual models (e.g., Qwen3-TTS) exist, they often suffer from pronunciation and prosodic errors in Thai due to the scarcity of Thai data in their training corpora.
• Limitations of Existing Thai Models: Dedicated Thai systems (e.g., ThonburianTTS) are often limited in zero-shot speaker cloning capabilities and struggle with long-duration speech generation because they are trained on datasets dominated by short utterances.
• Complex Pre-processing Requirements: Traditional Thai TTS pipelines rely heavily on complex text normalization to handle numbers and frequent Thai-English code-switching (common in real-world usage). This adds deployment complexity and potential points of failure.
• Codec Limitations: Recent autoregressive models using discrete audio codecs (like X-codec2) lack verified ability to reconstruct Thai-specific phonetic contrasts, such as lexical tones and consonant clusters, as their training data reportedly excludes Thai.

2. Methodology

The authors present JaiTTS-v1.0, a state-of-the-art voice cloning model adapted from VoxCPM. The system is designed to be tokenizer-free, bypassing external speech codecs entirely.

Architecture Overview

The model operates via a hierarchical pipeline of four core components (illustrated in Figure 1 of the paper):

1. Text-Semantic Language Model (TSLM): A decoder-only Transformer (initialized from MiniCPM-4) that conditions on tokenized text and historical acoustic context. It generates a continuous semantic-prosodic representation, jointly capturing content and delivery style.
2. Finite Scalar Quantization (FSQ): This layer compresses the TSLM's continuous output into a "semi-discrete skeleton." It acts as a regularizer, stabilizing the planning signal while maintaining differentiability via a straight-through estimator.
3. Residual Acoustic Language Model (RALM): A decoder-only Transformer that refines the semi-discrete skeleton. It specializes in acoustic expressivity and speaker characteristics, adding details that the discrete bottleneck cannot represent.
4. Local Diffusion Transformer (LocDiT): A bidirectional Transformer that decodes the final continuous latent speech patches. It uses flow-matching denoising, conditioned on the combined signal from the FSQ skeleton and RALM residuals, as well as the previous audio patch to ensure cross-patch continuity.
5. Stop Predictor: A lightweight head that predicts the end-of-sequence to terminate generation.

Training Strategy

• Data: Trained on a Thai-centric corpus of ~10,000 hours, combining general conversational/formal speech with domain-specific content (Finance, Healthcare, Education, Law).
• Direct Processing: The model is trained to process raw text directly, handling Thai-English code-switching and numerals without explicit text normalization.
• Objective: The system is optimized end-to-end using a combined loss function of Flow Matching (for the diffusion process) and Binary Cross-Entropy (for the stop predictor).

3. Key Contributions

• First High-Performance Thai Voice Cloner: JaiTTS-v1.0 is the first model to achieve state-of-the-art (SOTA) performance in Thai voice cloning, surpassing human ground truth in short-duration tasks and matching it in long-duration tasks.
• Tokenizer-Free Architecture: By adapting VoxCPM, the model avoids the phonetic reconstruction limitations of discrete audio codecs, allowing for more accurate modeling of Thai tones and clusters.
• Robustness to Real-World Inputs: The model eliminates the need for complex text normalization pipelines, directly synthesizing speech from raw text containing mixed languages and numbers.
• Novel Benchmark: The authors introduce a new evaluation framework splitting tasks into Short-Duration (1–15s) and Long-Duration (16–30s) to better reflect real-world use cases and test stability over time.

4. Experimental Results

Objective Metrics

• Short-Duration Tasks: JaiTTS-v1.0 achieved a Character Error Rate (CER) of 1.94%, surpassing the human ground truth (1.98%) and outperforming all baselines (Qwen3-TTS-1.7B: 2.56%; ThonburianTTS: 6.26%).
• Long-Duration Tasks: The model maintained a CER of 2.55%, closely matching human performance (2.47%). In contrast, Qwen3-TTS-1.7B degraded to 3.64%, and ThonburianTTS failed to generate sensible output.
• Speaker Similarity (SIM): Achieved a competitive SIM of 0.62 for short tasks and 0.76 for long tasks.

Efficiency

• Real-Time Factor (RTF): JaiTTS-v1.0 achieved an RTF of 0.1136, meaning it generates speech nearly 9x faster than real-time. This is approximately 13x faster than the autoregressive Qwen3-TTS models (RTF > 1.5) and slightly faster than the non-autoregressive ThonburianTTS (0.1150).

Human Judgment Evaluation

• Comparison: Tested against commercial flagships ElevenLabs (eleven_v3) and MiniMax (speech-2.8-hd).
• Outcome: In 400 pairwise comparisons, JaiTTS-v1.0 won 283 times, tied 59 times, and lost only 58 times.
• Preference: Human judges preferred JaiTTS-v1.0 approximately 70% of the time, citing superior naturalness, intelligibility (especially with code-switching/numbers), and speaker similarity.

5. Significance

JaiTTS-v1.0 represents a major leap forward in low-resource language TTS. It demonstrates that tokenizer-free, autoregressive architectures can outperform both traditional multilingual models and dedicated small-language models when trained on high-quality, domain-diverse data.

Its ability to handle raw text with code-switching and numbers without normalization simplifies deployment pipelines significantly. Furthermore, its superior performance in long-duration generation addresses a critical failure point in existing models, making it viable for applications like audiobooks, long-form content creation, and complex dialogue systems. The model sets a new benchmark for Thai speech synthesis, proving that specialized, high-quality training data can overcome the limitations of generalist multilingual models.