Technical Summary: Qwen3-TTS

Problem Statement

The field of neural text-to-speech (TTS) aims to achieve stable, controllable, and human-like speech synthesis, a capability viewed as critical for the path to Artificial General Intelligence (AGI). While modern models trained on large-scale datasets can generate high-quality speech from short reference audio, challenges remain in balancing stability with naturalness, achieving ultra-low latency for real-time streaming, and maintaining robust performance across diverse languages and long-form contexts. Existing approaches often struggle with the trade-off between semantic fidelity and acoustic richness in tokenization, or face latency bottlenecks due to complex diffusion-based decoders and look-ahead requirements. Furthermore, integrating TTS seamlessly with Large Language Models (LLMs) to support fine-grained instruction following and voice cloning without overfitting to ASR metrics remains a significant hurdle.

Methodology

Architecture and Tokenization

Qwen3-TTS introduces a family of models built on a dual-track autoregressive architecture designed for real-time synthesis. The core innovation lies in the development of two distinct speech tokenizers to address different latency and quality requirements:

1. Qwen-TTS-Tokenizer-25Hz: A single-codebook codec (25 Hz) emphasizing semantic content. It is built upon the Qwen2-Audio encoder with an added resampling and vector quantization (VQ) layer. It utilizes a block-wise Diffusion Transformer (DiT) with Flow Matching for waveform reconstruction. This tokenizer integrates semantic and acoustic cues, enabling seamless integration with Qwen-Audio and supporting streaming via a sliding-window block attention mechanism. However, its single-codebook design and look-ahead requirements limit its suitability for ultra-low-latency applications.
2. Qwen-TTS-Tokenizer-12Hz: A multi-codebook codec (12.5 Hz) featuring a 16-layer Residual Vector Quantization (RVQ) design. The first codebook captures semantic content, while subsequent layers model acoustic details. Inspired by the Mimi architecture, it employs a GAN-based training framework with a teacher-student setup (WavLM) for the semantic path. Crucially, it uses fully causal feature encoders and decoders, allowing for immediate first-packet emission without look-ahead. Waveform reconstruction is handled by a lightweight causal ConvNet, eliminating the need for speaker vector extraction or complex diffusion models.

Model Architecture

The Qwen3-TTS models leverage the Qwen3 Large Language Model (LM) family.

• Dual-Track Representation: Text and acoustic tokens are concatenated along the channel axis. Upon receiving a textual token, the model predicts corresponding acoustic tokens, which are then converted to waveforms by a Code2Wav module.
• Hierarchical Prediction (12Hz variant): The backbone predicts the zeroth codebook, while a Multi-Token Prediction (MTP) module generates all residual codebooks, capturing intricate acoustic details and enhancing vocal consistency.
• Speaker Control: A learnable speaker encoder is jointly trained with the backbone to maintain precise identity control. The system supports voice cloning via reference speech (speaker embedding) or text-speech pairs (in-context learning), as well as voice design via natural language descriptions.

Training Strategy

The training process involves pre-training and post-training phases, with all data formatted in ChatML:

1. Pre-training:
   • General Stage (S1): Trained on over 5 million hours of multilingual speech data to establish monotonic text-to-speech mapping.
   • High-Quality Stage (S2): Continual pre-training (CPT) on stratified high-quality data to reduce hallucinations and improve speech quality.
   • Long-Context Stage (S3): Token length increased from 8,192 to 32,768, with upsampling of long speech data to enhance context handling.
2. Post-training:
   • Direct Preference Optimization (DPO): Aligns outputs with human preferences using constructed preference pairs.
   • Rule-based Rewards & GSPO: Enhances capabilities and stability across tasks.
   • Lightweight Speaker Fine-tuning: Adapts the base model to specific voices while improving naturalness and controllability.
   • Instruction Following: A probabilistically activated thinking pattern is introduced during training to improve adherence to complex descriptions.

Key Contributions

1. Dual-Tokenizer Framework: The introduction of two specialized tokenizers (25Hz for high-fidelity semantic integration and 12Hz for ultra-low-latency streaming) allows the model to balance expressive power with real-time performance.
2. Ultra-Low Latency Streaming: The 12Hz variant achieves a first-packet latency as low as 97 ms (0.6B model) and 101 ms (1.7B model) by utilizing a causal, multi-codebook design that enables immediate waveform emission.
3. Advanced Controllability: The model supports 3-second voice cloning, predefined voice profiles, and fine-grained control via natural language descriptions for both voice creation (Voice Design) and editing (Target Speaker).
4. Robust Multilingual Performance: Trained on over 5 million hours of data across 10 languages, the model demonstrates speaker-consistent multilingual generation and superior cross-lingual voice transfer capabilities.
5. Long-Form Stability: The architecture resolves stability issues common in autoregressive models, enabling the seamless generation of over 10 minutes of fluent speech without the artifacts typical of chunk-based systems.

Experimental Results

Speech Tokenizer Evaluation

• 25Hz Tokenizer: Achieved competitive or superior Word Error Rate (WER) on ASR tasks (CommonVoice, Fleurs) compared to S3 Tokenizer series, particularly in Stage 1. Stage 2 showed a slight WER increase, attributed to the deliberate trade-off of adding acoustic richness for better TTS generation.
• 12Hz Tokenizer: Set a new state-of-the-art in speech reconstruction on the LibriSpeech test-clean set across all metrics (PESQ, STOI, UTMOS, SIM), outperforming models like Mimi, SpeechTokenizer, and X-Codec, while maintaining high encoding efficiency.

Speech Generation Evaluation

• Zero-Shot Cloning: On the Seed-TTS benchmark, Qwen3-TTS-12Hz-1.7B achieved a WER of 1.24 (English) and 0.77 (Chinese), outperforming baselines like CosyVoice 3, Seed-TTS, and MiniMax-Speech.
• Multilingual Generation: The model achieved the lowest WER in 6 out of 10 languages (including Chinese, English, Korean, Russian) and the highest speaker similarity scores across all 10 languages compared to MiniMax and ElevenLabs.
• Cross-Lingual Transfer: Demonstrated exceptional generalization, reducing error rates by ~66% in Chinese-to-Korean generation compared to CosyVoice 3.
• Controllable Generation: On InstructTTSEval, the 12Hz Voice Design variant outperformed open-source models and commercial systems (Hume, VoiceSculptor) in Description-Speech Consistency (DSD) and Response Precision (RP). In Target Speaker editing, it significantly outperformed GPT-4o-mini-tts.
• Long-Form Generation: The 25Hz variant achieved the lowest WER (1.533 for Chinese, 1.571 for English) on a 10-minute long-form test set, demonstrating superior stability compared to chunk-based systems like Higgs-Audio-v2 and VibeVoice.

Efficiency

• Streaming Latency: The 12Hz models demonstrated significantly lower first-packet latency (97–101 ms) compared to 25Hz models (138–150 ms) under single-concurrency conditions.
• Concurrency: The models maintained stable performance under varying concurrency levels (1, 3, 6), with the 12Hz variants showing lower steady-state costs per packet.

Significance

The paper claims that Qwen3-TTS represents a significant step toward stable, controllable, and human-like speech synthesis. By unifying diverse tasks—zero-shot cloning, cross-lingual transfer, and fine-grained instruction control—within a single autoregressive framework, it paves the way for next-generation omni-capable audio systems. The release of both the models and the tokenizers under the Apache 2.0 license is intended to facilitate community research and the development of more natural and accessible human-computer interfaces. The work highlights that balancing semantic and acoustic tokenization, combined with causal decoding and advanced training strategies, can overcome the latency and stability limitations of previous TTS architectures.