OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models

1. Problem Statement

Current state-of-the-art (SOTA) zero-shot Text-to-Speech (TTS) systems face two primary limitations:

• Limited Language Coverage: Most models support only a few dozen languages, leaving hundreds of low-resource languages underserved.
• Architectural Trade-offs:
  • Autoregressive (AR) models offer high quality but suffer from slow inference speeds due to sequential decoding.
  • Non-Autoregressive (NAR) models offer fast parallel decoding but often struggle with intelligibility and prosody diversity.
  • Discrete NAR models (which use token-based generation) typically rely on complex two-stage cascaded pipelines (Text $\to$ Semantic Tokens $\to$ Acoustic Tokens). This approach introduces error propagation (errors in the semantic stage degrade audio quality) and information bottlenecks (low-bitrate semantic tokens lose fine-grained acoustic details).

The goal is to build a massive multilingual zero-shot TTS model that supports hundreds of languages while maintaining high intelligibility, naturalness, and inference speed, without relying on error-prone cascaded pipelines.

2. Methodology

OmniVoice introduces a streamlined, single-stage discrete non-autoregressive (NAR) architecture based on Diffusion Language Models (DLM).

A. Core Architecture

• Single-Stage Mapping: Unlike traditional two-stage systems, OmniVoice directly maps text tokens to multi-codebook acoustic tokens using a Bidirectional Transformer.
• Input/Output:
  • Input: A concatenated sequence of instruction tokens and transcript text ($Y$), plus an acoustic prompt matrix ($X$) containing a prefix (prompt) and a masked target segment.
  • Output: The model predicts the masked acoustic tokens across $C$ codebooks (using the Higgs-audio tokenizer for 8 codebooks).
• Loss Function: The model is trained to recover masked tokens using a discrete diffusion objective.

B. Key Technical Innovations

To overcome the limitations of single-stage discrete NAR models, the authors propose two critical innovations:

1. Full-Codebook Random Masking Strategy:

   • Problem: Previous methods used "per-layer" masking (masking only one codebook layer per sample), leading to inefficient convergence.
   • Solution: OmniVoice employs a fully stochastic masking strategy where every entry in the $T \times C$ token matrix is independently sampled for masking with a probability drawn from a uniform distribution.
   • Benefit: This results in an average of 50% of all tokens being used for loss calculation per step (vs. sparse subsets in per-layer masking), significantly accelerating training convergence and improving generative quality.

2. LLM Initialization for Intelligibility:

   • Problem: Single-stage discrete NAR models historically suffer from poor intelligibility compared to AR models.
   • Solution: The model backbone is initialized with weights from a pre-trained Autoregressive Large Language Model (LLM) (specifically Qwen3-0.6B).
   • Benefit: This transfers linguistic knowledge and pre-trained language priors to the TTS task. Despite the LLM being trained with causal masking and OmniVoice using bidirectional attention, the initialization proves highly effective, making OmniVoice the first NAR TTS model to successfully leverage LLM initialization for intelligibility gains.

C. Data Strategy & Scaling

• Dataset: A curated 581,000-hour multilingual dataset derived exclusively from open-source resources, covering 646 languages.
• Preprocessing: Includes speech restoration for degraded audio and rule-based filtering for invalid transcriptions.
• Resampling: To address data imbalance (high-resource vs. low-resource languages), a language-level resampling strategy is applied with a smoothing hyperparameter ($\beta=0.8$), ensuring the model sees sufficient low-resource data without completely sacrificing high-resource performance.
• Text Processing: Uses subword tokenizers from pre-trained LLMs, eliminating the need for Grapheme-to-Phoneme (G2P) conversion.

D. Controllability

OmniVoice supports multi-dimensional control:

• Prompt Denoising: Trained with synthetic noise/reverberation on prompts to generate clean speech from degraded inputs.
• Voice Design: Synthesis based on speaker attributes (gender, age, pitch) without audio prompts.
• Linguistic Control: Support for paralinguistics (e.g., laughter) and explicit phonetic overrides (e.g., Pinyin for Chinese polyphones) via hybrid text input.

3. Key Contributions

1. Architectural Breakthrough: A novel single-stage discrete NAR framework that bypasses the error propagation of two-stage pipelines, directly mapping text to multi-codebook acoustic tokens.
2. Training Efficiency & Quality: The Full-Codebook Random Masking strategy significantly improves training efficiency and output quality over existing masking schedules.
3. Intelligibility Solution: Demonstrates that LLM initialization is a viable and powerful method to resolve intelligibility issues in single-stage NAR TTS.
4. Unprecedented Scale: The first zero-shot TTS model to support 600+ languages (646 specifically), trained on a massive 581k-hour open-source corpus.
5. Open Source: The code and pre-trained models are publicly released.

4. Experimental Results

OmniVoice was evaluated on Chinese, English, and massive multilingual benchmarks (MiniMax-24 and FLEURS-102).

• Chinese & English Performance:

  • Outperforms all NAR baselines (F5-TTS, ZipVoice, MaskGCT) trained on the same Emilia dataset.
  • Competes with or surpasses SOTA AR models (IndexTTS2, CosyVoice3, Qwen3-TTS) in Speaker Similarity (SIM-o) and Intelligibility (WER/CER), while maintaining faster inference.
  • Achieves a WER of 1.30% on LibriSpeech-PC and 0.84% on Seed-TTS test-zh.

• Multilingual Performance:

  • MiniMax-24: Outperforms commercial systems (ElevenLabs, MiniMax-Speech) in average SIM-o (0.830 vs 0.766) and average WER (2.850 vs 3.774).
  • FLEURS-102: Achieves an average Character Error Rate (CER) of 4.00%, comparable to ground truth. It maintains high intelligibility (CER < 5%) even for languages with less than 10 hours of training data.

• Inference Speed:

  • Achieves a Real-Time Factor (RTF) of 0.0319 (16 steps, batch size 1) on an H20 GPU, outperforming ZipVoice (0.0557) with the same configuration.

5. Significance

OmniVoice represents a significant leap forward in democratizing speech technology:

• Global Accessibility: By supporting 600+ languages, it bridges the gap for low-resource languages, enabling TTS applications for a vast majority of the world's population previously excluded by technology.
• Efficiency vs. Quality: It successfully decouples the trade-off between inference speed (NAR) and quality/intelligibility (AR), proving that a single-stage discrete diffusion model can achieve SOTA results.
• Open Science: By relying entirely on open-source data and releasing the model, it sets a new standard for transparency and reproducibility in multilingual speech synthesis, challenging the dominance of proprietary, closed-source commercial models.

In conclusion, OmniVoice establishes a new frontier for high-quality, high-coverage, and efficient multilingual TTS, demonstrating that minimalist architectures combined with strategic initialization and masking can outperform complex cascaded systems.