Technical Summary: dots.tts

1. Problem Statement

The paper addresses two primary limitations in current Text-to-Speech (TTS) systems:

1. Discrete Token Limitations: Mainstream autoregressive (AR) TTS systems rely on discrete acoustic tokens. While this allows the reuse of Large Language Model (LLM) stacks, the quantization bottleneck limits the system's ability to model a single distribution covering speech, emotional paralinguistics, singing, and general audio.
2. Continuous AR Instability: Recent attempts to model speech as continuous-representation AR generation avoid quantization but suffer from long-range error accumulation. Unlike discrete systems where a codec snaps imperfect samples back to valid configurations, continuous latents faithfully reconstruct every small prediction error, feeding it back as conditioning for the next step. This leads to drift and instability during long rollouts.
3. Immature Post-Training: The post-training stack for continuous AR models is less mature than that of discrete-token systems, lacking robust methods to correct inference-time errors without external reward models.

2. Methodology

dots.tts is a 2B-parameter, fully continuous, end-to-end autoregressive TTS foundation model. It models speech in a continuous latent space using three core architectural and training innovations:

A. Semantic Causal AudioVAE

The system is built upon a high-fidelity AudioVAE that encodes 48 kHz mono speech into a 128-dimensional latent stream at 25 Hz (1920× temporal downsampling).

• Architecture: It uses a causal variant of the BigVGAN-v2 decoder and a fully causal convolutional encoder.
• Training Objectives:
  • Stage 1: Focuses on reconstruction quality using multi-period/sub-band CQT adversarial loss, multi-scale mel-spectral loss, and feature-matching loss, regularized by a KL + flow prior.
  • Stage 2: Enhances learnability for the AR backbone. It adds a WavLM representation-alignment loss (frame-level cosine alignment) and a multitask downstream block (ASR, emotion, speaker classification). This ensures the latent space is semantically structured and prediction-friendly without sacrificing reconstruction fidelity.
• Semantic Encoder: A module transplanted from the AudioVAE training pipeline projects each generated 25 Hz VAE latent patch into a single 6.25 Hz audio-semantic embedding. This strips high-variance acoustic details, providing the LLM with a compact semantic summary rather than raw latents.

B. Autoregressive Flow-Matching Backbone

The backbone consists of three decoupled components:

1. Semantic Encoder: Converts VAE patches to LLM inputs.
2. LLM Backbone: Initialized from Qwen2.5-1.5B, it consumes BPE text tokens and the 6.25 Hz semantic embeddings. It handles long-range text-to-content modeling.
   • Dual Streaming: Supports a 1T1A interleaved mode (alternating text tokens and audio steps) for low-latency dialogue, allowing synthesis to begin before a full utterance is available.
3. Autoregressive Flow-Matching (AR-FM) Head: A Diffusion Transformer (DiT) that generates the next four-frame VAE latent patch.
   • Full-History Conditioning: The head conditions on the LLM hidden state, all previous clean patches, and the current noisy patch.
   • Block-Causal Attention: A specialized training mask allows parallel training across all patches while strictly reproducing the per-step autoregressive context of inference, ensuring numerical identity between training and rollout.

C. Training Pipeline & Post-Training

The training proceeds in three stages:

1. Pretraining:
   • Modality Alignment: Aligns the semantic encoder and AR-FM head with the frozen LLM.
   • General Training: End-to-end training on 1.5M hours of multilingual data.
   • Annealing: Fine-tuning on a high-quality subset.
2. Reward-Free Self-Corrective Alignment (SOAR):
   • Adapts the SOAR concept to the AR flow-matching head.
   • The model performs a detached Euler rollout from its own inference-time state to generate "off-trajectory" errors.
   • It is then trained to steer these errors back toward the clean latent endpoint. This improves robustness and acoustic quality without requiring a reward model or external teacher.
3. CFG-Aware MeanFlow Distillation:
   • Distills the self-corrected teacher DiT into a student DiT.
   • The student predicts the mean velocity over a time interval, fusing Classifier-Free Guidance (CFG) directly into the target.
   • This enables low-latency inference with only 2–4 function evaluations (NFE) per step, eliminating the need for separate conditional/unconditional passes.

3. Key Contributions

1. dots.tts Model: A 2B-parameter fully continuous AR TTS system that removes discrete acoustic tokens while achieving state-of-the-art stability and zero-shot voice cloning quality.
2. Stability Mechanisms: A triad of designs to alleviate continuous AR instability:
   • A semantic causal AudioVAE with WavLM alignment.
   • Decomposition into semantic planning (LLM) and acoustic rendering (AR-FM).
   • Reward-free Self-corrective alignment for the flow-matching head.
3. Efficiency: Implementation of CFG-aware MeanFlow distillation and fully causal 1T1A interleaved streaming, achieving low first-packet latencies (85 ms in plain mode, 54 ms in interleaved mode) suitable for real-time deployment.

4. Experimental Results

The model was trained on 1.5M hours of speech and evaluated across multiple benchmarks:

• Seed-TTS-Eval (Zero-Shot Voice Cloning):

  • Achieved the best average performance among open-source systems.
  • WER: 0.94% (zh), 1.30% (en), 6.60% (zh-hard).
  • SIM (Speaker Similarity): 81.0 (zh), 77.1 (en), 79.5 (zh-hard).
  • Outperformed baselines like CosyVoice 3, Seed-TTS, and VoxCPM 2.

• MiniMax Multilingual Test Set (24 Languages):

  • Led in average speaker similarity (83.9), outperforming the next best baseline (VoxCPM 2 at 82.3).
  • Achieved the highest SIM on 19 of 24 languages.
  • WER was competitive, though slightly higher on low-resource languages due to BPE token coverage limits.

• CV3-Eval (Cross-Lingual Cloning):

  • Led speaker similarity in cross-lingual tasks (75.0 for en→zh, 72.8 for zh→en), significantly outperforming CosyVoice 3.
  • Showed significant gains in the "hard-en" subset after the SOAR post-training stage.

• EmergentTTS-Eval (Expressiveness):

  • Achieved the highest Syntactic Complexity score (65.7%) among all systems (open and closed-source), surpassing Gemini-2.5-Pro.
  • Led open-source systems in Emotions (72.7%) and overall win-rate against gpt-4o-mini-tts.
  • Noted a trade-off where SOAR improved text faithfulness (Syntax) at a slight cost to emotional expressiveness compared to the pre-trained checkpoint.

• Efficiency:

  • On a single NVIDIA H800 GPU, the MeanFlow-distilled model achieved 54 ms Time-to-First-Packet (TTFP) in interleaved streaming mode with an RTF of 0.245.

5. Significance and Claims

The paper claims that dots.tts successfully bridges the gap between the expressiveness of continuous latent generation and the stability required for production. By decoupling semantic reasoning from acoustic rendering and introducing a reward-free self-correction mechanism, the authors demonstrate that continuous AR TTS can achieve state-of-the-art stability and voice cloning quality without discrete quantization.

The work is presented as a reproducible reference stack for continuous-AR TTS, released under the Apache 2.0 license with full training and inference code. The authors position the system as a viable solution for real-time, conversational applications, offering a balance of high fidelity, multilingual coverage, and low latency that previous continuous models struggled to achieve simultaneously.

Limitations Acknowledged:

• Low-Resource Languages: Performance on languages with divergent scripts or under-representation (e.g., Arabic, Hindi) is limited by BPE token coverage.
• Scope: The current release focuses on speech; singing and unified speech-sound generation are not covered.
• Control: The system is evaluated under canonical zero-shot conditions without explicit style or instruction control (though a caption-paired dataset suggests a path forward).
• Safety: The authors acknowledge the risks of high-fidelity zero-shot voice cloning and encourage the use of consent-aware policies and detection tools.