💬 NLP

Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

Confucius4-TTS is a multilingual zero-shot text-to-speech system that enables transcript-free cross-lingual voice cloning by employing a two-stage architecture with a learnable speaker encoder to extract timbre features from self-supervised speech representations, achieving high intelligibility and speaker similarity across 14 languages.

Original authors: Huaxuan Wang, Huimin Wang, Ruiyu Zhang, Yingjie Li, Yitao Duan

Published 2026-08-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Huaxuan Wang, Huimin Wang, Ruiyu Zhang, Yingjie Li, Yitao Duan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a robot to speak like your favorite movie character. In the world of computer science, this is called "Text-to-Speech" (TTS). Usually, to make the robot sound exactly like that character, you need two things: a short audio clip of them talking, and a written transcript (a script) of exactly what they said in that clip. Think of the transcript as a map; without it, the robot gets lost trying to match the voice to the words. But here's the problem: in the real world, most audio clips we find—like a random TikTok video or a podcast snippet—don't come with a script. They are "untranscribed." This makes it incredibly hard to clone voices from the wild, especially when you want the robot to speak a different language than the original clip. This paper tackles that exact headache: how do we get a robot to sound like a specific person in a new language, even if we don't have the script for the original voice?

Enter Confucius4-TTS, a new system from NetEase Youdao that acts like a masterful voice chameleon. The researchers built a two-stage machine that can take a short audio clip of a stranger and a piece of text in a different language, and generate speech that sounds like that stranger speaking the new language—all without needing the original script.

Here is how it works, using a simple analogy. Imagine the system is a two-person team: a Translator and an Actor.

First, the Translator (called the Text-to-Semantic module) looks at the target text you want spoken. But instead of just reading the words, it also listens to the reference audio clip. Crucially, it doesn't need to know what the reference audio says; it just needs to know who is saying it. To do this, it uses a special "Learnable Speaker Encoder." Think of this encoder as a voice fingerprint scanner. It takes the messy, untranscribed audio and extracts the unique "timbre" or "color" of the voice—like the texture of a guitar string—ignoring the specific words. It then hands this "voice fingerprint" to the Translator, which combines it with your new text to create a set of "semantic tokens." These tokens are like a blueprint for the speech: they contain the meaning and the rhythm, but not the final sound yet.

Next, the Actor (called the Semantic-to-Acoustic module) takes that blueprint and the voice fingerprint. This part uses a fancy technique called "conditional flow matching," which is like a sculptor slowly turning a block of clay into a statue. It starts with random noise and gradually shapes it into a mel-spectrogram (a visual map of sound waves) that matches the voice fingerprint and the blueprint. Finally, a "vocoder" (a sound synthesizer) turns that map into actual audio waves you can hear.

The paper's main finding is that this system works incredibly well without the script. The authors tested it on 14 languages, including Chinese, English, Japanese, Korean, and several European and Southeast Asian languages. When they pitted Confucius4-TTS against other top systems on a benchmark called CV3-Eval (which tests cross-language voice cloning), it achieved an average Word Error Rate (WER) of 3.73% across six different language directions. That's a very low error rate, meaning the speech is highly intelligible. In fact, on the X-Voice benchmark, it beat or matched other leading systems in seven different source-to-Chinese language pairs.

The researchers also found that if you do happen to have the script for the reference audio, the system can switch to a "continuation cloning" mode. This is like giving the Actor the script and the voice fingerprint. While this makes the voice sound even more like the original speaker (improving speaker similarity scores), it comes with a tiny trade-off: the speech might be slightly less intelligible than the script-free version. However, the default "transcript-free" mode is the real star, proving you don't need a map to find the voice.

In human tests, where real people listened to the results and ranked them, Confucius4-TTS often took the top spot for timbre similarity (how much it sounded like the original person) and naturalness (how human it sounded). For example, in tests converting Chinese to English, it ranked first or second in almost every category, beating commercial giants and other open-source models.

The paper explicitly rules out the idea that you must have a transcript to get high-quality cross-lingual cloning. Previous methods relied heavily on forced alignment (matching words to sounds perfectly) or synthetic training pairs, but Confucius4-TTS suggests that by using self-supervised speech representations (AI that learned to understand speech just by listening to it, without being taught the words), you can extract the voice identity directly. The authors are confident in these results because they tested them on large-scale data (about 500,000 hours of speech) and validated them across multiple public benchmarks and internal human evaluations.

So, what does this mean for the future? The authors suggest this opens the door for video dubbing, audiobooks, and accessibility tools where you can instantly clone a voice from a random clip and have it speak any language, no script required. They have even released the code and models to the public, inviting others to build on this "transcript-free" magic. While the system isn't perfect (it still has a small error rate and could be faster), it represents a significant step toward making voice cloning as easy as just hitting "play" on a video.

Technical Summary: Confucius4-TTS

Problem Statement

Recent advances in zero-shot text-to-speech (TTS) have significantly improved speech quality and voice cloning fidelity. However, a critical limitation persists: many state-of-the-art zero-shot TTS systems require a transcript of the reference audio prompt at inference time. This dependency hinders cross-lingual voice cloning, as real-world ("in-the-wild") reference audio is frequently untranscribed, particularly for low-resource languages and dialects. While some approaches attempt to mitigate this using forced alignment or synthetic prompt pairs, a robust solution that eliminates the transcript requirement while maintaining high intelligibility and speaker similarity across languages remains a challenge.

Methodology

Confucius4-TTS is a multilingual zero-shot TTS system designed to clone unseen speakers across 14 languages without requiring reference transcripts. The system employs a two-stage architecture:

1. Text-to-Semantic (T2S) Module

Architecture: A decoder-only Transformer (24 layers, hidden size 1280) that predicts semantic speech tokens.
Input Conditioning: The model takes target text embeddings and a speaker embedding.
- Text Processing: Uses a BPE-based tokenizer from a pre-trained Large Language Model (LLM). A lightweight MLP projects these embeddings to the T2S hidden dimension.
- Speaker Conditioning: Instead of using a pre-trained speaker verification encoder, the system utilizes a jointly trained speaker encoder. This encoder operates on self-supervised speech representations (SSL) derived from a w2v-BERT 2.0 encoder. It pools frame-level SSL features into a fixed-size speaker embedding ( $e_r$ ). This approach allows the extraction of timbre features directly from raw audio without transcripts.
Training: The T2S module is trained with an autoregressive cross-entropy objective to predict semantic tokens conditioned on the target text and the speaker embedding. The pre-trained LLM embedding table remains frozen.

2. Semantic-to-Acoustic (S2A) Module

Architecture: A conditional flow-matching (CFM) model with a Diffusion Transformer (DiT) backbone.
Function: Converts predicted semantic tokens into mel-spectrograms.
Conditioning:
- Semantic: Combines semantic-token embeddings with hidden states from the T2S module ( $H_{T2S}$ ). Using continuous hidden states alongside discrete tokens helps reduce the information bottleneck and preserves context for prosody and timbre.
- Speaker: Uses a global speaker embedding ( $g$ ) extracted from a frozen speaker verification model (CAM++).
- Acoustic Context: Incorporates a prompt mel-spectrogram derived from the reference audio.
Training: Optimized using conditional flow matching along an optimal transport path. The model is trained with classifier-free guidance (CFG) by jointly dropping conditions (semantic, speaker, prompt) with a fixed probability.

Dual-Mode Inference

The system supports two inference recipes using the same model:

Reference Cloning (Default): Conditions only on the target text and the speaker embedding extracted from the reference audio. This mode requires no transcript and offers flexibility in prosody and style.
Continuation Cloning: If a reference transcript is available, the model conditions on the reference transcript and reference semantic tokens in addition to the speaker embedding. This mode yields higher speaker similarity at the cost of slightly higher error rates.

Key Contributions

Transcript-Free Cross-Lingual Cloning: Confucius4-TTS is a multilingual zero-shot TTS system capable of cloning unseen speakers in 14 languages without requiring a reference transcript.
Jointly Trained Speaker Encoder: Introduces a speaker encoder trained jointly with the T2S module that operates on self-supervised speech representations, enabling effective timbre extraction without forced alignment or synthetic prompt pairs.
Flexible Inference Modes: Supports both transcript-free reference cloning and transcript-based continuation cloning within a single model architecture.
Comprehensive Evaluation and Release: Provides objective and subjective evaluations against open-source and commercial baselines, accompanied by an open-source release of code, model checkpoints, and demos.

Experimental Results

The system was trained on approximately 500,000 hours of multilingual speech data covering 14 languages (Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay, and Vietnamese).

Cross-Lingual Performance

CV3-Eval: On the cross-lingual subset of this benchmark, Confucius4-TTS achieved an average Word Error Rate (WER) of 3.73% across six language directions. It obtained the lowest error rate on four of the six directions, significantly outperforming systems like CosyVoice 2 (e.g., 4.87% vs. 48.10% for Japanese-to-Chinese).
X-Voice: Evaluated on seven source languages targeting Chinese, the system achieved the lowest Character Error Rate (CER) on four directions and remained within 0.3 absolute points of the best system on the others.

Intra-Lingual and Multilingual Performance

Seed-TTS-eval: In intra-lingual zero-shot tasks, the system achieved 1.49 WER and 0.700 Speaker Similarity (SIM) for English, and 0.94 CER and 0.765 SIM for Chinese. Continuation cloning improved SIM to 0.715 (English) and 0.766 (Chinese).
MiniMax-MLS-Test: Across 11 evaluated languages, Confucius4-TTS achieved speaker similarity above 72% in reference cloning mode. Continuation cloning consistently improved SIM across all languages when transcripts were available.

Human Evaluation

On an internal cross-lingual test set (zh↔en, zh↔ko, zh↔ja), Confucius4-TTS ranked first or second in timbre similarity and naturalness across all language pairs. It achieved the best overall rank in human evaluation for en→zh, zh→ko, and zh→ja, outperforming strong baselines including commercial systems like ElevenLabs and MiniMax-Speech.

Significance

Confucius4-TTS addresses the practical bottleneck of transcript dependency in zero-shot TTS, enabling high-fidelity voice cloning for real-world, untranscribed audio. By leveraging self-supervised representations and a jointly trained speaker encoder, the system achieves competitive intelligibility and speaker similarity across diverse languages without auxiliary alignment modules. The authors claim that the system performs competitively against both open-source and commercial systems, offering a robust solution for applications such as video dubbing, audiobooks, and accessibility tools where reference transcripts are unavailable. The open-source release aims to facilitate further research in multilingual and cross-lingual speech generation.