⚡ electrical engineering

UniVoice: A Unified Model for Speech and Singing Voice Generation

UniVoice is a unified speech and singing voice generation framework based on conditional flow matching that factorizes conditioning into content, melody, and timbre, utilizing a learned null melody token to enable explicit melody control for singing while allowing natural prosody inference for speech.

Original authors: Junjie Zheng, Huixin Xue, Shihong Ren, Chaofan Ding, Hao Liu, Zihao Chen

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junjie Zheng, Huixin Xue, Shihong Ren, Chaofan Ding, Hao Liu, Zihao Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a single robot to do two very different jobs: telling a story (speaking) and performing an opera (singing).

Usually, you would build two separate robots. One is designed to sound natural and conversational, picking up its rhythm from the words it's reading. The other is designed to follow a strict musical score, hitting every note and beat exactly as written.

The problem with trying to build just one robot to do both is that the instructions for the two jobs contradict each other.

For singing, you need to force the robot to follow a specific melody (like a train on a track).
For speaking, if you force it to follow a track, it sounds robotic and unnatural. It needs to be free to wander based on the emotion of the story.

If you just mix the training data together, the robot gets confused. It tries to follow the track when it should be free, and tries to be free when it should be following the track. The result is usually bad at both.

The Solution: UniVoice

The researchers created UniVoice, a new "robot" (a computer model) that solves this by using a clever trick called Factorized Conditioning. Think of this as giving the robot three separate control knobs instead of one big, messy switch.

Here is how the three knobs work:

The Content Knob (What is being said): This reads the lyrics or the text. It's the same for both speaking and singing.
The Timbre Knob (Who is saying it): This listens to a short sample of a person's voice (like a 5-second clip) to copy their unique sound, whether they are whispering or belting a high note.
The Melody Knob (The tune): This is the magic part.
- When Singing: You plug in a specific musical score (MIDI notes). The robot must follow this track.
- When Speaking: Instead of plugging in a track, you plug in a special "Null Token." Think of this as a "Do Not Disturb" sign for the melody. It tells the robot: "Ignore the music track; just listen to the words and figure out the rhythm yourself."

The Engine: A Shared Brain

Under the hood, UniVoice uses a powerful engine called a Diffusion Transformer (DiT). Imagine this as a highly skilled artist who can paint anything.

In the past, you might have needed two different artists (one for speech, one for singing).
UniVoice uses one artist who is very good at listening to the three control knobs.
When the "Melody Knob" is set to the "Null Token," the artist knows to improvise the rhythm naturally. When the knob is set to a specific song, the artist follows the sheet music perfectly.

Why This Matters (The Results)

The researchers tested this on a massive amount of data (30,000 hours of speech and 35,000 hours of singing). Here is what they found:

It's a master of both: UniVoice speaks almost as well as the best speech-only robots (like F5-TTS) and sings much better than previous "all-in-one" robots.
It's efficient: It does all this with a relatively small size (0.3 billion parameters), whereas older unified models were much larger and still performed worse.
The "Null Token" works: They proved that using that special "Do Not Disturb" sign for speech is mathematically the right way to let the model ignore the melody without breaking its ability to sing later.

The New Test: UNISINGING-EVAL

To make sure their robot was truly good, the team built a new test called UNISINGING-EVAL. Imagine a talent show with 12 different musical genres (from Pop to Jazz to Hip-Hop). They tested the robot to see if it could switch between speaking and singing while keeping the same voice and handling different styles.

The Bottom Line

UniVoice is like a universal voice actor. It doesn't need to be retrained or have a different brain for speaking vs. singing. It simply looks at the instructions:

"Here is the text, here is the voice, and here is a song" -> Sings perfectly.
"Here is the text, here is the voice, and here is no song" -> Speaks naturally.

By separating the instructions, they stopped the robot from getting confused, allowing it to be excellent at both jobs simultaneously.

Technical Summary: UniVoice

Problem Statement

Text-to-speech (TTS) and singing voice synthesis (SVS) share the goal of generating human vocal audio from symbolic inputs but impose fundamentally conflicting requirements on the generation process. TTS relies on flexible, language-driven prosody inferred from linguistic context, whereas SVS requires explicit melody control (e.g., MIDI notes) and precise rhythmic alignment.

Training a single unified model on mixed data is challenging because the conditioning semantics are incompatible: melody conditions must act as strong constraints for singing but should not restrict the natural, context-dependent prosody of speech. Naïve joint training forces the model to assign conflicting meanings to melody-related conditions, leading to gradient interference and degraded generation quality for both modalities.

Methodology

UniVoice is a unified framework for speech and singing generation based on Conditional Flow Matching (CFM) with a Diffusion Transformer (DiT) backbone. The core innovation lies in factorized conditioning and a learned null melody token.

1. Architecture

The model utilizes a shared DiT backbone (adapted from Wan-DiT) with 24 layers, 16 attention heads, and approximately 0.3 billion parameters. It operates on latent features extracted by a Song Bloom VAE at 25 Hz.

Shared Backbone: Both speech and singing are treated as conditional modes of the same continuous flow.
Adaptive Modulation: A lightweight Task Token (indicating "speech" or "singing") is injected via Adaptive Layer Normalization (AdaLN) to modulate the shared backbone without introducing separate task-specific heads.

2. Factorized Conditioning

To resolve the representation conflict, UniVoice decomposes the conditioning signal $c$ into four orthogonal axes:

Content ( $c_{cnt}$ ): Phoneme sequences encoded by a ConvNeXt-based encoder.
Melody ( $c_{mel}$ ):
- For Singing: Encoded as MIDI note sequences using a Conformer encoder.
- For Speech: Replaced by a learned null melody token ( $e_\emptyset$ ). This token signals the model to infer prosody from linguistic and acoustic context rather than explicit melody constraints.
Timbre ( $c_{tmb}$ ): Derived from a 3–10 second reference audio prompt, encoded via the same VAE used for the target audio. This enables zero-shot voice cloning across both modalities.
Task Token ( $c_{tsk}$ ): Discrete embedding to distinguish generation modes.

3. Theoretical Justification

The paper posits that the learned null token acts as an efficient amortized approximation of melody marginalization. In speech generation, the ideal vector field should marginalize over all possible melody realizations. The null token allows the model to learn the optimal "melody-absent" representation without interference from singing-specific gradients, effectively performing axis-selective classifier-free guidance.

4. Training and Inference

Data: Trained on a mixed dataset of 30,000 hours of speech (from the Emilia dataset) and 35,000 hours of singing (studio and in-the-wild recordings).
Training: Uses a batch size of 32,000 frames across 8 A800 GPUs for 100 epochs. Conditions (text, melody, audio prompt) are randomly dropped with 10% probability to enable classifier-free guidance.
Inference: Generates audio using an Euler ODE solver with 32 steps. Axis-specific guidance scales are applied (e.g., $w_{txt}=5$ for text, $w_{mel}=1$ for melody).

Key Contributions

Unified Architecture: A single flow-matching model with a shared DiT backbone that generates both speech and singing without task-specific heads.
Factorized Conditioning: A scheme separating content, melody, and timbre to eliminate gradient conflicts between speech prosody modeling and singing melody control.
Learned Null Melody Token: A mechanism for speech generation that replaces explicit melody inputs with a learned token, theoretically grounded as a marginalization over the melody variable.
UNISINGING-EVAL Benchmark: A new evaluation benchmark covering 12 musical styles, designed to test unified voice generation across diverse styles, phrase-level timestamps, and cross-modal consistency.

Experimental Results

Evaluated on a cross-sentence generation task with 10-second clips:

Speech Generation: UniVoice achieves a Phoneme Error Rate (PER) of 5.26%, comparable to dedicated TTS systems like F5-TTS (5.21%) and CosyVoice3 (5.30%), despite using significantly less speech data and a unified architecture. It achieves the highest speech naturalness score (S-MOS 3.76) among unified models.
Singing Generation: UniVoice achieves a PER of 16.22% and a Speaker Similarity (SIM) of 35.70%. This substantially outperforms the unified baseline Vevo1.5 (PER 45.07%) and the dedicated singing model Soul-X-Singer (PER 26.22%).
Efficiency: With only 0.3B parameters and 65k training hours, UniVoice outperforms larger unified (Vevo1.5, 1B params) and dedicated singing models, demonstrating the efficiency of factorized conditioning.

Ablation Studies confirm that:

Removing factorized conditioning causes severe degradation (Speech PER rises to 12.31%; Singing PER to 23.45%).
The learned null token is superior to a fixed zero vector.
Task tokens are essential for modality adaptation.
Explicit melody conditioning is critical for singing but detrimental if forced onto speech.

Significance and Limitations

Significance:
UniVoice demonstrates that a single model can effectively bridge the gap between speech and singing by decoupling their conflicting conditioning requirements. It establishes a new standard for unified voice generation, offering competitive quality to dedicated systems while enabling consistent speaker identity across speaking and singing styles. The introduction of UNISINGING-EVAL addresses the lack of comprehensive benchmarks for this emerging field.

Limitations:

Speaker Similarity: There remains a gap in Speaker Similarity (SIM) compared to speech-only systems, indicating a trade-off between cross-modal sharing and voice cloning fidelity.
Language Scope: Evaluation is currently limited to English and Chinese.
Latency: The 32-step ODE sampling process introduces higher latency compared to few-step generation methods.
Style Coverage: Mixed vocal styles like rap or spoken singing are not explicitly modeled.
Ethical Risks: The capability for zero-shot voice cloning across modalities raises concerns regarding deepfakes and copyright, necessitating responsible deployment measures like watermarking and consent-based enrollment.