Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation

Vevo2 is a unified framework for controllable speech and singing voice generation that employs novel audio tokenizers and a two-stage modeling approach with specialized training strategies to achieve flexible control over content, prosody, style, and timbre while demonstrating strong generalization across diverse synthesis tasks.

The Gist

Imagine you have a magical voice box that can do anything: it can tell a story, sing a song, whisper a secret, or even turn a humming tune into a full-blown opera. For a long time, building this box was like trying to build two separate machines—one for talking and one for singing. They were built differently, needed different blueprints, and rarely talked to each other.

This paper introduces Vevo2, a new "universal voice engine" that finally unifies speech and singing into one single, powerful system. Think of it as upgrading from having a toaster and a blender to having a Swiss Army Knife that can do both jobs perfectly, plus a few extra tricks.

Here is how Vevo2 works, explained through simple analogies:

1. The Problem: The "Language Barrier" Between Talking and Singing

Previously, computers struggled to learn singing because there wasn't enough "singing homework" (data) compared to talking. Also, singing is like talking on steroids—it has strict rules about pitch (melody) and rhythm that regular talking doesn't have.

• The Old Way: You had to teach the computer to talk, then start over and teach it to sing from scratch.
• The Vevo2 Way: It realizes that talking and singing are actually cousins. If you teach the computer to understand the rhythm and emotion of talking, it can easily learn the melody and rhythm of singing. They help each other!

2. The Secret Sauce: Two Special "Translators" (Tokenizers)

To make this work, Vevo2 uses two special translators that turn sound into a language the computer can read (called "tokens").

• Translator #1: The "Melody & Rhythm" Decoder (Prosody Tokenizer)

  • The Analogy: Imagine you are trying to describe a song to a friend who can't hear the words, only the tune. Instead of writing down sheet music (which is hard to get for every song), Vevo2 uses a color wheel.
  • How it works: It looks at the sound and converts the pitch and rhythm into a sequence of colors. Whether it's a human voice, a piano, or a violin, they all get translated into the same color pattern. This means the computer can learn to sing by listening to a piano, or hum a tune and have it turned into a song, without needing a music teacher to write down the notes first.

• Translator #2: The "Personality & Content" Decoder (Content-Style Tokenizer)

  • The Analogy: Think of this as a recipe card. It separates the ingredients (the words you want to say) from the chef's style (the accent, emotion, or voice timbre).
  • How it works: It takes the words and the "vibe" (like "sad," "happy," or "opera singer") and writes them down as a code. Crucially, it strips away the identity of the speaker (the voice timbre) so the computer can swap voices later. It's like writing a script that says "Say 'Hello' with a sad voice," without specifying who says it.

3. The Training: The "Two-Stage Cooking Class"

Vevo2 learns in two distinct steps, like a cooking school:

• Stage 1: The Scriptwriter (Auto-Regressive Model)

  • The computer reads the "Recipe Card" (words + style + melody hints) and writes a rough draft of the song.
  • The Trick: The paper introduces a clever training method where the computer practices two modes at once:
    1. Guessing the Rhythm: "Here are the words, guess the melody." (Good for talking).
    2. Following the Rhythm: "Here are the words AND the melody, now write the song." (Good for singing).
  • By mixing these up randomly, the computer learns to bridge the gap between talking and singing seamlessly.

• Stage 2: The Sound Engineer (Flow-Matching Model)

  • Once the script is written, this stage takes the rough draft and turns it into high-quality audio. It's like a sound engineer taking a demo tape and mixing it into a studio-quality track, ensuring the voice sounds natural and matches the target "timbre" (the specific voice you want to clone).

4. The "Polishing" Step: Multi-Objective Post-Training

After the initial training, the computer is good but sometimes makes mistakes (like singing the wrong words or forgetting the melody).

• The Analogy: Imagine a student who is great at math but terrible at spelling. You don't just teach them more math; you give them a special exam that grades them on both math and spelling simultaneously.
• Vevo2 uses a "Multi-Objective" reward system. It gets points for being clear (intelligibility) and for following the tune (prosody). If it tries to cheat by being clear but ignoring the melody, it gets a lower score. This forces the model to master both skills at once.

5. What Can Vevo2 Actually Do?

Because it's so flexible, Vevo2 can do things that were previously very hard or impossible:

• Hum-to-Sing: You hum a tune, and it turns it into a professional song with lyrics.
• Instrument-to-Sing: You play a melody on a flute, and the computer sings it back to you.
• Lyric Editing: You can change the words of a song without changing the melody or the singer's voice.
• Voice Swapping: You can make a singer sound like a different person, or make a whisper sound like a shout, all while keeping the original emotion and rhythm.

The Bottom Line

Vevo2 is a universal voice engine that treats talking and singing as two sides of the same coin. By using smart "translators" to convert sound into a universal language and training the system to handle both styles together, it creates a tool that is not only better at singing than previous models but also better at talking. It's a giant leap toward a future where your computer can be your personal singer, storyteller, and voice actor all at once.

Technical Summary

Here is a detailed technical summary of the paper "Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation."

1. Problem Statement

Controllable human voice generation faces significant challenges when unifying speech and singing voice within a single framework.

• Data Scarcity & Annotation: High-quality singing datasets are scarce and heavily reliant on expert annotations (e.g., precise music notation, lyrics-to-note alignment), which limits scalability compared to massive speech corpora.
• Controllability Complexity: Existing unified models struggle to independently manipulate multiple attributes: text/lyrics, prosody/melody, style (accent, emotion, technique), and timbre (speaker identity).
• Domain Gap: Speech and singing differ fundamentally; speech relies on implicit prosody, while singing requires strict adherence to melody. Bridging these domains without sacrificing quality or controllability remains an open problem.

2. Methodology

Vevo2 employs a two-stage architecture (Auto-Regressive + Flow-Matching) designed to unify speech and singing through novel tokenization and training strategies.

A. Unified Audio Tokenizers

Vevo2 introduces two distinct tokenizers to handle the diverse nature of audio data:

1. Unified Prosody Tokenizer (Music-Notation-Free):
   • Input: Chromagram features (statistical distribution of spectral energy) extracted from raw audio.
   • Mechanism: A single-codebook VQ-VAE operating at 6.25 Hz.
   • Advantage: Removes octave information (unifying speech pitch and singing melody) and eliminates the need for manual music notation. It can capture melody from speech, singing, and even instrumental sounds.
2. Unified Content-Style Tokenizer:
   • Input: Reconstructs both Chromagram features and Whisper hidden features (linguistic content).
   • Mechanism: A VQ-VAE operating at 12.5 Hz.
   • Advantage: Encodes linguistic content, prosody, and style while achieving robust timbre disentanglement. The lower frame rate reduces sequence length for the AR model.

B. Speech-Singing Joint Training

To bridge the gap between speech (implicit prosody) and singing (explicit melody), the Auto-Regressive (AR) Content-Style Modeling stage utilizes two strategies applied randomly to all data:

• Implicit Prosody Learning (IPL): The model predicts content-style tokens solely from text (standard TTS approach).
• Explicit Prosody Learning (EPL): The model takes both text and prosody tokens (from the tokenizer) as input to predict content-style tokens.
• Strategy: By randomly mixing IPL and EPL for both speech and singing data, the model learns a unified representation where it can infer prosody from text or be guided by explicit melody tokens.

C. Multi-Objective Post-Training

To enhance controllability and robustness (especially for out-of-distribution data like instrumental sounds), Vevo2 employs a post-training phase using Group Relative Policy Optimization (GRPO):

• Intelligibility Reward: Trained on preference data (INTP) to ensure the generated lyrics match the text.
• Prosody Similarity Reward: Evaluates the cosine similarity between the reconstructed chromagram of the generated audio and the ground-truth chromagram (derived from MIDI or source audio).
• Goal: Prevents over-optimization on a single metric, ensuring the model follows both text and melody accurately.

D. Inference Controllability

Vevo2 supports versatile tasks by flexibly combining inputs:

• Text-to-Speech/Singing: Text + Style/Timbre Reference.
• Voice/Singing Conversion: Source Audio + Timbre Reference.
• Editing: Edited Text + Raw Audio (preserving original prosody).
• Unique Capabilities: Humming-to-Singing and Instrument-to-Singing (using instrumental MIDI/rendered audio as prosody prompts).
• Duration & Pitch Control: Achieved by linearly scaling the chromagram features of the prosody source or shifting the source F0 before token extraction.

3. Key Contributions

1. Unified Tokenization: Proposes the first music-notation-free prosody tokenizer capable of encoding melody from speech, singing, and instruments, and a timbre-disentangled content-style tokenizer for unified modeling.
2. Unified Framework (Vevo2): A single model that achieves state-of-the-art performance in both speech and singing generation, enabling seamless transitions between synthesis, conversion, and editing tasks.
3. Joint Training Strategy: Introduces Explicit and Implicit Prosody Learning to effectively bridge the domain gap between speech and singing.
4. Multi-Objective Alignment: Demonstrates that combining intelligibility and prosody similarity rewards via GRPO significantly improves stability and generalization, particularly for complex tasks like instrument-to-singing.

4. Experimental Results

Vevo2 was evaluated on zero-shot TTS, SVS, Voice Conversion (VC), Singing Voice Conversion (SVC), and editing tasks across English and Chinese.

• Unified Modeling Benefits: Joint pre-training on speech (101k hours) and singing (7k hours) data improved performance in both domains. For singing, it reduced Word Error Rate (WER) from 25.38 (singing-only) to 15.78 (unified).
• Singing Voice Synthesis (SVS): Vevo2 outperformed baselines (StyleSinger, TCSinger) in naturalness and speaker similarity. It achieved a Melody-MOS > 2.0, indicating it "roughly follows the melody contour" in most cases.
• Editing Tasks: Vevo2 demonstrated superior prosody preservation (PS-CMOS/MS-CMOS) during speech and lyric editing compared to F5-TTS and SSR-Speech, thanks to explicit prosody token control.
• Instrument-to-Singing: The model successfully converted instrumental sounds (flute, violin) and humming into singing voice, a capability enabled by the chromagram-based tokenizer.
• Duration & Pitch Control: Achieved >97% duration consistency and significantly improved speaker similarity in VC/SVC tasks when using pitch-shifted source inputs.

5. Significance

• Scalability: By removing the dependency on expert music notation, Vevo2 allows for the use of large-scale, in-the-wild singing data, addressing the data scarcity bottleneck in SVS.
• Versatility: It unifies a wide range of tasks (TTS, SVS, VC, SVC, Editing) into a single framework, simplifying deployment and enabling cross-domain applications.
• New Applications: The ability to use instrumental sounds or humming as prosody prompts opens new creative avenues for music generation and human-AI collaboration.
• Generalization: The multi-objective post-training proves effective in aligning models for out-of-distribution inputs, suggesting a path toward more robust, general-purpose audio foundation models.

The code and models are released via the Amphion toolkit, facilitating further research in unified audio generation.