MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation

Imagine a world where you can take your favorite song and instantly remix it into a completely new style—maybe turning a sad ballad into an upbeat rock anthem—while keeping the original melody and lyrics perfectly intact. This is the dream of "Cover Song Generation," a branch of artificial intelligence that tries to teach computers to be musical chameleons. To do this, AI needs to understand two very different things at once: the "music" (the tune, the rhythm, the notes) and the "words" (the specific sounds that make up language, like the difference between a "b" and a "p"). Think of it like baking a cake where you have to follow a strict recipe for the flavor (the lyrics) while simultaneously improvising a new decoration style (the music). If the AI gets the decoration right but the flavor wrong, you might end up with a chocolate cake that tastes like a lemon. For a long time, AI was great at the decoration but terrible at the flavor, often mangling the words so badly they became gibberish.

This paper introduces a new system called MPEcho that fixes this problem by teaching the AI to listen to the "tiny building blocks" of speech, not just the big words. The researchers realized that to get the lyrics right, the computer needs to know exactly when each tiny sound starts and stops, much like a conductor knowing exactly when a violinist should play a single note. To do this, they built a special tool called Phonsa, which acts like a super-accurate transcriptionist, listening to a singer and writing down the exact timing of every single sound in the song. By combining this super-precise timing with the melody, MPEcho can generate new cover songs where the words are clear and the tune is faithful. The results show that this approach drastically reduces the number of mistakes the AI makes when singing, dropping the error rate from nearly half of all words to less than one-fifth, proving that paying attention to the tiny details of sound is the secret to making AI sing better.

The Problem: When AI Sings, It Mumbles

Cover song generation is a tricky balancing act. You want the AI to keep the core melody and the exact lyrics of a reference song, but change the style, the instruments, and the voice. Previous attempts, like a model called SongEcho, tried to solve this by feeding the AI the "pitch" (how high or low the notes are) and a simple tag that just said "singing" or "not singing."

Imagine trying to tell a friend to sing a song by only humming the tune and saying "I'm singing now" or "I'm not singing now." It's not very helpful! The AI could guess the melody, but it had no idea which words to sing or when to switch from one sound to the next. The result? The AI would often sing the right tune but garble the lyrics, turning "Hello" into "Helo" or "Halo." In fact, the old method made mistakes in about 45.62% of the words. That's like trying to read a menu where half the items are spelled wrong.

The Solution: A New Way to Listen and Sing

The authors, led by Wei-Jaw Lee and Yi-Hsuan Yang, realized that to fix the lyrics, they needed to borrow a trick from a different field called Singing Voice Synthesis (SVS). SVS is usually used to make a robot sing a specific song from a score, and it's very good at it because it knows the exact timing of every tiny sound (phoneme).

They built MPEcho, a new framework that adds a "phoneme encoder" and a "length regulator" to the system.

• The Phoneme Encoder: Instead of just knowing "singing is happening," the AI now gets a list of every single sound (like /h/, /e/, /l/, /o/) and exactly how long each one should last.
• The Length Regulator: This acts like a metronome for the words, stretching or shrinking the sounds so they fit perfectly into the musical timeline.

But there was a catch: to teach MPEcho this, they needed a massive library of songs where every single sound was already timed perfectly. Such data didn't exist. So, they built Phonsa.

Phonsa is like a magical translator. It takes a recording of a singer and automatically writes down the exact start and stop times for every sound in the song. It's based on a famous speech tool called Whisper but was tweaked specifically for singing. It uses a special "chunked" attention system (thinking in small, overlapping pieces) and added special tokens for "breaths" and "boundaries" to handle the messy, emotional nature of singing.

The Results: From Gibberish to Clear Lyrics

The team tested their new system on a dataset of over 1,427 hours of Chinese pop and traditional songs. They compared MPEcho against the old SongEcho model and some other variations.

1. The Magic of Timing: When they used Phonsa to give MPEcho precise phoneme timing, the number of word mistakes (called the Phoneme Error Rate, or PER) dropped dramatically.

   • Old Model (SongEcho): Made mistakes in 45.62% of the words.
   • New Model (MPEcho): Made mistakes in only 18.65% of the words.
   • That is a massive improvement, turning a mumbled mess into something you can actually understand.

2. The Sweet Spot: They found that using only the melody wasn't enough (the words were still messy), and using only the phoneme timing wasn't great either (the melody got weird). But when they combined both—giving the AI the tune and the precise sound timing—the result was the best of both worlds. The song sounded musical, and the lyrics were clear.

3. The "Jam" vs. "SVS" Style: They also tested a different way of organizing the sounds (called "Jam-style," which groups sounds by word) versus their new "SVS-style" (grouping by individual sound). The SVS-style worked much better. The "Jam-style" was like trying to give directions by saying "go to the next street," while the SVS-style was like saying "turn left at the red house, then right at the blue mailbox." The extra precision helped the AI avoid getting lost.

4. Human Opinion: Finally, they asked real people to listen to the songs. The listeners rated the MPEcho songs higher in melody consistency, vocal naturalness, and overall quality. Interestingly, the human listeners seemed to appreciate the clarity of the lyrics even more than the computer metrics suggested, proving that clear words make for a much better listening experience.

What This Means

This paper suggests that if you want an AI to generate full songs that sound human and sing clearly, you can't just look at the big picture (the melody and the words). You have to zoom in and understand the tiny, split-second details of how sounds are made. By combining the best of music generation with the precision of speech synthesis, MPEcho shows that AI can finally sing the lyrics correctly while still sounding like a creative artist.

The researchers are careful to note that this system currently works best with single singers and Mandarin Chinese. They suggest that future work could explore making it work with multiple singers, different languages, and even more emotional expressions. But for now, they've cracked the code on making AI cover songs that don't just sound good, but actually make sense.