21Codec: Decoder-side Speaker Control for Text-to-Speech
The paper proposes 21Codec, a streaming text-to-speech model that achieves robust, decoder-side speaker control by using adversarial training and data augmentation to minimize speaker information in the token stream, thereby ensuring consistent speaker identity across chunks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where a computer can read a sentence and speak it aloud in a voice that sounds exactly like a specific person, complete with their unique tone and rhythm. This is the promise of modern text-to-speech technology, a field that has moved from robotic beeps to near-human realism. At the heart of this ability lies a crucial step called audio compression, or a codec. Think of this process as a translator that converts raw sound waves into a compact stream of digital tokens, much like turning a long novel into a series of shorthand notes. These notes are then sent to a decoder, which reconstructs the sound. For years, these systems have worked well, but they hit a snag when asked to speak in real-time. To keep up with live conversation, the computer must process sound in tiny, rapid bursts. The problem is that when the system tries to guess who is speaking based only on these tiny bursts, it often loses its way, causing the voice to wobble or drift into a different identity mid-sentence.
Researchers at Leiden University in the Netherlands have tackled this instability with a new system they call 21Codec. Their work addresses a fundamental flaw in how current machines handle voice identity. In existing models, the computer tries to figure out who is speaking by looking at the entire sequence of notes it has generated so far. When the system is forced to work in real-time, it can only see a small chunk of those notes at once. Consequently, it re-guesses the speaker's identity for every single chunk, and these guesses often conflict with one another, leading to a voice that sounds like it is changing its mind about who it is supposed to be. The researchers argue that this is not a problem with how the sound is decoded, but rather a problem with how the information is stored. They propose that the voice's identity should not be hidden inside the notes themselves, waiting to be guessed, but should instead be supplied directly to the decoder as a separate instruction.
To achieve this, the team built a system that actively strips the speaker's identity out of the digital notes before they are sent to the decoder. They trained their model using a clever technique involving a "teacher" that tries to identify the speaker from the notes. The model learns to fool this teacher, effectively hiding the speaker's identity within the compressed data stream. Instead of relying on the notes to carry the voice, the system uses a separate control signal, a specific digital fingerprint of the desired speaker, to guide the reconstruction. This signal is applied at the very end of the process, ensuring that the voice remains consistent from the first word to the last, regardless of how small the chunks of data are. The researchers tested this by taking a single sequence of notes and playing them back with different voice fingerprints, successfully changing the speaker without altering the words or the meaning of the sentence.
The results show that this approach works remarkably well for maintaining a steady voice. When the system decoded the same sequence of notes in small, real-time chunks, the voice did not drift, unlike in previous models where the voice would noticeably shift between speakers. The team found that they could change the speaker identity even while the audio was being generated, simply by swapping the control signal, without needing to send any extra data or increase the size of the file. However, this improvement in control comes with a slight trade-off. While the voice sounds natural and the speaker identity is stable, the system is slightly less accurate at preserving the exact words compared to models that do not strip out the speaker information. The researchers measured this by checking how often the computer misheard the words, finding a small increase in errors, though the overall quality of the sound remained high.
This work suggests that for text-to-speech systems to become truly reliable for real-time applications, such as live translation or interactive assistants, the way they handle voice identity needs to change. By moving the responsibility of defining the speaker from the compressed notes to an external control signal, the system avoids the confusion that causes voices to drift. The researchers acknowledge that their model is still being refined and that the numbers they reported are likely conservative, as they stopped training early due to budget constraints. Nevertheless, their findings offer a clear path forward: a system where the voice is not a mystery to be solved from a few notes, but a choice made explicitly by the machine, ensuring that the person speaking remains the same person from start to finish.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.