LibriTTS-VI: A Public Corpus and Novel Methods for Efficient Voice Impression Control

This paper introduces LibriTTS-VI, the first public corpus for numerical voice impression control, and proposes novel training methods to mitigate impression leakage, achieving significantly improved controllability over existing prompt-based approaches.

The Gist

Imagine you have a super-smart robot voice that can read any text out loud. You want this robot to sound a specific way: maybe "calm," "bright," or "youthful." In the world of AI, this is called Voice Impression Control.

However, until now, building this robot has been like trying to bake a cake without a recipe book (no public data) and with a sticky problem: whenever you asked the robot to sound "calm," it would accidentally sound like the person you showed it as an example, rather than just being calm.

This paper, LibriTTS-VI, solves both problems. Here is the breakdown in simple terms:

1. The Problem: The "Sticky" Reference

Think of the old way of doing this like giving a chef a photo of a specific person (the "reference") and saying, "Make the soup taste like this person's voice, but also make it 'spicy'."

The problem is Impression Leakage. The chef gets confused. They look at the photo, see the person's unique features, and accidentally mix those features into the "spiciness." The result isn't just "spicy soup"; it's "spicy soup that tastes exactly like that specific person." You wanted to control the flavor (the impression), but the AI got stuck on the chef (the speaker identity).

Also, nobody had a public "cookbook" (a dataset) with these specific flavor ratings, so researchers couldn't easily learn how to fix it.

2. The Solution: The New Cookbook (LibriTTS-VI)

First, the authors created LibriTTS-VI.

• The Analogy: Imagine they took a massive library of audiobooks and hired four expert food critics to taste every sentence and rate it on 11 different "flavor scales" (like Bright vs. Dark, Calm vs. Restless, Young vs. Old).
• The Result: They turned this into a public "recipe book" that anyone can use. Now, researchers have a clear map of what "calm" or "bright" actually sounds like in data form.

3. The Fix: Untangling the Knots

The authors realized the "leakage" happened because they were using the same audio clip to teach the AI two things at once: "Who is speaking?" and "What is the mood?" It was like asking a student to learn math and history from the exact same paragraph of text; the brain gets mixed up.

They proposed two clever ways to fix this:

Method A: The "Double-Date" Strategy (Disentanglement)

Instead of using one audio clip for both lessons, they use two different clips from the same person.

• The Analogy: Imagine you want to teach a student how to draw a "happy face" (the mood) using a specific artist's style (the speaker).
  • Old Way: Show them a drawing of a sad face by that artist and say, "Draw this, but make it happy." The student gets confused and draws a sad face that looks like the artist.
  • New Way (VIC-dis): Show them a drawing of a neutral face by that artist to learn the style. Then, show them a different drawing (maybe a photo of a happy person) to learn the mood.
  • Result: The AI learns the artist's "handwriting" separately from the "emotion," so it can mix and match perfectly without leaking the wrong emotion.

Method B: The "Ghost" Strategy (Reference-Free)

This is even more radical. What if you didn't need a reference photo at all?

• The Analogy: Instead of showing the chef a photo of a person, you just give them a precise instruction card: "Make it 3.5 out of 7 on the 'Youthful' scale."
• How it works: The AI generates a "ghost" voice (random noise) that has no personality, and then paints the "Youthful" mood onto it. Since there is no real person in the picture to leak their identity, the AI can focus 100% on hitting the exact number you asked for.

4. The Results: Why It Matters

The team tested these methods against the old way and even against a brand-new AI that uses big language models (like the ones that write essays).

• Precision: The old methods were like a dimmer switch that was stuck; you asked for "bright," and it gave you "kind of bright but also a bit like the reference speaker." The new methods are like a precise slider; you ask for "3.5," and it gives you exactly 3.5.
• The LLM Comparison: They tested a fancy new AI that uses text prompts (e.g., "Make the voice sound like a tired old man"). They found that this new AI was imprecise. If you told it to make the voice "calm," but the text said "I am so excited!", the AI got confused and made the voice sound excited anyway. The new methods in this paper kept the mood and the text separate, so the voice stayed calm even if the text was exciting.

Summary

In short, this paper gave the AI world a public dictionary of voice feelings and invented two new teaching tricks to stop the AI from getting confused between "who is speaking" and "how they are feeling." The result is a voice robot that can be dialed in with mathematical precision, just like turning a volume knob, without accidentally changing the character of the voice.

Technical Summary

Here is a detailed technical summary of the paper "LibriTTS-VI: A Public Corpus and Novel Methods for Efficient Voice Impression Control."

1. Problem Statement

The paper addresses two critical limitations in Voice Impression Control (VIC) for Text-to-Speech (TTS) systems:

1. Lack of Public Data: Existing VIC methods rely on private corpora, hindering reproducibility and further research.
2. Impression Leakage: Current methods suffer from "impression leakage," where the synthesized voice is biased toward the acoustic characteristics of the reference audio rather than the target numerical voice impression (VI). This occurs because the same reference utterance is used to condition both the speaker identity and the target voice impression, causing the model to entangle these two factors.

Voice Impression (VI) is defined as an 11-dimensional vector (e.g., Brightness, Calmness, Power) rated on a 7-point scale, allowing for fine-grained, numerical control of voice characteristics.

2. Methodology

The authors propose a comprehensive solution involving a new dataset and two novel architectural strategies to mitigate impression leakage.

A. LibriTTS-VI Corpus

To solve the data scarcity issue, the authors created LibriTTS-VI, the first public corpus for VIC, built upon the LibriTTS-R dataset.

• Annotation: 130 utterances from distinct speakers were manually annotated by four experts on 10 of the 11 VI dimensions using a 7-point Likert scale. The "Slow–Fast" dimension was derived via ASR timestamps.
• Scaling: To label the entire LibriTTS-R corpus, they trained a VI Estimator (VIE). To handle the assumption that a speaker's VI is constant (which is false for mixed narration/expressive data), they employed a data augmentation strategy selecting acoustically similar utterances (based on pitch, energy, and WavLM embeddings) to assign manual labels, achieving a root-MSE of 0.68.

B. Proposed Architectures

The authors hypothesize that using a single reference utterance ($r$) for both speaker identity and VI conditioning causes leakage. They propose two methods to disentangle these factors:

1. VIC-dis (Disentanglement via Different Utterances):

   • Concept: During training, the model uses two different utterances from the same speaker.
   • Mechanism: One utterance ($r'$) provides the speaker identity, while a different utterance ($r$) from the same speaker provides the target VI (via the VIE).
   • Goal: This forces the speaker encoder to learn identity independently of the specific VI cues present in the reference audio, breaking the entanglement.

2. VIC-srf (Speaker-Reference-Free):

   • Concept: Eliminates the reference audio entirely for synthesis.
   • Mechanism: The synthesis is conditioned solely on the target VI vector ($v$). The speaker identity is generated by injecting Gaussian noise ($z$) into the control module, replacing the reference audio input.
   • Goal: By removing the reference audio, the model cannot "leak" the reference's impression, relying entirely on the explicit numerical target.

C. Baselines and Comparisons

The study compares these methods against:

• VIC-base: The original VIC architecture using a single reference for both identity and VI.
• Qwen3-TTS (QVD): A state-of-the-art LLM-based TTS using natural language (NL) prompts for voice design.

3. Key Contributions

1. LibriTTS-VI: The release of the first public corpus for Voice Impression Control, including manual annotations and estimated values for the full LibriTTS-R dataset.
2. Leakage Mitigation Strategies: The proposal of VIC-dis and VIC-srf, which structurally address the entanglement of speaker identity and voice impression.
3. Empirical Validation: Comprehensive objective and subjective evaluations demonstrating that disentangled training significantly improves controllability without sacrificing audio quality.
4. LLM Analysis: A critical evaluation of LLM-based TTS, revealing their inability to provide precise numerical control and their tendency to entangle text semantics with voice impressions.

4. Results

Objective Evaluation

• Controllability (VI-MSE): The proposed methods significantly reduced the Mean Squared Error between the target VI and the predicted VI.
  • VIC-srf achieved the best results, reducing 11-dimensional VI MSE from 0.61 (base) to 0.41.
  • Impression Leakage ($\Delta V$): Measured as the difference between cross-speaker and same-speaker error. VIC-srf reduced leakage from 0.22 (base) to 0.05, effectively eliminating structural leakage.
• Modulation Sensitivity: In experiments modulating VI from 1 to 7, VIC-srf showed the highest responsiveness (average slope of 0.199), compared to VIC-dis (0.159) and VIC-base (0.121).
• LLM Performance: The Qwen3-TTS model (QVD) showed poor speaker similarity (SECS < 0.63) and high VI-MSE (0.97), indicating that NL prompts lack the precision for fine-grained numerical control.

Subjective Evaluation

• Controllability: Human raters confirmed that VIC-dis and VIC-srf achieved lower MSE against target VIs compared to the baseline, particularly in multi-VI modulation tasks (reducing MSE from 1.15 to 0.92).
• Audio Quality: Mean Opinion Scores (MOS) remained comparable to the baseline (mostly 3.2–3.7), proving that improved controllability did not degrade synthesis quality.

5. Significance

This work establishes a new standard for controllable TTS by:

• Democratizing Research: Providing a public corpus allows the community to build upon VIC research without proprietary data barriers.
• Solving a Fundamental Flaw: It identifies and solves the "impression leakage" problem, which has been a bottleneck in zero-shot voice control.
• Bridging the Gap: It demonstrates that explicit numerical control via disentangled training is superior to the intuitive but imprecise natural language prompting used by current LLM-based TTS models.

The paper concludes that separating speaker identity from voice impression conditioning is essential for achieving high-fidelity, fine-grained voice control in TTS systems.