⚡ electrical engineering

WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

The paper introduces WordVoice, a unified framework that addresses the coarse-grained control limitations of LLM-based TTS by leveraging a massive, five-dimensionally annotated dataset and a novel bound-token mechanism to enable explicit, decoupled, and precise word-level manipulation of acoustic attributes like duration, pitch, and energy.

Original authors: Sihang Nie, Jinxin Ji, Xiaofen Xing, Deyi Tuo, Chengbin Jin, Jialong Mai, Xiangmin Xu

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sihang Nie, Jinxin Ji, Xiaofen Xing, Deyi Tuo, Chengbin Jin, Jialong Mai, Xiangmin Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are directing a movie, but the actors (the AI voice) are improvising. They sound great and natural, but you can't tell them exactly how to say a specific word. You can't say, "Pause for a split second here," or "Make this word sound angry," or "Sing this note higher." You have to hope the AI guesses your intent correctly.

This paper introduces WordVoice, a new system that gives directors (users) a remote control for every single word in a sentence. It turns the AI from an improvising actor into a script-perfect performer who follows your exact instructions.

Here is how they did it, explained through simple analogies:

1. The Problem: The "Fuzzy" Remote

Current AI voice systems are like a remote control with only a few big buttons: "Happy," "Sad," or "Fast." If you want to change the pitch of just one word in a long sentence, the remote doesn't work. The AI treats the whole sentence as one big blob of sound, making it impossible to tweak individual parts precisely.

2. The Solution: A Massive "Instruction Manual" (WordVoice-5A)

To teach the AI how to listen to these tiny, specific instructions, the researchers first had to build a massive library of examples.

The Analogy: Imagine trying to teach a student to play the piano by only showing them whole songs. They might learn the general vibe, but they won't know how to hit a specific note with a specific force.
What they did: The team created WordVoice-5A, a dataset of 4,700 hours of speech (a huge library!). They didn't just label the whole sentence; they went word-by-word and wrote down five specific "instructions" for every single word:
1. Duration: How long the word lasts.
2. Boundary: Is there a pause before or after it?
3. Energy: How loud or emphasized it is.
4. Pitch: How high or low the voice is.
5. Tone: The shape of the melody (rising, falling, flat, etc.).

They used a strict, linguist-guided process to ensure these labels were perfect, creating the ultimate "instruction manual" for the AI.

3. The Brain: The "Acoustic Planner" (WordVoice-LLM)

The core of their system is a Large Language Model (LLM), which is the "brain" that generates the speech.

The Old Way: The brain would just guess the next sound, hoping it sounded right.
The New Way (WordVoice): The researchers added a special "planning token" (think of it as a sticky note). Before the AI speaks a word, it stops and writes a plan on that sticky note.
- Example: "For the word 'Hello', I will plan: 0.5 seconds long, high energy, rising pitch."
- Once the plan is written, the AI speaks the word exactly according to that plan.
The Magic: Users can either let the AI write its own plan (Free Mode) or write the plan themselves (Control Mode). If you want a specific word to sound dramatic, you just write "High Energy" on the sticky note, and the AI obeys.

4. The Voice Box: The "Fine-Tuner" (WordVoice-FM)

Even with a perfect plan, the AI's "voice box" (the part that turns digital notes into actual sound waves) can sometimes lose detail, like a low-resolution photo.

The Analogy: Imagine you have a perfect architectural blueprint (the plan), but the builder uses rough bricks. The shape is right, but the texture is off.
What they did: They added a "Fine-Tuner" module at the end. This module looks at the blueprint and the rough bricks, then smooths them out to ensure the final sound wave matches the plan perfectly. It bridges the gap between the digital "notes" and the continuous "sound," ensuring the energy and pitch are exactly what you asked for.

5. The Results: Total Control

The team tested this system against other top AI voice tools.

The Verdict: WordVoice allows users to change the duration, loudness, pitch, and tone of specific words with extreme precision.
The Trade-off: The paper notes a tiny trade-off. Because the AI is focusing so hard on following your specific instructions for individual words, the overall "natural flow" of the voice is slightly less perfect than if it were just improvising. However, the gain in control is massive.
The Proof: When they asked the AI to change just one word's tone or length, it did so with high accuracy, whereas other systems failed to make those specific changes without messing up the whole sentence.

Summary

WordVoice is like giving a voice actor a script where every single word has a specific direction attached to it (e.g., "Say this word loudly," "Say this word slowly"). It solves the problem of "coarse control" by breaking speech down into tiny, manageable pieces and giving the user a remote control for each one, all while keeping the voice sounding natural and human.

Technical Summary: WordVoice

1. Problem Statement

Recent Large Language Model (LLM)-based Text-to-Speech (TTS) systems have achieved remarkable naturalness but rely predominantly on implicit, end-to-end generation paradigms. This architecture results in coarse-grained control, creating a critical bottleneck for applications requiring precise, deterministic interventions at the word level, such as audiobook narration and video dubbing. Current models struggle to explicitly manipulate specific acoustic attributes (e.g., duration, energy, pitch, tone) for individual words without compromising global stability. Furthermore, this limitation is exacerbated by the scarcity of large-scale, fine-grained annotated datasets and the architectural difficulty of integrating multi-dimensional control signals into discrete autoregressive generation.

2. Methodology

The authors propose WordVoice, a unified framework that transforms implicit generation into an explicit, highly controllable paradigm through a dual-stage architecture and a massive new dataset.

2.1. Data Foundation: WordVoice-5A

To address the data scarcity, the authors constructed WordVoice-5A, a bilingual (Chinese and English) dataset comprising approximately 4,700 hours of speech.

Annotation Pipeline: A rigorous, linguistically-guided pipeline was developed involving:
- Timestamp Optimization: Dual-model alignment (MFA and Qwen3FA) with loudness-based optimization and consistency checks to ensure high-precision word boundaries.
- Temporal Attributes: Extraction of duration and discretization of boundaries into five levels (from continuous phonation to period-level pauses).
- Acoustic/Prosodic Attributes: Extraction of energy (top 50% frame average), pitch (central 80% F0 average), and tone. Tone is modeled via expert-guided morphological regression, mapping continuous F0 contours into seven universal prosodic categories (flat, rise, strong rise, fall, strong fall, peak, valley).

2.2. Architectural Framework

WordVoice operates on a backbone derived from CosyVoice3, refining sentence-level generation into a word-level "acoustic planning" process.

Stage 1: WordVoice-LLM (Macro-Prosodic Planning)

The authors introduce a bound-token mechanism (⟨b⟩) within the autoregressive LLM to restructure the causal generation process.

Acoustic Planning: Upon predicting the bound token for a word, the model's hidden state is routed to a lightweight Word Decoder to predict five acoustic attributes: duration, boundary, energy, pitch, and tone.
Explicit Control: These attributes are embedded into a style token and concatenated with the semantic embedding to form a unified condition. This enables two inference modes:
- Free Mode: The model autonomously plans prosody.
- Control Mode: Users can bypass the decoder and manually specify attribute values for specific words, achieving zero-shot, deterministic control without altering LLM weights.

Stage 2: WordVoice-FM (Micro-Acoustic Modulation)

To bridge the resolution gap between discrete tokens and continuous waveforms, a fine-grained acoustic modulation module is integrated into the Flow Matching (FM) stage.

Upsampling: Word-level style tokens are upsampled to the frame level based on predicted durations and pauses.
Modulation: These frame-level style tokens are injected into the FM backbone via a dedicated module that generates scale ( $\gamma_t$ ) and shift ( $\beta_t$ ) parameters. This explicitly guides the waveform synthesis to adhere to the planned energy, pitch, and temporal boundaries, compensating for acoustic details lost during quantization.

3. Key Contributions

Data Foundation: The creation and open-sourcing of WordVoice-5A, a 4.7k-hour bilingual dataset with rigorous five-dimensional word-level annotations and a linguistically-guided pipeline.
Architectural Innovation: The proposal of the WordVoice framework, which combines an explicit "acoustic planning" process via bound-tokens in the LLM with a fine-grained style modulation module in the Flow Matching stage.
Word-Level Control: The demonstration of highly precise, decoupled control across multiple acoustic dimensions (duration, boundary, energy, pitch, tone) at the word level, maintaining competitive zero-shot synthesis stability.

4. Experimental Results

Experiments were conducted on the WordVoice-5A test sets (Chinese and English) using CosyVoice3 as a baseline.

Subjective Evaluation: WordVoice-Control achieved the highest scores in Naturalness (N-MOS) and Word Style Control (Ctrl-MOS), with statistically significant improvements ( $p < 0.01$ ) in controllability over the baseline. While Speaker Similarity (Spk-MOS) saw a marginal trade-off due to localized variations, the overall expressiveness was significantly enhanced.
Objective Evaluation:
- Control Precision: In Control mode, WordVoice drastically reduced Mean Absolute Error (MAE) for continuous attributes (Duration, Energy, Pitch) and Error Rate (ER) for discrete attributes (Boundary, Tone) compared to baselines.
- Temporal Alignment: WordVoice outperformed MagicTTS (a strong baseline for temporal control) in both duration and boundary metrics, demonstrating superior stability when manipulating multiple attributes simultaneously.
- Decoupling: Ablation studies confirmed that the control mechanism is largely decoupled; constraining a single attribute significantly reduced its specific error with minimal impact on others, except for tone, which showed slight entanglement due to its dynamic nature.
Ablation Studies: Removing the WordVoice-FM module resulted in significant degradation in energy and pitch accuracy, confirming its necessity for compensating quantization losses. Conversely, temporal attributes were primarily modeled in the LLM stage.

5. Significance and Claims

The paper claims that WordVoice addresses a fundamental limitation in current LLM-based TTS: the inability to perform high-precision, fine-grained acoustic control at the word level. By transforming implicit generation into an explicit, interpretable process, the framework enables:

Deterministic Interventions: Creators can meticulously manipulate the duration, emphasis, and intonation of specific words to convey nuanced emotions or achieve strict cross-modal alignment (e.g., lip-sync).
Unified Multi-Dimensional Control: Unlike previous approaches that were often unidimensional (e.g., only duration or only emotion), WordVoice simultaneously controls five acoustic dimensions.
Future Potential: The authors suggest this explicit planning mechanism could serve as an "acoustic chain-of-thought" for highly expressive spoken dialogue systems and provide interpretable generation for instruction-based TTS.

The work positions itself not merely as an incremental improvement in naturalness, but as a structural shift toward explicit, controllable, and decoupled speech synthesis.