⚡ electrical engineering

Computational Narrative Understanding for Expressive Text-to-Speech

This paper introduces LibriQuote, a large-scale dataset of 5.3K hours of expressive character quotations from audiobooks enriched with contextual pseudo-labels, which demonstrates that fine-tuning or training on this data significantly enhances the expressivity and intelligibility of text-to-speech models.

Original authors: Gaspard Michel, Elena V. Epure, Christophe Cerisara

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gaspard Michel, Elena V. Epure, Christophe Cerisara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are listening to a story being read aloud. Sometimes the narrator speaks in a calm, flat voice, just telling you what happened next. But then, a character speaks! Suddenly, the voice changes. It might whisper a secret, shout in anger, or giggle with joy.

For a long time, computers trying to read stories aloud (Text-to-Speech or TTS) have been great at the "calm narrator" part but often sounded like robots when trying to act out the characters. They could say the words, but they couldn't capture the feeling.

This paper introduces a new project called LibriQuote to fix that. Here is the story of what they did, explained simply:

1. The Problem: The "Robot Narrator"

Think of existing computer voices like a very polite, monotone librarian reading a book. They are clear and accurate, but if the story says, "The dragon roared!" the computer just says, "The dragon roared," in the same boring voice. It lacks the drama.

The researchers realized that human-read audiobooks are full of these dramatic moments. When a human reads a novel, they naturally switch between the "narrator" voice and the "character" voices. They wanted to teach computers to do the same thing.

2. The Solution: A New "Acting School" Dataset

The team created a massive new library of audio called LibriQuote.

What is it? It's a collection of over 5,000 hours of audio, but they didn't just grab random sentences. They specifically cut out the parts where characters speak (the dialogue).
The Secret Sauce: They didn't just take the audio; they also looked at the text right next to it. In books, authors often write clues like "he whispered softly" or "she screamed loudly."
The Analogy: Imagine giving a student actor a script. Most TTS systems just give them the line: "I'm scared!"
- LibriQuote gives them the line plus the director's note: "I'm scared!" (whispered softly, trembling).
- This extra note is called a "pseudo-label." It tells the computer exactly how to say the words.

3. The Experiments: Teaching the Computer to Act

The researchers tried two different ways to teach computers using this new dataset:

Method A: The "Fine-Tuning" Approach (Polishing a Pro)
They took a smart computer voice that was already good at reading and gave it a crash course using LibriQuote.
- Result: The computer became much clearer and easier to understand, but it didn't become a great actor yet. It was like a good student who learned the rules but hadn't found their creative spark.
Method B: The "From Scratch" Approach (Building a New Actor)
They built a new computer voice from the ground up, training it only on these dramatic character lines.
- Result: This new voice was very expressive! It could whisper and shout. However, because it had less data to learn from, it sometimes stumbled over the words (it was less clear).
The Breakthrough: They found that a specific type of computer model (called F5-TTS) was like a sponge. When they used LibriQuote to train it, it became both clear and expressive. It learned to act and speak clearly at the same time.

4. The Big Test: The "TTS Olympics"

To see if their new dataset actually worked, they held a competition. They took the best computer voices in the world today and asked them to read the LibriQuote test set.

The Verdict: Most of the top computers still sounded a bit flat or got the emotions wrong. They might say "I'm angry!" but sound happy.
The Winner: One system called IndexTTS2 did the best job. It was able to guess the right emotion just by looking at the context (the story around the line). However, even the winner sometimes got it wrong, proving that this is still a very hard challenge.

5. Why This Matters

This paper is like handing a new, super-detailed script to the computer voice industry.

For Listeners: In the future, audiobooks read by AI could sound just as dramatic and emotional as a human actor, making stories more immersive.
For Researchers: They now have a "gym" (the dataset) to train AI to understand the difference between a narrator's voice and a character's voice.

The Catch (Limitations)

The researchers are honest about the flaws:

The Source: The audio they used came from volunteers (amateur readers), not Hollywood voice actors. So, sometimes the "human" examples weren't perfect either.
The Bias: They don't know the gender or accent of every reader in their dataset, so the AI might accidentally learn to sound like only one type of person.
The Future: They warn that while AI is getting better, we still need to be careful about using it to replace real human actors, as that could hurt jobs and lead to fake voices being used for bad things (like scams).

In a nutshell: The researchers built a giant library of "character voices" with special notes on how to act. They used it to teach computers to stop sounding like robots and start sounding like storytellers. It's a huge step forward, but the computer actors still have a lot of practice to do before they can beat the best human narrators.

1. Problem Statement

While recent Text-to-Speech (TTS) systems have achieved remarkable naturalness by scaling to large, multi-domain corpora (e.g., LibriSpeech, MLS), they often struggle with expressivity, particularly in fictional contexts.

The Gap: Existing audiobook datasets often treat speech as a monolithic stream, mixing neutral narration with expressive character dialogue. This creates complex, mixed pitch distributions that hinder models from learning distinct expressive styles.
The Challenge: Current TTS systems frequently fail to capture the nuanced prosody required for character impersonation (e.g., whispering, shouting, sarcasm) because they are trained on data where the "intent" of the speech is not explicitly labeled or separated from neutral narration.
The Goal: To create a dataset and evaluation framework that isolates character quotations from narrative text, providing rich contextual cues (speech verbs/adverbs) to train and benchmark TTS systems capable of high-fidelity, expressive storytelling.

2. Methodology

A. The LibriQuote Dataset

The authors introduced LibriQuote, a large-scale dataset derived from LibriVox audiobooks, specifically focusing on fictional works.

Data Composition:
- Total Size: 12,723 hours of narration and 5,359 hours of character quotations (approx. 5.3K hours of expressive speech).
- Speakers: 3,314 unique speakers across 2,991 books.
- Segmentation: Unlike standard datasets that use random 30-second cuts, LibriQuote uses narrative-aware segmentation to isolate direct speech (quotations) from neutral narration.
Contextual Pseudo-Labels:
- The dataset is augmented with contextual pseudo-labels extracted using Large Language Models (LLMs).
- For each quotation, the system extracts speech verbs (e.g., "whispered," "shouted") and adverbs (e.g., "softly," "angrily") from the surrounding narrative text. These serve as explicit instructions for the intended delivery.
- A High-Expressivity Subset ( $Q_f$ ) was created containing ~379 hours of quotations with non-empty adverb or specific verb labels, designed for data-efficient training.
Pipeline:
1. Filter LibriVox for fiction.
2. Align audio with text using BookNLP for quotation detection.
3. Segment audio into quotations and narration.
4. Use LLMs (specifically Phi-4) to extract speech descriptors from the context window.

B. Training Experiments

The authors evaluated two main approaches using SparkTTS (autoregressive) and F5-TTS (flow-matching/non-autoregressive):

Fine-tuning: Adapting pre-trained models on LibriQuote subsets ( $Q$ or $Q_f$ ).
Training from Scratch: Training models from the ground up on LibriQuote data.
Context Conditioning: Experimenting with replacing standard text input with the full narrative context (including the extracted verbs/adverbs) to guide the model.

C. Evaluation Framework

Benchmark: LibriQuote-test (7.4 hours, 15 speakers) serves as a challenging testbed for expressive synthesis.
Metrics:
- Objective: Word Error Rate (WER), Speaker Similarity (SIM), Emotion Similarity (E-Sim).
- Contextual: ContextMOS (1-5 scale) and Win-Rate using LALM-as-a-Judge (Gemini-2.5 Pro) to evaluate how well the speech matches the narrative context (e.g., does "whispered" sound like a whisper?).
- Subjective: Human Mean Opinion Score (MOS) and Comparative MOS (CMOS).

3. Key Contributions

LibriQuote Dataset: The first large-scale dataset explicitly separating character quotations from narration, enriched with pseudo-labels for speech intent (verbs/adverbs).
Narrative-Aware Segmentation: A novel approach to dataset construction that avoids the "pitch distribution mixing" problem found in standard audiobook corpora.
Empirical Insights on TTS Architectures:
- Flow-Matching vs. Autoregressive: Fine-tuning a flow-matching model (F5-TTS) on expressive data yielded simultaneous gains in expressivity and intelligibility. Conversely, fine-tuning an autoregressive model (SparkTTS) improved intelligibility but did not significantly boost expressivity without sacrificing performance.
- Training from Scratch: Training autoregressive models from scratch on quotations increased expressivity but reduced intelligibility due to data scarcity; combining narration and quotations ( $N \cup Q$ ) mitigated this.
Benchmarking Results: Established LibriQuote-test as a rigorous benchmark where current SOTA models (SparkTTS, F5-TTS, MaskGCT) show significant variability, with IndexTTS2 (which decouples emotion prediction from synthesis) performing best in matching ground-truth expressivity.

4. Results

Intelligibility: Fine-tuning SparkTTS on LibriQuote significantly improved WER on both in-domain and out-of-domain datasets (LibriSpeech-PC, SeedTTS), acting as a "data curriculum."
Expressivity:
- F5-TTS: Fine-tuning on the expressive subset ( $Q_f$ ) increased ContextMOS by 0.4 points compared to the baseline, proving flow-matching is better suited for capturing expressive nuances in this domain.
- IndexTTS2: Achieved a 54% Win-Rate against ground truth (beating the human baseline in some metrics) and positive CMOS, indicating it was rated as more expressive than the amateur human narrators in the test set.
Human vs. AI: Surprisingly, the amateur human narrators in LibriQuote-test often failed to convey sufficient emotion (low ContextMOS for ground truth), yet TTS systems often failed even more frequently or produced "wrong" emotions. However, IndexTTS2-Context (using context for emotion prediction) significantly reduced these errors.

5. Significance and Future Directions

Advancing Expressive TTS: The paper demonstrates that narrative understanding (separating "what" is said from "how" it is said) is critical for the next generation of TTS.
Digital Humanities: The dataset provides 3 million character quotations with intent labels, offering new avenues for analyzing prosody in literature (e.g., gender interpretation of characters).
Ethical Considerations: The authors highlight the risks of voice spoofing and the displacement of voice actors, urging responsible deployment. They also note the dataset's lack of explicit gender/accent metadata, which could lead to bias in commercial applications.
Future Work: The authors suggest LibriQuote is ideal for training Neural Audio Codecs for higher-fidelity reconstruction and call for further research into how narrative context conditioning can be better integrated into end-to-end TTS systems.

In conclusion, LibriQuote bridges the gap between computational narrative understanding and speech synthesis, proving that isolating and labeling character dialogue with narrative context is a powerful strategy for training highly expressive TTS systems.