💬 NLP

S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature

This paper introduces S-VoCAL, the first dataset and evaluation framework designed to assess the inference of fictional character voice attributes from literature, demonstrating through a Retrieval-Augmented Generation pipeline that while attributes like age and gender can be reliably inferred, others such as origin and physical health remain challenging.

Original authors: Abigail Berthe-Pardo, Gaspard Michel, Elena V. Epure, Christophe Cerisara

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abigail Berthe-Pardo, Gaspard Michel, Elena V. Epure, Christophe Cerisara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are listening to a magical audiobook where the narrator doesn't just read the story; they become every character. When a grumpy old king speaks, the voice is deep and raspy. When a shy teenager whispers, the voice is high and trembling. When a dragon roars, the voice is guttural and loud.

This is the dream of "synthetic audiobooks." But right now, computers are like actors who can only play themselves. They struggle to figure out who a character is just by reading the text, so they can't change their voice to match.

This paper introduces S-VoCAL, a new tool designed to teach computers how to "listen" to a book and guess the right voice for every character.

Here is the breakdown of how it works, using some everyday analogies:

1. The Problem: The "Needle in a Haystack"

Imagine you are trying to figure out a character's age just by reading a 500-page novel. The author might mention the character is "old" in Chapter 3, but then talk about their "frail knees" in Chapter 40, and never mention their birth year again.

For a human, this is easy. For a computer, it's like finding a specific needle in a massive haystack of text. The computer needs to connect dots that are miles apart in the story.

2. The Solution: S-VoCAL (The "Character ID Card" Dataset)

The researchers created a massive database called S-VoCAL. Think of this as a giant library of "Character ID Cards."

The Content: They looked at 192 classic books (like Pride and Prejudice or Moby Dick) and created profiles for 952 characters.
The 8 Clues: They didn't just write a paragraph about the character. They focused on 8 specific clues that change how a voice sounds:
1. Age: (Child, Teen, Adult, Senior) – Affects pitch.
2. Gender: (Male/Female) – Affects resonance.
3. Type: (Human, Elf, Dragon) – Affects the whole sound.
4. Origin: (Where they are from) – Affects accent.
5. Residence: (Where they live now) – Affects dialect.
6. Occupation: (Job) – Affects speech style.
7. Languages: (What they speak) – Affects pronunciation.
8. Health: (Sick, injured, healthy) – Affects breathiness or stability.

3. The Challenge: Grading the Computer's Homework

How do you grade a computer's guess?

Easy Questions: If the computer guesses "Male" and the answer is "Male," that's a perfect score.
Hard Questions: If the computer guesses "Stuttering" and the answer is "Speech impediment," is that right? If it guesses "New York" but the answer is "USA," is that right?

The researchers built a special grading system (Evaluation Framework).

For simple things (like Gender), they use a standard "Right or Wrong" score.
For complex things (like Health or Origin), they use a "Smart AI Teacher" (a Large Language Model) to read the computer's answer and the real answer and give a score from 0 to 1 based on how similar they feel, not just how they look. It's like a teacher saying, "You didn't get the exact word, but you got the meaning right, so you get an A-."

4. The Experiment: The "RAG" Detective

To test this, they built a simple detective system called RAG (Retrieval-Augmented Generation).

Step 1 (The Search): The system scans the book, looking for every sentence that mentions a character. It's like a detective gathering all the clues about a suspect.
Step 2 (The Guess): It feeds those clues to a smart AI (the "Brain") and asks, "Based on these clues, what is this character's age and health?"
Step 3 (The Cleanup): The AI sometimes talks too much or makes formatting errors, so the system cleans up the answer to make it neat.

5. The Results: The "Good News" and the "Bad News"

The experiment showed a clear split in performance:

The Superstars (Closed-Class Attributes): The computer was amazing at guessing Gender, Type (Human vs. Non-human), and Age. It got these right almost every time. It's like a detective who is great at spotting if someone is wearing a red hat or a blue hat.
The Strugglers (Open-Class Attributes): The computer got confused by Health, Origin, and Occupation.
- Why? Because these are harder to find. A book might say a character is "sick" but never say what sickness. Or it might say they are from "the North" without naming the country.
- The computer often guessed "Unknown" or made wild guesses. It's like a detective trying to guess a suspect's favorite food when the text only says "they were hungry."

Why Does This Matter?

This isn't just about making cool audiobooks. It's about immersion.

Right now, if you listen to a synthetic audiobook, the narrator sounds the same for the villain, the hero, and the sidekick. It's like a play where everyone wears the same mask.

With tools like S-VoCAL, we are teaching computers to read between the lines. Soon, an AI narrator might hear a character cough in Chapter 10 and automatically make their voice sound raspy in Chapter 50, creating a truly magical, living story experience.

In short: The researchers built a training manual and a test for computers to learn how to guess a character's voice from a book. They found that computers are great at the basics (Age/Gender) but still need to study harder for the complex details (Health/Origin).

1. Problem Statement

The rise of Text-to-Speech (TTS) technology has enabled the creation of synthetic audiobooks with increasing naturalness. However, a critical gap remains in synthetic character impersonation. Current systems struggle to assign plausible, distinct voices to fictional characters based on their descriptions in literature.

To generate a character-specific voice, a system must infer voice-relevant attributes (e.g., age, gender, origin, physical health) from the text. This task is challenging because:

Implicit Information: Character details are often not explicitly stated but scattered across distant narrative sections or implied through context.
Long-Context Reasoning: Extracting these traits requires reasoning over entire books, a known difficulty for current NLP models.
Lack of Benchmarks: Existing datasets focus on general character summaries or high-level literary analysis, lacking specific, structured attributes grounded in sociophonetics (the study of how social factors influence speech). There is no standardized framework to evaluate how well a system can extract these specific voice-driving traits.

2. Methodology

The authors propose S-VoCAL (Speaking Voice Character Attributes in Literature), comprising a dataset and a novel evaluation framework.

A. The S-VoCAL Dataset

Source: 192 unique novels from Project Gutenberg (English or translated world literature published before 1940).
Scale: 952 character-book pairs.
Attributes: 8 voice-relevant attributes derived from sociophonetic studies:
1. Age: Categorical (Child, Teenager, Adult, Senior).
2. Gender: Categorical (Male, Female).
3. Type: Categorical (Human, Non-human).
4. Origin: List (Country/Region of origin).
5. Residence: List (Current location).
6. Occupation: List (Profession/Role).
7. Spoken Languages: List.
8. Physical Health: Text (Medical conditions affecting voice).
Construction:
- Primary data is extracted from Wikidata entries for characters, mapped to the target attributes.
- Manual Annotation: The Age attribute was significantly underrepresented in Wikidata. The authors manually annotated 319 additional instances (totaling 359) using guidelines based on explicit mentions, relative cues, and behavioral indicators. Inter-annotator agreement was measured using Cohen's Kappa and soft F1-scores.

B. Evaluation Framework

The framework addresses the heterogeneity of the attributes (categorical vs. open-text):

Closed/Semi-Closed Attributes (Age, Gender, Type, Spoken Languages): Evaluated using Weighted F1-scores.
- Soft F1: Used for Age to penalize adjacent errors (e.g., "Teenager" vs. "Adult") less severely than distant errors.
Open-Class Attributes (Origin, Residence, Occupation, Physical Health): Evaluated using a Semantic Similarity Metric.
- Problem: Standard metrics like BERTScore perform poorly on short, entity-based lists without context.
- Solution: The authors use Qwen-3 8b Embeddings (an instruction-aware model) to generate attribute-conditioned representations. They compute the cosine similarity between the model's prediction and the ground truth.
- Human-Aligned Score (HAS): To make raw similarity scores interpretable, the authors trained an isotonic regression model to map cosine similarities to a 0–1 scale based on human judgments (where 0 is incorrect, 1 is perfect).

C. Baseline Experiment (RAG Pipeline)

To demonstrate the dataset's utility, the authors implemented a Retrieval-Augmented Generation (RAG) pipeline:

Retrieval: Using E5-large embeddings, the system retrieves the top 10 most relevant 200-word passages for a specific character and attribute query (e.g., "Is John a child or an adult?").
Inference: A Large Language Model (LLM) processes these passages to infer the attribute values.
Post-processing: Cleaning steps remove reasoning traces, fix JSON formatting, and normalize lists.
Models Tested: Qwen3-8b and Phi-4-14b.

3. Key Contributions

S-VoCAL Dataset: The first dataset dedicated to voice-relevant character attributes in literature, covering 952 characters across 192 books with 8 distinct attributes grounded in phonetic research.
Specialized Evaluation Framework: A hybrid evaluation system that combines standard classification metrics for categorical data and a novel instruction-aware embedding similarity metric (Qwen-3 based) for open-text attributes, validated against human judgment.
Empirical Analysis: A comprehensive evaluation of RAG pipelines on this task, establishing baselines for future research in synthetic audiobook narration.

4. Results

The RAG pipeline was evaluated against a majority-value baseline and across different LLM backbones:

Closed-Class Attributes (High Performance):
- The system performed exceptionally well on Gender (Weighted F1 $\approx$ 0.99) and Type (Human/Non-human).
- Age showed strong performance (Weighted F1 $\approx$ 0.78–0.80), with Soft F1 scores reaching $\approx$ 0.94, indicating most errors were minor (adjacent categories).
- Spoken Languages showed moderate success (Micro F1 $\approx$ 0.71–0.75), though some omissions occurred.
Open-Class Attributes (Mixed to Low Performance):
- Type (Non-human): High performance (mHAS $\approx$ 0.78).
- Occupation & Residence: Partial correctness (mHAS $\approx$ 0.50), often missing specific details or providing broader categories.
- Origin: Moderate performance (mHAS $\approx$ 0.42), often correct at the continent level but incorrect at the country level.
- Physical Health: Poor performance (mHAS $\approx$ 0.15). The system frequently failed to detect whether a character had a health condition or not.
Model Comparison: No single LLM (Qwen3 vs. Phi4) consistently outperformed the other across all open attributes. The errors appeared independent, suggesting different "memorization" patterns of the source books.

5. Significance and Future Work

Advancing Synthetic Audiobooks: S-VoCAL provides the necessary infrastructure to move TTS from generic narration to immersive, character-driven storytelling, where voices are dynamically adapted to character identities.
Benchmarking Long-Context Reasoning: The dataset highlights the current limitations of LLMs in extracting implicit, distributed information from long texts, specifically for open-ended attributes like health and origin.
Limitations:
- The dataset relies on public domain books (Project Gutenberg), which may be over-represented in LLM training data, potentially inflating performance metrics.
- Attributes are currently static; future work needs to address dynamic attributes (e.g., a character's health deteriorating over the story).
- The dataset has demographic imbalances (predominantly adult, male, human characters).

In conclusion, S-VoCAL establishes a critical benchmark for the intersection of Digital Humanities and Speech Synthesis, proving that while current RAG systems can reliably extract basic demographic traits, significant challenges remain in inferring complex, open-ended voice characteristics from literature.