⚡ electrical engineering

Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation

This paper demonstrates that Text-to-Speech (TTS) evaluation requires context-aware metrics rather than a one-size-fits-all approach, as the appropriateness of speech varies significantly across different domains and often conflicts with traditional naturalness scores.

Original authors: Dominika Woszczyk, Andreas Triantafyllopoulos, Jura Miniota, Éva Székely, Bjoern Schuller

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dominika Woszczyk, Andreas Triantafyllopoulos, Jura Miniota, Éva Székely, Bjoern Schuller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a voice actor. If you need someone to read a bedtime story to a child, you want a calm, steady, and soothing voice. But if you need someone to play a frantic character in a video game or to sound like a friend chatting over coffee, that same calm voice would feel weird and out of place.

This paper argues that Text-to-Speech (TTS) technology is facing the same problem. For years, developers have tried to make computer voices sound as "natural" (human-like) as possible. But this study shows that being "natural" isn't always the same as being "appropriate."

Here is a breakdown of what the researchers found, using simple analogies:

1. The "Swiss Army Knife" Myth

For a long time, the goal was to build one perfect TTS system that could do everything well—like a Swiss Army knife that is supposed to be the best knife, screwdriver, and scissors all at once.

The researchers tested five of the smartest, most advanced TTS systems available today. They asked human listeners to rate them in five different "roles":

The Reader: Reading a book aloud.
The Assistant: Acting like a helpful AI (like Siri or Alexa).
The Actor: Acting out a dramatic scene.
The Character: Playing an animated cartoon character.
The Spontaneous Speaker: Having a casual, unscripted chat.

The Result: No single system won at everything.

Some systems were great at reading books but sounded robotic and boring when trying to have a casual chat.
Other systems were amazing at sounding like a real person having a messy, spontaneous conversation, but they sounded too raw and unpolished to be a helpful assistant.

The Analogy: It's like trying to wear a tuxedo to a muddy soccer game. The tuxedo is "high quality" and "formal," but it is the wrong tool for the job. Similarly, a TTS system optimized for a formal reading might fail miserably at a casual conversation.

2. The "Human-Like" Trap

The study found a surprising twist: Just because a voice sounds human doesn't mean it's the right voice for the job.

The "Robotic" Assistant: When people listened to an AI assistant, they actually preferred a voice that sounded slightly less human and more steady. They didn't want a chatty, emotional friend; they wanted a reliable tool.
The "Too Real" Character: When listening to an animated character, a voice that sounded too much like a real, imperfect human (with pauses, breaths, and stutters) felt wrong. Listeners wanted the character to sound stylized, not like a real person caught off guard.

The Takeaway: "Naturalness" is a tricky metric. Sometimes, sounding less human is actually the most "appropriate" choice for a specific task.

3. The "Imperfection" Paradox

The researchers looked at the actual sound waves to see what made a voice feel right. They found that different roles need different kinds of "flaws."

For Casual Chat: Listeners actually liked hearing small imperfections like "um," "uh," and slight voice cracks. These made the AI sound like a real person thinking on their feet.
For Reading or Assistants: Those same imperfections made the voice sound unprofessional or broken.

The Analogy: Think of a jazz musician improvising. A few wrong notes or a messy rhythm might make the performance feel "alive" and "spontaneous." But if a news anchor reading the evening news made those same mistakes, it would be a disaster. The "mistake" is only bad if it's in the wrong context.

4. The Broken Ruler

Finally, the paper points out that the tools we use to measure TTS quality are often broken.

Most automatic scoring systems (the "rulers" developers use to check their work) are designed to reward "clean" and "perfect" audio.

The Problem: These rulers penalize the very things that make a voice sound good in a casual or dramatic setting (like pauses, emotion, and variation).
The Result: A TTS system might get a high score from a computer because it sounds "clean," but a human listener might hate it because it sounds boring or wrong for the situation.

Summary

The main message of this paper is that we need to stop asking "Does this sound like a human?" and start asking "Does this sound right for the job?"

There is no single "best" voice. Just as you wouldn't wear a swimsuit to a business meeting or a suit to the beach, you shouldn't use the same TTS settings for a video game character that you use for a navigation app. To make better AI voices, we need to evaluate them based on the specific role they are playing, not just how "human" they sound in a vacuum.

Technical Summary: Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation

Problem Statement

Text-to-speech (TTS) evaluation has historically prioritized "naturalness," often measured via Mean Opinion Scores (MOS). However, naturalness is increasingly recognized as an ill-defined, multi-dimensional concept that fails to capture contextual suitability. A voice optimized for an audiobook may be inappropriate for an emergency hotline or a conversational agent. While recent work has introduced "appropriateness" (whether speech fits its communicative purpose) and persona-based benchmarks, the systematic impact of the target domain on appropriateness judgments remains under-explored. Current "task-agnostic" metrics often ignore situational needs, leading to scenarios where a model with high accuracy or naturalness is perceived as inappropriate for its intended real-world use.

Methodology

The authors conducted a perceptual study to evaluate how perceived naturalness (framed as "human-likeness") and appropriateness (framed as "convincingness") vary across different downstream applications.

Participants and Setup: 150 native English speakers were recruited via Prolific and split into six sessions. They evaluated 30 curated sentences across five TTS systems and ground truth (GT) samples.
Domains (Personas): The study assessed five distinct use cases:
1. Reader: Descriptive narration (LibriQuote).
2. Spontaneous Speaker: Informal, disfluent conversation (MSP-Podcast).
3. Actor: Emotional dialogue (MELD).
4. Animated Character: Stylized performance (AnimeVox).
5. AI Assistant: Informative, conversational interaction.
Systems Evaluated: Five state-of-the-art TTS systems were selected to cover a spectrum of expressivity and spontaneity:
- Kokoro: Lightweight, prosodically consistent, low spontaneity.
- Gemini TTS: Highly expressive and stylized.
- Kyutai-TTS: Trained on raw conversational data, high spontaneity.
- GPT-4o-mini-tts: Balanced profile with medium-high spontaneity.
- ElevenLabs: Professional delivery, medium spontaneity.
Metrics: Participants rated "human-likeness" and "convincingness" on a 5-point Likert scale. The study also analyzed acoustic features (rhythm, expressivity, voice quality) and correlated them with automatic evaluation metrics (e.g., UTMOS, DNSMOS, WER, embedding distances).

Key Contributions

Cross-Domain Perceptual Analysis: The study systematically measures how naturalness and appropriateness shift across different settings, demonstrating that a system's performance is not uniform.
Human-Likeness Paradox: The authors analyze the relationship between human-likeness and domain suitability, showing that high naturalness does not guarantee appropriateness, and vice versa.
Domain-Specific Metric Profiling: The work profiles how common automatic metrics correlate with perceived appropriateness, revealing that "one-size-fits-all" metrics often fail in expressive or spontaneous domains.

Key Results

1. Appropriateness is Domain-Dependent

Appropriateness varies significantly across domains and is not strictly tied to naturalness.

Reading/Assistant Roles: Most systems achieved high scores here. Notably, Kokoro performed well in these roles despite lower naturalness, suggesting listeners may expect AI assistants to sound somewhat robotic or stable.
Expressive/Spontaneous Roles: Kyutai-TTS excelled in spontaneous conversation but scored poorly as an assistant or animated character, sounding "too raw." Conversely, systems like ElevenLabs and Gemini scored well in acting domains but struggled with spontaneous conversation.
Subjectivity: Low inter-rater agreement (Krippendorff's $\alpha$ = 0.2 for TTS) indicates that appropriateness is highly subjective and influenced by individual listener expectations.

2. Decoupling of Human-Likeness and Appropriateness

Correlation analysis revealed that human-likeness and appropriateness are distinct dimensions:

Positive Correlation: Observed in Actor, Spontaneous, and Reader domains.
Negative Correlation: Observed in the Assistant domain ( $\rho = -0.44$ ). Systems deemed highly appropriate for assistants (e.g., Kokoro) often received lower naturalness scores.
Near-Zero Correlation: Observed in the Animated Character domain.
Conclusion: High naturalness does not guarantee suitability for specific tasks (e.g., Kyutai-TTS was natural but inappropriate for an assistant role).

3. Impact of Speech Tasks on Naturalness

Stylization Penalties: Highly stylized delivery (e.g., AnimeVox) was penalized in naturalness scores regardless of the task.
Task Difficulty vs. Model Capability: Narration is often considered easier to synthesize, yet Kyutai-TTS performed more naturally on conversational tasks than on reading, suggesting naturalness scores reflect a model's ability to handle specific styles rather than inherent task difficulty.
Ground Truth Variance: Even human ground truth ratings varied by domain, with conversational samples preferred over narration, potentially due to accents or voice maturity.

4. Acoustic Features and Appropriateness

Animated Characters: Correlated strongly with articulation rate variability and rhythm fluctuations ( $\rho \approx 0.4$ ).
Readers: Preferred steady, controlled rhythmic delivery (negative correlation with variability).
Assistants: Preferred stability and neutrality; negatively correlated with $f0$ range and valence (emotional charge).
Spontaneous Speech: Uniquely showed positive correlations with "human imperfections" like jitter ( $\rho = 0.34$ ) and creakiness ( $H1-H2$ ), indicating these features are expected in casual interaction.

5. Limitations of Automatic Metrics

Quality Estimators: Metrics like UTMOS and DNSMOS showed negative correlations with appropriateness for Actor and Spontaneous styles. They penalize the disfluencies and dynamic ranges essential for expressive speech.
Embedding Metrics: AudioBox and AutoPCP were effective for Assistant and Reader styles but failed to capture nuances in human-likeness, showing inverse correlations.
WavLM: Negatively correlated with Assistant appropriateness but positively with human-likeness, suggesting it represents neutral speech better than task-specific suitability.

Significance and Claims

The paper argues that TTS performance is not "solved" but is fundamentally dependent on the target domain. The authors claim that:

Context is King: Evaluation must be context-aware. A single naturalness score cannot predict suitability across diverse applications.
Distinct Dimensions: "Human-likeness" and "appropriateness" are not interchangeable. Optimizing for one can degrade performance in the other.
Metric Blind Spots: Current "one-size-fits-all" evaluation metrics have blind spots in expressive domains, often penalizing the very features (disfluency, variability) required for natural interaction in those contexts.
Future Direction: The field should shift from asking if an utterance "sounds like a human" to whether it "sounds right" for its specific purpose. Future TTS systems and voices should be designed with specific delivery styles in mind, rather than as universal solutions.