💬 NLP

PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech

This paper introduces PSP (Phoneme Substitution Profile), an interpretable, per-dimension benchmark that evaluates Indic Text-to-Speech systems on specific accent features like retroflexion and aspiration, revealing that current commercial and open-source leaders often fail to capture these phonological nuances despite high standard intelligibility scores.

Original authors: Venkata Pushpak Teja Menta

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Venkata Pushpak Teja Menta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot that can read text out loud. If you ask it to read a sentence in Hindi, Telugu, or Tamil, it might get every single word "correct" according to a spell-checker. It won't misspell a single word. But if a native speaker from that region listens to it, they might say, "That sounds like a foreigner trying to speak my language." The robot is getting the spelling right, but the accent is wrong.

This paper introduces a new tool called PSP (Phoneme Substitution Profile) to measure exactly how that accent is wrong, rather than just saying "it sounds bad" or "it sounds good."

Here is a breakdown of the paper's ideas using simple analogies:

1. The Problem: The "Perfect Spelling, Wrong Accent" Trap

Current ways of testing speech robots (Text-to-Speech or TTS) are like a spelling bee. They check:

Intelligibility: Did it say the right words? (Word Error Rate).
Naturalness: Does it sound human-ish? (MOS scores).

The Flaw: A robot can get 100% on the spelling bee but still sound like a tourist. In Indian languages, there are specific "flavor" sounds that are easy for non-natives to mess up:

Retroflex sounds: Tongue curls back (like a 't' or 'd' made at the roof of the mouth).
Aspiration: A puff of air (like a soft 'h' after a consonant).
Vowel length: Holding a sound longer (like "cat" vs. "caat").
The "Zha" sound: A unique Tamil sound that doesn't exist in English.

If a robot turns a "curled tongue" sound into a regular "flat tongue" sound, it's not a spelling error; it's an accent error. Current tools miss this.

2. The Solution: The "Six-Point Accent Report Card"

The authors created PSP, which is like a six-dimensional health check for a robot's accent. Instead of one single score, it gives you a report card with six specific grades:

Retroflex Collapse Rate (RR): How often did the robot fail to curl its tongue? (Like a student who keeps forgetting to do the homework).
Aspiration Fidelity (AF): Did it puff out the air when it was supposed to?
Vowel Length Fidelity (LF): Did it hold the long vowels long enough?
Tamil "Zha" Fidelity (ZF): Did it get that one tricky Tamil sound right?
Audio Distance (FAD): How far away does the robot's voice sound from a native speaker's voice in a "sound space"? (Think of this as a distance meter).
Prosodic Signature (PSD): Does the robot have the right rhythm, pitch, and melody? (Is it singing the song flatly, or with the right emotion?).

The Magic: The first four are measured by comparing the robot's sounds to a "native speaker average" (a centroid) using AI listening tools. The last two measure the overall "vibe" of the audio.

3. The Experiments: Testing the Robots

The authors tested four famous commercial robots (like ElevenLabs and Cartesia) and some open-source ones on Hindi, Telugu, and Tamil.

The Big Discoveries:

Difficulty Ladder: The robots found Hindi easiest, Telugu harder, and Tamil the hardest.
- Hindi: Robots got almost perfect scores on the tricky sounds.
- Telugu: Robots started messing up about 40% of the "curled tongue" sounds.
- Tamil: Robots messed up about 68% of those sounds.
The "Spelling" vs. "Accent" Disconnect: The robot that got the best spelling scores (lowest Word Error Rate) was not always the one with the best accent.
- Analogy: Imagine a student who gets an 'A' on a math test but fails the handwriting portion. The old grading system only looked at the math grade. PSP looks at both.
No Perfect Robot: No single robot was the best at everything. One robot might have great rhythm (PSD) but bad "curled tongue" sounds. Another might have great sounds but a flat, boring rhythm. You have to pick the right tool for the specific job.

4. The "Voice Prompt" Trick

The authors also tested their own robot (Praxy Voice). They found that if they gave the robot a 9-second audio clip of a real human speaking Telugu as a "reference," the robot suddenly sounded much more native.

Analogy: It's like a student listening to a native speaker for a few seconds before taking a test. The robot "caught the vibe" and improved its accent, even though it didn't relearn the whole language.

5. Why This Matters

The paper argues that we need to stop treating "accent" as a vague feeling. By breaking it down into these six specific dimensions, developers can see exactly where their robot is failing.

If the "Retroflex" score is low, they know to fix the tongue-curling sounds.
If the "Prosody" score is low, they know to fix the rhythm and emotion.

In short: This paper gives engineers a detailed map to navigate the tricky terrain of Indian accents, showing that getting the words right is only half the battle; getting the flavor right is the other half.

(Note: The paper explicitly states that these results are based on pilot tests and that formal correlation with human listening scores will be finalized in a future version, v2.)

1. Problem Statement

Current Text-to-Speech (TTS) evaluation for Indic languages relies heavily on intelligibility metrics (Word Error Rate - WER, Character Error Rate - CER) and overall naturalness scores (Mean Opinion Score - MOS, UTMOS).

The Gap: A synthesizer can achieve near-perfect WER and high MOS scores while still sounding "non-native" due to specific phonological errors.
The Specific Challenge: Indic languages possess systematic phonological features that non-native speakers (and many AI models) routinely collapse or mispronounce. These include:
- Retroflex articulation (e.g., distinguishing ṭ from t).
- Aspiration (e.g., kh vs. k).
- Phonemic vowel length (long vs. short vowels).
- Language-specific sounds (e.g., the Tamil retroflex approximant /r̲/ or ḻ).
Limitation of Existing Metrics: Metrics like WER measure transcription accuracy, not acoustic fidelity to native phonology. Distributional metrics like Fréchet Audio Distance (FAD) provide a single scalar score but lack interpretability regarding which phonological features are failing.

2. Methodology: The Phoneme Substitution Profile (PSP)

The authors propose PSP, a benchmark that decomposes "accent" into six interpretable, per-dimension metrics. It moves away from rule-based transcription matching toward acoustic probing using deep learning embeddings.

Core Mechanism

PSP uses forced alignment (via language-specific Wav2Vec2-XLS-R CTC aligners) to locate phoneme spans in generated audio. It then compares the acoustic embeddings of these spans against native-speaker centroids and substitute-speaker centroids (representing the non-native error) using rectified cosine similarity.

The Six Dimensions

The benchmark measures two types of dimensions:

Per-Phoneme Dimensions (measured via acoustic probes):
- Retroflex Collapse Rate (RR): Measures the failure to distinguish retroflex consonants (ṭ, ḍ, ṇ, ṣ, ḷ) from their dental counterparts.
- Aspiration Fidelity (AF): Measures the preservation of aspirated stops (kh, gh, etc.), primarily for Hindi.
- Length Fidelity (LF): Evaluates the ratio of long-to-short vowel durations against native priors.
- Tamil-zha Fidelity (ZF): Specifically measures the production of the Tamil retroflex approximant /r̲/ (ழ).
Corpus-Level Dimensions (measured via distributional distances):
- Fréchet Audio Distance (FAD): Calculates the distance between the generated and native audio distributions in the Wav2Vec2-XLS-R Layer-9 embedding space. This captures timbre, co-articulation, and global acoustic quality.
- Prosodic Signature Divergence (PSD): Calculates the Fréchet distance in a 5-dimensional prosodic feature space (pitch range, log-F0 mean, speech rate, nPVI, and log-duration).

Implementation Details

Reference Data: Native centroids are constructed from 500 studio-recorded clips per language (Hindi, Telugu, Tamil) using diverse speakers.
Scoring: The system computes fidelity scores by comparing generated embeddings to the native centroid ( $\mu_{nat}$ ) versus the substitute centroid ( $\mu_{sub}$ ).
Open Source: The code, centroids, and reference embeddings are released under MIT and CC-BY licenses.

3. Key Contributions

Formalization of Accent Metrics: Defined six specific, interpretable dimensions for Indic TTS evaluation, moving beyond single-scalar scores.
Acoustic-Probe-Based Approach: Unlike rule-based systems (e.g., PSR for English), PSP relies on acoustic embeddings, making it robust even when ASR transcription is error-prone for specific accents.
Resource Release: Released native-speaker centroids, 1000-clip utterance-level embeddings for FAD, 500-clip prosodic matrices for PSD, and 300-utterance "golden sets" for future benchmarking.
Comprehensive Benchmarking: Evaluated four systems (ElevenLabs v3, Cartesia Sonic-3, Sarvam Bulbul, Indic Parler-TTS) and an in-progress open-source system (Praxy Voice) across Hindi, Telugu, and Tamil.

4. Key Results & Findings

A. The "Accent vs. Intelligibility" Dissociation

WER $\neq$ Accent: Systems with the lowest WER (e.g., ElevenLabs, Cartesia) do not consistently lead in accent fidelity.
- Example: In Hindi, ElevenLabs had the lowest WER but ranked second in FAD. In Telugu, Cartesia had low WER but the worst retroflex collapse rate (50%).
Pareto Inefficiency: No single system is optimal across all six dimensions. For instance, on Tamil, Indic Parler-TTS won four of five dimensions, while Sarvam Bulbul won only FAD.

B. Phonological Difficulty Gradient

The study confirmed a monotonic increase in difficulty for Indic TTS, which PSP successfully tracks:

Hindi: ~1% retroflex collapse (Mature, near-native).
Telugu: ~40% retroflex collapse.
Tamil: ~68% retroflex collapse (Hardest).
Significance: Western-built commercial systems degrade significantly as the target language moves away from English phonology, whereas Indic-focused systems (Sarvam, Parler) generalize better.

C. Specific Failure Modes Detected

Prosodic Flatness (PSD): ElevenLabs' Telugu output had a massive PSD score (154 vs. ~11 for natives), indicating a "flat, non-expressive" delivery despite correct word pronunciation. WER completely missed this.
Tamil-zha Collapse: The unique Tamil sound /r̲/ collapsed at ~86% across most commercial systems.
Training Scale vs. Acoustic Decoder: In the Praxy Voice case study (R5 $\to$ R6), increasing training data (85h $\to$ 1220h) improved FAD (acoustic similarity) but worsened PSD (prosody) and left Retroflex Collapse unchanged. This proved that more data alone does not fix specific phonological articulation if the acoustic decoder weights aren't updated correctly.

D. Voice-Prompt Recovery

Using a "Bring Your Own Reference" (BYOR) approach, feeding a 9-second native Telugu clip into the Praxy R6 system significantly improved results:

Retroflex collapse dropped from 40% to ~27-33%.
PSD improved drastically, matching commercial baselines.
This demonstrated that conditioning on a native reference can bypass the need for massive retraining to fix accent issues.

5. Significance and Future Work

Paradigm Shift: PSP shifts TTS evaluation from "Can the machine read the text?" (WER) to "Does the machine sound like a native speaker?" (Accent).
Diagnostic Utility: The per-dimension breakdown allows developers to pinpoint specific failures (e.g., "The model has good prosody but fails on retroflexes") rather than relying on a single aggregate score.
Complementarity: PSP is designed to complement, not replace, existing metrics like PSR (for English) or MUSHRA listening tests.
Future Roadmap (v2): The authors plan to:
- Conduct formal MOS correlation studies with native raters.
- Scale benchmarks to 300 utterances per language for statistical significance.
- Normalize per-phoneme scores to account for alignment noise floors in Telugu/Tamil.
- Add code-mixed input evaluation.

Conclusion: The paper establishes that accent in Indic TTS is a measurable, decomposable vector of phonological features. PSP provides the first interpretable framework to quantify these features, revealing that current state-of-the-art TTS systems often fail to capture native phonology despite high intelligibility scores.