Towards a Phonology-Informed Evaluation of Multilingual TTS

Technical Summary: Towards a Phonology-Informed Evaluation of Multilingual TTS

Problem Statement
Current evaluation paradigms for Neural Text-to-Speech (TTS) systems rely heavily on Mean Opinion Scores (MOS) and automatic speech recognition (ASR) metrics like Word Error Rate (WER). While these metrics assess perceived naturalness and intelligibility, they fail to detect systematic phonological errors where a system fails to preserve sound contrasts that distinguish words or grammatical forms. The authors argue that a system can score highly on MOS while neutralizing or misplacing phonological contrasts dictated by a language's grammar, such as vowel harmony. Existing diagnostic methods are rare, particularly for multilingual systems, leaving a gap in evaluating whether TTS outputs respect the specific phonological patterns of a target language.

Methodology
The paper proposes a classifier-based framework to audit TTS output against language-specific phonological patterns, using human speech as a benchmark. The study focuses on Assamese Advanced Tongue Root (ATR) vowel harmony, a grammatical rule where suffix vowels trigger harmony on stem vowels, constraining which vowel qualities can co-occur.

The methodology involves a two-task evaluation pipeline:

1. Data Collection and Preparation:

   • Human Benchmark: Speech from 14 native Assamese speakers was recorded, segmented, and normalized (Lobanov normalization) to extract acoustic features (F1, F2, F3, B1, duration, vowel height, backness).
   • TTS Dataset: Meta's Massively Multilingual Speech (MMS) TTS model was used to synthesize the same target words. The dataset was processed with the same acoustic measurement protocol.
   • Labels: Tokens were assigned binary ATR labels ([+ATR], [-ATR]) and harmony categories (e.g., AgrYesMixNo, AgrYesMixYes, AgrNoMixYes) based on phonological transcription.

2. Task 1: Vowel-Level ATR Classification (Cross-Domain Transfer):

   • Logistic Regression (LR) and Random Forest (RF) classifiers were trained on human speech features to predict ATR classes.
   • The models were tested in four transfer directions: Human→Human (H→H), Human→TTS (H→TTS), TTS→TTS, and TTS→Human.
   • Goal: To determine if the acoustic-to-phonology mapping learned from humans transfers to synthesized speech without significant accuracy loss, indicating a divergence in acoustic structure.

3. Phonological Faithfulness Audit:

   • The framework compares the "gold" ATR label (from phonological transcription) against the ATR label predicted by the classifier trained on human speech.
   • Metrics: Mismatch rates are calculated, distinguishing between over-generation (predicting [+ATR] for underlying [-ATR]) and under-production (predicting [-ATR] for underlying [+ATR]).

4. Task 2: Word-Level Harmony Classification:

   • A Random Forest classifier predicts word-level harmony categories using aggregated vowel features.
   • The study compares performance using gold ATR labels versus predicted ATR labels (derived from Task 1) to measure the gap between intended phonology and produced acoustics.

Key Results

• Cross-Domain Transfer: The linear classifier (LR) maintained high accuracy (approx. 82%) when transferring from Human to TTS, suggesting a stable acoustic-to-ATR mapping. However, the non-linear classifier (RF) showed a significant drop (90.5% H→H vs. 74.7% H→TTS), indicating that complex, speaker-conditioned boundaries in human speech do not generalize well to the single synthetic voice.
• Faithfulness Audit:
  • Human Speech: Showed a symmetric mismatch rate (0.185) with balanced over-generation and under-production errors.
  • TTS (MMS): Showed a distinct directional bias. While the overall mismatch rate was slightly lower (0.164), errors were heavily skewed toward under-production (7:1 ratio). Specifically, underlying [+ATR] mid-vowels (/e/ and /o/) were frequently misclassified as [-ATR] by the human-trained classifier.
  • Acoustic Evidence: Spectrograms revealed that in TTS, the F1 of /e/ and /o/ tokens often sat near or above the [-ATR] boundary derived from human speech, confirming the acoustic realization of [+ATR] vowels as [-ATR].
• Word-Level Analysis: When applying word-level classifiers to TTS, models trained on gold ATR labels performed significantly worse (Macro-F1 0.49) than those trained on predicted ATR labels (Macro-F1 0.62). This gap indicates that the acoustic output of the TTS system does not consistently align with the intended phonological structure.

Significance and Claims
The paper claims that standard metrics like MOS are insufficient for detecting systematic phonological failures in multilingual TTS. The proposed framework offers a task-specific, diagnostic approach that:

1. Identifies Hidden Biases: It reveals that even high-performing TTS systems (like MMS) can systematically neutralize grammatical contrasts (specifically ATR harmony in Assamese) without sounding unnatural to human listeners.
2. Provides Targeted Feedback: The audit format quantifies the divergence between intended phonology and acoustic output, offering developers specific insights into where and how a system deviates (e.g., specific vowel classes).
3. Generalizability: While the study focuses on Assamese ATR harmony, the authors state the methodology is replicable and adaptable to other phonological contrasts with measurable acoustic correlates.

Limitations
The authors modestly note that the study evaluates a single TTS system (MMS) for a single phonological phenomenon in a single language using a relatively small, class-imbalanced TTS dataset. Consequently, the specific error rates and the strength of claims are constrained by these factors, though the general approach is presented as extensible.