Pairwise Evaluation of Accent Similarity in Speech Synthesis

Technical Summary: Pairwise Evaluation of Accent Similarity in Speech Synthesis

Problem Statement
Despite the growing interest in generating high-fidelity accents for inclusive speech technology, the evaluation of accent similarity remains underexplored and lacks consensus. Existing subjective methods often rely on reference-free ratings that incorrectly assume accents exist on a one-dimensional intensity scale or are purely categorical. Reference-based methods (e.g., XAB, XMOS) borrowed from naturalness evaluation have not been rigorously validated specifically for accent similarity. Furthermore, objective evaluation often relies on proxy metrics like Word Error Rate (WER) or embeddings from Accent Identification (AID) models, which may suffer from accent bias and fail to accurately represent underrepresented accents.

Methodology
The authors propose a dual approach to refine both subjective and objective evaluation frameworks.

• Subjective Evaluation (Refined XAB Test):
  The study enhances the standard XAB listening test (where listeners choose the candidate speech most similar in accent to a reference) with three specific components to improve statistical significance with fewer listeners:

  1. Transcription Provision: Listeners are provided with text transcripts to focus attention on pronunciation details.
  2. Highlighting Task: Listeners are asked to highlight specific phonetic segments in the transcript that influenced their decision regarding accent differences, inspired by Rapid Prosody Transcription.
  3. Meticulous Screening: A two-stage screening process filters invalid submissions: (a) attention-check questions using speech samples of known accent similarity, and (b) an open-ended question requiring listeners to identify the specific country or city of the reference accent.

• Objective Evaluation (Pronunciation-Related Metrics):
  The authors propose utilizing metrics based on phonetic identity rather than general intelligibility or speaker verification alone:

  1. Vowel Formant Distance (VF RMSE): Calculating the Root Mean Square Error between the first and second formants (F1 & F2) of vowels, extracted via Montreal Forced Aligner and Fast Track estimation.
  2. Phonetic Posteriorgram (PPG) Distance: Measuring pronunciation distance using Dynamic Time Warping (DTW) alignment on PPGs. The authors utilize cosine similarity and Jensen-Shannon distance as cost functions, explicitly avoiding normalization that might bias results toward General American phones.
  3. Comparative Baselines: The study also evaluates standard metrics including Cosine Similarity of AID/Speaker Verification embeddings, WER/CER, UTMOS (naturalness), and Mel Cepstral Distortion (MCD).

Experimental Setup

• Stimuli: Utterances from the VCTK corpus (Rainbow passage) featuring the Edinburgh accent, an underrepresented variety in English TTS.
• Systems: A ground-truth reconstruction (copysyn), a Zero-Shot TTS system (xtts), and a series of "corrupted" systems (corrupt30k to corrupt150k) created by fine-tuning xtts on a General American corpus to induce catastrophic forgetting of the target accent.
• Listeners: Native speakers of the Edinburgh accent recruited via Prolific.

Key Results

• Subjective Findings: The refined XAB test design significantly outperforms naive XAB tests.
  • Standard XAB tests failed to distinguish between copysyn and xtts (preference near 50%, high p-values).
  • Adding transcription (+trans) and screening (+screen) achieved statistical significance (p < 0.05).
  • The most effective design, XAB+trans+highlight+screen, achieved the highest preference for the correct system (64.1%) and required as few as 5 valid submissions to reach statistical significance, demonstrating high efficiency.
  • The highlighting task nearly doubled completion time but proved effective for accents with limited participant pools.
• Objective Findings:
  • Pronunciation Metrics: Vowel Formant RMSE, PPG distances, and Cosine Similarity of AID/SV embeddings showed strong correlations (Spearman Rank Correlation Coefficient > 0.85) with the hypothesized system ranking.
  • Limitations of Standard Metrics: WER and CER showed poor correlation and potential bias (e.g., ranking copysyn worse than xtts despite copysyn being a ground-truth reconstruction). UTMOS and F0 metrics also showed weak correlations, likely due to accent bias in training data or the specific nature of the corruption.
  • Visual Validation: Formant distribution plots confirmed that copysyn maintained the vowel space of the ground truth, while xtts showed clear mismatches, validating the VF RMSE metric.

Significance and Claims
The paper claims to address a critical gap in speech synthesis evaluation by providing:

1. Reliable Subjective Protocols: A refined XAB framework that achieves higher statistical significance with fewer resources, making it feasible to evaluate underrepresented accents.
2. Robust Objective Indicators: Evidence that pronunciation-specific metrics (Vowel Formants and PPGs) are superior to standard intelligibility metrics (WER/CER) for evaluating accent generation, particularly for underrepresented varieties.
3. Cautionary Findings: The study underscores the significant limitations of common metrics like WER and UTMOS in assessing underrepresented accents due to inherent biases, calling for a shift toward more inclusive evaluation frameworks.

The authors conclude that a combination of pronunciation-related metrics, accent/speaker similarity, and audio quality metrics offers a viable path for evaluating accent generation, while explicitly recommending against the use of WER, CER, and UTMOS for this specific purpose. Future work is proposed to explore broader accent varieties and alternative test designs.