Technical Summary: An Evaluation Framework for Text-to-Speech Voice Reconstruction

1. Problem Statement

Voice reconstruction using Text-to-Speech (TTS) aims to create personalized communication aids for individuals with speech disorders (e.g., caused by ALS, Parkinson's, Cerebral Palsy). The goal is to improve speech intelligibility while retaining the speaker's unique identity. However, evaluating these systems is challenging because:

• Lack of Ground Truth: There is no objective "correct" output for what a reconstructed voice should sound like; it is inherently subjective.
• Limitations of Standard Metrics: Previous work relies heavily on Mean Opinion Score (MOS) for naturalness and similarity, which suffers from low sensitivity, reliability issues, and saturation.
• Inadequacy of Objective Proxies: Standard objective measures (Word Error Rate, Cosine Similarity of speaker embeddings) often fail to predict listener preferences for voice reconstruction, particularly for speakers with highly unintelligible speech. They tend to prioritize intelligibility over identity or fail to disentangle disorder-induced articulation patterns from speaker identity.

2. Methodology

The authors propose a comprehensive evaluation framework combining subjective and objective components, tested on 193 speakers from the Speech Accessibility Project (SAP) dataset across 17 zero-shot TTS systems.

2.1 Dataset and Systems

• Speakers: 193 speakers with four conditions (Parkinson's, Cerebral Palsy, ALS, Down Syndrome), stratified by intelligibility (High: WER < 30%; Low: WER ≥ 30%).
• Systems: 17 zero-shot voice-cloning TTS systems (e.g., StyleTTS2, IndexTTS2, Qwen3-TTS, OpenVoice) covering various architectures and training data types.
• Stimuli: For each speaker, one utterance serves as the reference for zero-shot cloning, and a second utterance's transcript is used to generate synthetic output.

2.2 Subjective Evaluation

Instead of standard MOS, the authors employ Best Worst Scaling (BWS) with situational framing to isolate specific evaluation dimensions:

• INTELLIGIBILITY Task: Listeners assess clarity ignoring speaker identity (i.e., "How easy is this to understand?").
• RECONSTRUCTION Task: Listeners assess the overall quality by comparing synthetic speech to their mental model of the speaker's pre-disorder voice, weighing both intelligibility and identity.
• Protocol: 50 listeners per task type completed online surveys. BWS rankings were computed using a Plackett-Luce model.

2.3 Objective Evaluation

The authors investigate the correlation of standard and novel metrics against the subjective rankings:

• Standard Metrics: Word Error Rate (WER), Phone Error Rate (PER), Speaker Cosine Similarity (Spk. Sim.), and UTMOS (MOS prediction).
• Novel Metric (Dual-Reference Distributional Measure): Inspired by distributional evaluation in computer vision, the authors introduce a trade-off measure using TTSDS2.
  • TTSDS|SAP: Distance to the original disordered speech (proxy for identity preservation).
  • TTSDS|LibriTTS: Distance to a high-intelligibility, typical speech corpus (proxy for intelligibility).
  • TTSDS Mean: The average of the two scores, intended to quantify the balance between intelligibility and identity.

3. Key Results

3.1 Subjective Findings

• Intelligibility: Most TTS systems outperformed the original recordings in intelligibility. StyleTTS2 ranked highest, followed by Fish Speech and OpenVoice.
• Reconstruction: Most systems ranked lower than the original recordings, indicating a failure to fully capture desired speaker characteristics. IndexTTS2 ranked highest, followed by Qwen3-TTS and E2-TTS.
• Low Intelligibility Speakers: For severely disordered speakers, most systems failed to maintain identity while improving intelligibility in the Reconstruction task. Only IndexTTS2 and Qwen3-TTS ranked higher than the original recordings.

3.2 Objective Correlations

• Intelligibility: WER, PER, and UTMOS correlated strongly with the subjective INTELLIGIBILITY scores. The proposed TTSDS|LibriTTS also showed strong correlation (ρ = 0.85), performing comparably to UTMOS but better on low-intelligibility subsets.
• Reconstruction: Standard metrics failed to predict Reconstruction rankings.
  • Speaker Cosine Similarity showed a moderate correlation (ρ = 0.75) for all speakers but dropped significantly for low-intelligibility speakers (ρ = 0.61).
  • TTSDS Mean (the dual-reference measure) achieved the strongest correlation with subjective RECONSTRUCTION rankings (ρ = 0.81 overall, ρ = 0.73 for low-intelligibility), outperforming speaker similarity.

3.3 System Performance

Three systems—Qwen3-TTS, IndexTTS2, and E2-TTS—consistently achieved the highest scores in the subjective RECONSTRUCTION task and the TTSDS Mean objective metric. These systems outperformed the original recordings for the full dataset but showed limitations with the most severely disordered speakers.

4. Contributions and Significance

The paper claims the following contributions:

1. Evaluation Framework: A reliable, task-aligned framework for voice reconstruction that moves beyond generic "naturalness" scores to specifically evaluate the trade-off between intelligibility and speaker identity.
2. Situational Framing: Demonstrating that listener instructions must be situationally framed (distinguishing between pure intelligibility and reconstruction) to yield meaningful evaluation data.
3. Novel Objective Metric: Introducing the TTSDS Mean (dual-reference distributional measure), which successfully predicts listener preferences for voice reconstruction where standard metrics (WER, Spk. Sim.) fail, particularly for highly unintelligible speakers.
4. Baseline Systems: Identifying Qwen3-TTS, IndexTTS2, and E2-TTS as strong zero-shot baselines for future voice reconstruction research, as they require no fine-tuning.

5. Limitations and Future Work

The authors note modestly that:

• The findings are based on zero-shot systems; future work should investigate if these results generalize to adapted or fine-tuned systems.
• The current framework focuses on intelligibility and identity; dimensions like accent similarity and suitability for daily conversation remain for future study.
• The evaluation relies on listeners rather than the actual end-users (VOCA users), suggesting a need for more user-centric approaches in future work.
• The dataset is skewed toward Parkinson's and high-intelligibility speakers, though the framework was stress-tested on low-intelligibility subsets.