Evaluating Speech Articulation Synthesis with Articulatory Phoneme Recognition

Technical Summary: Evaluating Speech Articulation Synthesis with Articulatory Phoneme Recognition

Problem Statement
The evaluation of speech articulation synthesis—specifically the generation of vocal tract shapes conditioned on phonetic sequences—lacks a robust, objective quality metric. Traditional metrics, such as point-to-closest-point distances, are limited by substantial inter- and intra-speaker variability and often fail to capture the nuances of phoneme production, such as correct places of articulation. Furthermore, these metrics struggle to differentiate between advanced generative models (e.g., model-free vs. autoencoder-based systems) that may appear indistinguishable by geometric distance but differ in temporal stability and articulatory accuracy. Subjective analysis is difficult due to the requirement for specialized knowledge in vocal tract anatomy and acoustics. Consequently, there is a need for a metric that better captures phonetic information and intelligibility without relying solely on acoustic reconstruction.

Methodology
The authors propose using phoneme recognition as a proxy metric to evaluate the quality of synthesized articulatory features. The core hypothesis is that a phoneme recognizer trained on articulatory data can better capture production nuances (e.g., place of articulation) than traditional distance metrics.

1. Dataset: The study utilizes a corpus of 2.5 hours of speech from a single female native French speaker, comprising Real-Time MRI (RT-MRI) images (50 Hz), denoised audio, and manual phonetic annotations. The dataset is split into training (1,399 utterances), validation (116), and test (114) sets.
2. Feature Extraction:
   • Articulatory Features: Contours of ten vocal tract articulators (arytenoid cartilage, epiglottis, lower incisor, lower lip, pharynx, soft palate, thyroid cartilage, tongue, upper lip, and vocal folds) are extracted. These are concatenated into a 2-channel, 500-dimensional vector.
   • Voicing Encoding: Since mid-sagittal RT-MRI lacks vocal fold excitation data, a categorical encoding representing voicing information is added to the feature set.
   • Synthetic Features: The study evaluates three synthesizers from existing literature:
     • Phoneme-wise mean-contour: A baseline averaging contours per phoneme.
     • Model-free system: Generates shapes directly from phoneme sequences without an explicit articulatory model.
     • Autoencoder-based system: Generates articulatory parameters based on an autoencoder architecture.
3. Model Architecture: A phoneme recognizer based on the Deep Speech 2 architecture is employed. It consists of an adapter block (to map 500D articulatory inputs to 80D), followed by five residual convolutional blocks and three recurrent blocks (GRU), ending with a linear classifier.
4. Training and Evaluation:
   • The recognizer is trained on acoustic features (Mel spectrograms) and true articulatory features (with and without voicing encoding) to establish baselines.
   • The trained recognizer is then tested on the synthetic articulatory features generated by the three synthesizers.
   • Metrics: The primary evaluation metric is the Phoneme Error Rate (PER) calculated via Levenshtein distance. Additionally, t-Distributed Stochastic Neighbor Embedding (t-SNE) is used to visualize the feature representations learned by the models.

Key Results

• Baseline Performance: The recognizer trained on true articulatory features (with voicing encoding) achieved a PER of 21.66%, which is comparable to the 23.30% PER achieved using the acoustic signal. This suggests that the articulatory contours alone retain sufficient phonetic information to rival acoustic signals, despite the absence of source information.
• Synthesizer Comparison:
  • Model-Free System: Surprisingly, this system achieved the lowest PER (20.59%) among all tested sets, even outperforming the true articulatory features. The authors suggest this may be because the synthesizer filters out noise present in the raw MRI data, generating cleaner articulations.
  • Autoencoder-Based System: This system yielded a higher PER (31.69%) than the model-free approach, indicating it reproduces less phonetic information, despite previous claims of better place-of-articulation learning.
  • Mean-Contour Baseline: As expected, the simplistic mean-contour approach performed the worst (PER of 43.18%).
• Confusion Analysis: The confusion matrices revealed specific weaknesses. The model-free system exhibited high deletion rates for labial, back, and rounded vowels, likely due to a lack of lip rounding data or difficulty in approximating lip closure. The autoencoder system showed high deletion rates across all classes, making it difficult to isolate specific articulatory failures.
• Feature Embeddings: t-SNE plots demonstrated that adding voicing encoding created distinct, separable groups in the embedding space for articulatory features, a structure not as apparent in the acoustic features.

Significance and Claims
The paper claims that Automatic Speech Recognition (ASR) serves as a viable and effective objective metric for evaluating speech articulation synthesis. The authors argue that:

1. Phonetic Richness: Articulatory features extracted from RT-MRI are phonetically rich and, when combined with voicing encoding, contain information comparable to or exceeding that of the acoustic signal in terms of phoneme recognition.
2. Differentiation Capability: Unlike geometric distance metrics, phoneme recognition can meaningfully discriminate between different generative models (e.g., distinguishing the temporal stability of the model-free system from the autoencoder-based system).
3. Alignment with Perception: The recognition error correlates with qualitative perceptions of the synthesized speech (e.g., stability and intelligibility), providing an objective measure for what was previously a subjective assessment.

The authors maintain a modest stance, noting that their results are specific to a single speaker due to the high cost and logistical constraints of RT-MRI data collection. They do not claim the method is a universal solution for all speakers yet but propose that speaker-independent phoneme recognition offers a promising direction for future multi-speaker evaluation and implicit speaker normalization in synthesis training pipelines.