Technical Summary: ARTI-6 – Towards Six-Dimensional Articulatory Speech Encoding

1. Problem Statement

Speech production involves coordinated movements of vocal tract articulators that generate acoustic signals. However, the mapping from acoustics to articulation is non-transparent, making articulatory inversion (inferring hidden movements from sound) and articulatory synthesis (reconstructing sound from movements) challenging.

Existing solutions face specific limitations:

• Data Modality: Previous high-performing models like SPARC rely on Electromagnetic Articulography (EMA) data. EMA captures only a subset of the vocal tract, missing critical regions such as the velum, tongue root, and larynx. Consequently, these models lack explicit information on nasality and voicing, often requiring acoustic correlates (e.g., pitch, energy) to compensate.
• Representation Complexity: While real-time MRI (rtMRI) provides full coverage of the vocal tract, the resulting data is high-dimensional and opaque. Conversely, overly simplified features risk discarding essential physiological information.
• The Gap: There is a need for a compact, interpretable, and physiologically grounded representation that captures key articulatory dynamics (including the missing EMA regions) while remaining computationally efficient for modeling and real-time applications.

2. Methodology: The ARTI-6 Framework

The authors propose ARTI-6, a six-dimensional articulatory speech encoding framework derived from rtMRI data. The framework consists of three core components:

A. Six-Dimensional Feature Selection

The authors selected six key Regions of Interest (ROIs) based on knowledge-driven selection of articulators critical for speech production goals (constriction formation and release). These regions are:

1. Lip Aperture (LA)
2. Tongue Tip (TT)
3. Tongue Body (TB)
4. Velum (VL)
5. Tongue Root (TR)
6. Larynx (LX)

Feature Extraction: Instead of complex kinematic tracking, the authors use image pixel intensity from rtMRI frames as a proxy measure for the degree of constriction in each region. This choice is motivated by the assumption that these regions represent the minimal articulatory requirement for speech, that other kinematic points correlate highly with them, and that models can infer full dynamics from these representatives.

B. Articulatory Inversion Model

This component predicts the six-dimensional articulatory features from speech acoustics.

• Backbone: The model leverages pre-trained speech foundation models (Whisper-Large, WavLM, HuBERT, Wav2Vec2).
• Architecture: The foundation model is fine-tuned using LoRA (Low-Rank Adaptation). Learnable low-rank matrices are inserted into the feed-forward layers of Transformer blocks.
• Processing: A learnable linear layer combines hidden states from convolutional and Transformer layers. The output passes through 1D pointwise convolutional layers and a projection layer to map to the six-dimensional target features.
• Data: Trained on the USC Long Single-Speaker (LSS) dataset (54 minutes of rtMRI data from a single speaker), with noise removal applied via the Miipher speech restoration model.

C. Articulatory Synthesis Model

This component reconstructs intelligible speech directly from the six-dimensional articulatory features.

• Architecture: Based on HiFi-GAN v1.
• Configuration: Uses specific upsample rates and kernel sizes to match a 16 kHz output sampling rate.
• Conditioning: The model is conditioned on speaker embeddings extracted from ECAPA-TDNN to capture individual speaker characteristics, assuming the synthesis model can learn vocal tract traits from these embeddings.
• Training Data: Since the LSS dataset is insufficient for training a deep synthesizer, the authors augmented training using the LibriTTS-R dataset. They generated pairs of predicted articulatory features (using the WavLM-based inversion model) and corresponding speech acoustics to train the synthesis model.

3. Key Results

Articulatory Inversion Performance

• Overall Correlation: The model achieved a prediction correlation of 0.87 (Pearson correlation) on the test set.
• Best Backbone: WavLM-Large and HuBERT yielded the highest performance (0.872 correlation on the test set), outperforming Whisper-Large and Wav2Vec2-XLSR.
• Phonemic Analysis: Performance varied by region and phoneme type.
  • High Accuracy: Lip and tongue regions (LA, TT, TB) showed high prediction accuracy.
  • Lower Accuracy: Velum (VL) and Larynx (LX) showed comparatively lower performance.
  • Attribution: Lower velum performance is attributed to noise in pixel intensity during closure (fluctuations unrelated to active movement). Lower larynx performance is attributed to the limitations of midsagittal rtMRI in capturing detailed internal geometry dynamics.

Articulatory Synthesis Performance

The synthesized speech was evaluated against ground truth and other intermediate feature sets (Mel-spectrograms, EMA+pitch+loudness).

• Intelligibility: Despite the extreme dimensionality reduction (6 dimensions), the speech remained acceptably intelligible.
  • Word Error Rate (WER): ~12.5%
  • Character Error Rate (CER): ~7.4%
• Naturalness:
  • UTMOS: 3.840 (out of 5)
  • Mean Opinion Score (MOS): 3.947
• Comparison: While the quality is lower than high-dimensional approaches (e.g., Mel-spectrograms or EMA-based methods), the results demonstrate that a 6-dimensional representation can generate natural-sounding speech.

4. Significance and Claims

The paper claims that ARTI-6 provides a unified, interpretable, and computationally efficient framework for speech technology. Its significance lies in three main areas:

1. Physiological Completeness: Unlike EMA-based systems, ARTI-6 explicitly captures the velum, tongue root, and larynx, providing a fuller coverage of the vocal tract without relying on acoustic proxies for nasality or voicing.
2. Interpretability: The six regions are grounded in articulatory kinematics and linguistic models, making the inferred features valuable for scientific research where direct articulatory data acquisition is difficult (e.g., correlating speech with brain or muscle signals).
3. Efficiency and Practicality: With only six dimensions, ARTI-6 is significantly more compact than conventional features (MFCCs, mel-spectrograms) or latent spaces (200+ dimensions). This low dimensionality offers computational efficiency and low latency, making it suitable for real-time applications such as on-device processing and streaming.

5. Limitations and Future Work

The authors acknowledge that the current framework is limited to a single-speaker articulatory space. Future work aims to:

• Extend the framework to multi-speaker spaces to enable generalization without speaker-specific conditioning.
• Refine tongue variables for better cross-speaker comparability.
• Optimize the selection of articulatory regions and proxy measures.
• Explore hybrid feature sets combining articulatory and acoustic features.

The source code and speech samples are publicly available to facilitate further research in articulatory inversion and synthesis.