ArtBoost: Synthetic Articulatory Data Augmentation for Acoustic-to-Articulatory Inversion

Technical Summary: ArtBoost – Synthetic Articulatory Data Augmentation for Acoustic-to-Articulatory Inversion

Problem Statement
Acoustic-to-articulatory inversion (AAI) aims to estimate physical articulatory movements from speech signals. While deep learning approaches have improved AAI performance by leveraging large-scale paired audio–articulatory data, their efficacy is fundamentally constrained by the scarcity of such datasets. Collecting high-quality data via Electromagnetic Articulography (EMA) is costly, requires specialized hardware and controlled environments, and results in datasets that are small in scale and limited in speaker diversity. Consequently, data-driven AAI models often suffer from insufficient coverage of articulatory variability across speakers and phonetic contexts.

Methodology: ArtBoost
To address the data scarcity bottleneck without requiring additional sensor-based recordings, the authors propose ArtBoost, a data augmentation strategy that repurposes large-scale, audio-aligned 3D facial mesh datasets (originally developed for speech-driven 3D facial animation) as a source of pseudo articulatory supervision. The methodology proceeds in three stages:

1. ASR-Guided Utterance Segmentation: Since speech–mesh datasets (e.g., TFHP) consist of long, continuous video recordings while EMA corpora are organized at the utterance level, the authors use Automatic Speech Recognition (ASR) to segment the long recordings into utterance-level clips. These clips are synchronized with the corresponding 3D mesh frames to create pairs compatible with standard AAI training protocols.
2. Pseudo Articulatory Trajectory Extraction: From the segmented mesh sequences, the system tracks specific facial anchors corresponding to visible articulators: the upper lip (UL), lower lip (LL), and lower incisor (LI). To approximate articulatory motion, the mean position of predefined vertex regions for these anchors is calculated. The system extracts motion components along the protrusion (z-axis) and mouth opening (y-axis) directions. These signals are formatted into a 12-channel trajectory representation (matching standard EMA formats), where channels corresponding to the visible anchors are populated with the extracted data, and remaining channels are set to zero.
3. Two-Stage Training Strategy: The AAI model is trained using a pre-training and fine-tuning approach:
   • Pre-training: The model is trained on the large-scale pseudo articulatory trajectories derived from the speech–mesh data. A channel-masked loss function is employed, calculating prediction error only on the channels corresponding to the visible anchors (UL, LL, LI).
   • Fine-tuning: The pre-trained model is subsequently fine-tuned on the limited, real EMA datasets (HPRC and USC-TIMIT) using full-channel supervision with ground-truth trajectories.

Key Contributions

• Novel Data Source: The paper introduces the use of speech–mesh datasets as a scalable, synthetic source of articulatory supervision, effectively expanding the training space for AAI without new sensor recordings.
• Extraction Pipeline: A robust pipeline is established to convert visible facial mesh geometry into pseudo articulatory trajectories that align with conventional EMA data formats.
• Architecture Agnosticism: The strategy is designed to be integrated into diverse AAI model architectures, not limited to a specific network structure.

Experimental Results
The authors evaluated ArtBoost on the HPRC and USC-TIMIT EMA datasets using leave-one-speaker-out cross-validation.

• Performance Gains: Applying ArtBoost resulted in consistent improvements in both Pearson Correlation Coefficient (PCC) and Root Mean Square Error (RMSE). Notably, on the USC-TIMIT dataset (which has fewer ground-truth samples), PCC improved by approximately +25% (from 0.351 to 0.510), while HPRC showed a +2% improvement.
• Generalization: Improvements were observed across multiple articulators, including those not directly supervised by the pseudo labels (e.g., tongue and palate movements), suggesting the augmentation enhances the model's learned representation of speech production dynamics.
• Robustness: The performance gains were consistent across different AAI architectures, including Self-Supervised Learning (SSL)-AAI and Speaker-Independent (SI)-AAI models.
• Qualitative Analysis: Visualizations confirmed that the extracted pseudo trajectories reflect physically meaningful visible articulatory dynamics (e.g., lip closure and jaw motion) and that the final model predictions align with the temporal trends of ground-truth EMA signals.

Significance and Claims
The paper claims that ArtBoost provides an effective and scalable solution to the data limitation problem in AAI. By leveraging abundant synthetic speech–mesh data, the method allows models to learn a strong prior of visible articulatory motion before refining their predictions on limited real-world sensor data. The authors posit that speech–mesh data serve as a practical source of articulatory supervision, enabling more robust AAI models that can generalize better across speakers and conditions, even when direct articulatory measurements are scarce. The work suggests that this approach can be seamlessly integrated into existing AAI frameworks to improve performance without the need for costly data collection campaigns.