Acoustic-to-Articulatory Inversion of Clean Speech Using an MRI-Trained Model

This study demonstrates that acoustic-to-articulatory inversion models trained on denoised MRI data can effectively reconstruct vocal tract shapes from clean speech, achieving performance comparable to MRI-based methods with an RMSE of 1.56 mm.

The Gist

Imagine you are trying to figure out exactly how someone's mouth, tongue, and throat are moving just by listening to their voice. This is called Acoustic-to-Articulatory Inversion. It's like being a detective who can look at a fingerprint (the sound) and perfectly reconstruct the hand that made it (the mouth movements).

For a long time, scientists tried to solve this puzzle, but they needed a "gold standard" to train their computer models. Usually, this meant putting a microphone inside a giant, loud MRI machine while a person spoke. The MRI camera could see the tongue moving in real-time, providing the perfect answer key.

However, there was a huge problem: MRI machines are incredibly loud. The audio recorded inside them is full of static and scanner noise, like trying to hear a whisper in a rock concert. Even after cleaning up the audio, it still sounds "robotic" and unnatural compared to normal speech.

The Big Question

The researchers asked: "Can we train our detective using the noisy MRI audio, but then have it solve cases using clean, quiet speech recorded in a normal room?"

If the answer is yes, we can finally use this technology in real life (like in voice assistants or medical apps) without needing a giant MRI machine.

The Experiment: The "Twin" Speakers

To test this, the team used a clever setup:

1. The MRI Twin: A woman spoke a list of sentences inside the MRI machine. The camera recorded her tongue, and the microphone recorded her noisy voice.
2. The Clean Twin: The same woman, speaking the exact same sentences, but in a quiet, soundproof room.

Because it was the same person saying the same words, the researchers could line up the two recordings perfectly, like matching two different maps of the same city.

The Training Methods

They tried three different ways to train their AI model:

1. The Perfect Match (M2M): Train on noisy MRI audio, test on noisy MRI audio. (This is the "control group" and the best possible score).
2. The Mismatch (M2C): Train on noisy MRI audio, but test on clean, quiet audio. (This is what happens if you try to use a model trained in an MRI machine in the real world without fixing it).
3. The Clean Swap (C2C): Train on clean audio, test on clean audio. (This is the goal: a model that works in the real world).

The Secret Sauce: Phonetic Alignment

The tricky part was that the woman spoke slightly faster or slower in the MRI machine compared to the quiet room. To fix this, the researchers didn't just match the sounds; they matched the phonemes (the tiny building blocks of speech, like "b," "a," or "t").

Think of it like syncing two different versions of a movie. One version has a slow scene, the other has a fast scene. Instead of just stretching the whole movie, they paused at every specific word and adjusted the timing so the actors' lips matched the dialogue perfectly. This ensured the AI learned the right connection between the sound and the tongue movement, regardless of speed.

The Results: A Happy Ending

Here is what they found:

• The Mismatch (M2C) failed a bit: When they took a model trained on noisy MRI data and tested it on clean speech, the accuracy dropped. The AI got confused because the "voice" it learned didn't quite match the "voice" it was hearing.
• The Clean Swap (C2C) worked amazingly: When they trained the model on clean speech, it performed almost as well as the "Perfect Match" model trained on the noisy MRI data.
  • The error rate was about 1.56 millimeters.
  • To put that in perspective: The MRI camera itself can only see details as small as 1.62 millimeters.
  • The Analogy: It's like trying to draw a picture of a face. The camera (MRI) can only see details down to the size of a grain of sand. The AI's drawing was only slightly less accurate than the camera's own view. That is incredibly precise!

Why This Matters

Before this study, people thought you had to use noisy MRI data to get good results. This paper proves that you don't.

By using a smart alignment method and training on clean speech, we can now build systems that understand how our mouths move just by listening to us speak in a normal room. This opens the door for:

• Better Voice Assistants: That understand accents and speech disorders better.
• Medical Tools: Helping people with speech issues practice without needing a hospital scan.
• Animation: Creating realistic talking avatars just from audio files.

In short, the researchers took a technology that was stuck in a noisy, expensive machine and successfully moved it out into the real world, proving that clean speech is just as powerful as the noisy kind for teaching computers how we speak.

Technical Summary

Here is a detailed technical summary of the paper "Acoustic-to-Articulatory Inversion of Clean Speech Using an MRI-Trained Model."

1. Problem Statement

Acoustic-to-articulatory inversion aims to reconstruct the geometric shape of the vocal tract (articulators) from an acoustic speech signal. While effective, most high-performance models rely on training data derived from Real-time Magnetic Resonance Imaging (rt-MRI).

• The Challenge: Speech recorded inside an MRI scanner is heavily corrupted by scanner noise. Even after denoising, this "MRI speech" differs significantly from natural "clean speech" recorded in quiet environments (e.g., lower high-frequency energy, Lombard effect due to supine position).
• The Gap: Current inversion systems trained on denoised MRI data often fail or degrade in performance when applied to clean speech, limiting their practical use in real-world applications where MRI data is unavailable.
• Objective: The study investigates whether a model trained on denoised MRI data can effectively invert clean speech, and whether training directly on clean speech yields comparable results to the MRI-based reference.

2. Methodology

A. Dataset Construction

The authors utilized two corpora recorded by the same native French female speaker:

1. MRI Corpus: 2.5 hours of data recorded at CHRU Nancy.
   • Visual: rt-MRI images at 136×136 pixels (50ms/frame, 20fps).
   • Audio: Recorded via optical microphone (16 kHz), then denoised using a specific algorithm [24].
   • Annotation: 44 phonemes, manually corrected.
2. Clean Corpus: The same speaker produced the identical set of sentences in a quiet environment.
   • Audio: Recorded at 48 kHz, downsampled to 16 kHz.
   • Annotation: Automatically generated via Montreal Forced Aligner (MFA) and manually corrected.
   • Crucial Step: Both corpora were strictly aligned to ensure identical phonetic segmentation.

B. Pre-processing and Alignment

• Feature Extraction: The study utilized HuBERT-Base (Self-Supervised Learning) embeddings (768 dimensions, 50 Hz) as input, outperforming traditional MFCCs and other SSL models.
• Target Definition: Articulatory contours were extracted from MRI images using an automatic tracking approach [13], dividing the vocal tract into 8 articulators (Arytenoid cartilage, Epiglottis, Lower lip, Pharyngeal wall, Velum, Tongue, Upper lip, Vocal folds), each represented by 50 points.
• Hierarchical Phonetic Alignment: To map clean speech to MRI frames, a custom alignment algorithm was developed:
  1. Sentence Level: Matching sentences using Gestalt pattern matching (similarity > 75%).
  2. Word/Phoneme Level: Aligning based on textual equality and relative position within the sentence.
  3. Temporal Normalization: Applying local time-stretching at the phoneme level to compensate for duration differences between the two recording conditions.

C. Model Architecture

The inversion model is a regression network inspired by previous works [14, 15]:

• Input: Audio embeddings (HuBERT).
• Layers:
  1. Two Dense layers (300 units each).
  2. Two Bidirectional LSTM (Bi-LSTM) layers (300 units each).
  3. One Dense output layer.
• Output: A tensor of size $8 \times 100$, representing the X and Y coordinates for 50 points across 8 articulators.
• Loss Function: Mean Squared Error (MSE).

D. Experimental Configurations

The study evaluated three distinct scenarios:

1. M2M (MRI-to-MRI): Train and test on denoised MRI speech (Baseline).
2. M2C (MRI-to-Clean): Train on denoised MRI speech, test on clean speech (Simulating real-world application without adaptation).
3. C2C (Clean-to-Clean): Train and test on clean speech (Simulating a model built entirely on clean data).

A secondary experiment compared the proposed Phonetic Alignment against Dynamic Time Warping (DTW) alignment.

3. Key Contributions

1. Feasibility of Clean Speech Inversion: Demonstrated that articulatory inversion can be effectively performed on clean speech, achieving performance close to the MRI-trained baseline.
2. Phonetic Alignment Strategy: Introduced a hierarchical, phoneme-based alignment method that outperforms standard acoustic alignment (DTW) by respecting phonetic boundaries and content rather than just acoustic similarity.
3. High-Resolution Dataset: Utilized a high-resolution (136×136) rt-MRI dataset with superior denoising compared to previous corpora (e.g., [9]), providing a robust benchmark.
4. Practical Applicability: Showed that inversion systems do not strictly require MRI-recorded training data to function on clean speech, bridging the gap between laboratory research and real-world deployment.

4. Results

The performance was measured using Root Mean Square Error (RMSE) and Median Error (in millimeters) across the 8 articulators.

Configuration	Mean RMSE (mm)	Mean Median (mm)	Observation
M2M (Baseline)	1.51	1.33	Best performance; no domain mismatch.
C2C (Clean)	1.56	1.33	Very close to baseline. Significant improvement over M2C.
M2C (No Adapt)	1.64	1.39	Performance drops when testing clean speech on an MRI-trained model without adaptation.

• Alignment Impact: The C2C configuration (1.56 mm) significantly outperformed the M2C configuration (1.64 mm), proving that training on clean speech (with proper alignment) is superior to simply testing clean speech on an MRI-trained model.
• Alignment Method: Models using the proposed Phonetic Alignment significantly outperformed those using DTW (e.g., C2C-DTW RMSE was 1.68 mm vs. 1.56 mm for C2C). This confirms that phonetic boundaries are critical for accurate temporal mapping.
• Statistical Significance: The differences between M2M and C2C were not statistically significant in many articulators, whereas M2C showed significant degradation compared to M2M.

5. Significance and Conclusion

• Practical Viability: The study concludes that acoustic-to-articulatory inversion is viable for real-world applications using clean speech. The error margin of 1.56 mm is highly acceptable given the spatial resolution of the MRI ground truth (1.62 mm per pixel).
• Domain Adaptation: The research highlights that while domain shift (MRI noise vs. clean audio) degrades performance, this can be mitigated by training directly on clean data or using robust alignment strategies.
• Future Impact: This work removes a major barrier to clinical and linguistic applications of articulatory inversion, as it eliminates the dependency on expensive and noisy MRI acquisition for model training and inference.