💬 NLP

AURORA Model of Formant-to-Tongue Inversion for Didactic and Clinical Applications

This paper introduces the AURORA model, a computational framework that predicts tongue shape from vowel formants using data from 40 English speakers, and demonstrates its utility through qualitative evaluation and accessible tools like a Shiny app and real-time biofeedback software for educational and clinical applications.

Original authors: Patrycja Strycharczuk, Sam Kirkham

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Patrycja Strycharczuk, Sam Kirkham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "X-Ray" for Your Voice

Imagine you are trying to learn how to play a musical instrument, like a flute. You can hear the notes you play (the sound), but you can't see your fingers moving inside the holes. If you could see your fingers, you'd know exactly how to move them to hit the right note.

For human speech, we have a similar problem. We can easily measure the sound of a vowel (like the "ee" in beet or the "ah" in father), but we can't easily see what our tongue is doing inside our mouth to make that sound.

The AURORA model is a digital "magic mirror." It takes the sound you make and instantly draws a picture of what your tongue looks like. It bridges the gap between hearing a sound and seeing the physical movement that creates it.

How It Works: The Recipe

The researchers built this model using a giant database of "recipes."

The Ingredients: They recorded 40 people speaking English words. They used special ultrasound cameras (like a sonogram for a baby) to take video of the tongues moving, while simultaneously recording the audio.
The Math: They fed this data into a computer program. The program learned the rules: "When the sound has a high 'F2' frequency, the tongue usually moves forward. When the sound has a low 'F1' frequency, the tongue usually goes up."
The Result: Now, if you type in any two numbers representing a vowel sound, the computer can guess the shape of the tongue that made it.

Think of it like a reverse GPS. Usually, a GPS tells you where you are based on your location. AURORA tells you what your "location" (tongue shape) is based on your "destination" (the sound you hear).

Why Do We Need This? (The Two Main Uses)

The paper suggests two main ways to use this "magic mirror":

1. For Students and Teachers (The "Lightbulb" Moment)

In a classroom, a teacher might say, "To make an 'ee' sound, push your tongue forward." But a student might not know what "forward" feels like.

The Analogy: Imagine trying to explain how to ride a bike by only talking about the physics of balance. It's confusing. But if you have a video showing exactly where the wheels are, it clicks.
The Tool: The authors built a free app (a Shiny app) where you can slide a bar to change the sound, and you instantly see the tongue move. It turns abstract science into a visual game, helping students understand why vowels sound the way they do.

2. For Therapy and Training (The "Video Game" for Your Voice)

This is the most exciting part. Some people need to change how they speak for medical reasons or personal goals.

The Scenario: Think of transgender women undergoing voice training. They often want to sound more feminine, which usually means raising the pitch of their vowels.
The Problem: Telling someone "raise your F2" is like telling a driver "increase your torque." It's too technical. Telling them "move your tongue forward" is better, but they can't see their tongue.
The Solution: The AURORA biofeedback tool acts like a video game HUD (Heads-Up Display).
- You speak into a microphone.
- The screen shows your sound waves.
- Crucially, it also shows a live, moving drawing of your tongue.
- If you want to hit a "target" sound, you can watch your tongue on the screen and adjust it in real-time until the picture matches the goal. It turns voice therapy into a visual feedback loop, making it much easier to learn.

What Are the Limits? (The "Fine Print")

No tool is perfect, and the authors are honest about the flaws:

It only sees the tongue: The model is great at showing the tongue, but it doesn't show the lips or the jaw. If you round your lips (like saying "oo"), the model might get a little confused because it doesn't "know" your lips are doing that.
It's based on a specific group: The data came from people in Northern England. While the physics of speech are mostly the same everywhere, the model might be slightly biased toward how those people speak.
It's a "Best Guess": It's not a medical-grade X-ray. It's a very good estimate based on math, designed to be helpful, not to replace a doctor's diagnosis.

The Bottom Line

The AURORA model is a translator. It translates the invisible, complex movements of your tongue into a simple, visual picture based on the sound you make.

Whether you are a student trying to understand linguistics, or someone trying to change their accent or voice, AURORA takes the guesswork out of speech. It turns "I think I'm doing it right" into "I can see I'm doing it right."

1. Problem Statement

The paper addresses the challenge of bridging the gap between acoustic speech properties (specifically vowel formants, F1 and F2) and articulatory mechanisms (tongue position and shape).

The Complexity Gap: While the general relationship is known (e.g., F1 correlates with tongue height, F2 with tongue advancement), the relationship is non-linear and involves complex interdependencies between tongue height, shape, and acoustic coupling.
Accessibility Issue: Explaining these biomechanical and acoustic complexities to non-specialists (students, clinical clients, or laypeople) is difficult. Traditional explanations often oversimplify the physics, while accurate models are often too computationally complex or require specialized equipment (like MRI or X-ray) to visualize.
Clinical Need: There is a need for accessible tools for speech therapy (e.g., gender-affirming voice training) where clients need to understand how to manipulate their tongue to achieve specific acoustic targets (formants) without needing to understand the underlying physics of resonance.

2. Methodology

Data Collection and Preprocessing

Dataset: The model was trained on synchronized ultrasound tongue imaging (UTI) and audio data from 40 native speakers of English (Northern England dialect).
Stimuli: 3,856 tokens of monosyllabic English words (e.g., bead, bird, bard) embedded in a carrier phrase.
Feature Extraction:
- Acoustic: F1 and F2 values were extracted at the acoustic midpoint of the vowel using FastTrack.
- Articulatory: Tongue contours were tracked using DeepLabCut (DLC), identifying 11 consistent points from the epiglottic vallecula to the tongue tip.
- Dimensionality Reduction:
  - Position/Rotation: Extracted coordinates of the epiglottic vallecula (Knot 1) and tongue tip (Knot 11).
  - Shape: Applied Procrustes analysis (to remove translation, scale, and rotation) followed by Principal Component Analysis (PCA). The first two Principal Components (PC1, PC2) were retained, capturing 67.54% of the shape variance.

Model Architecture (AURORA)

The AURORA (Acoustic Understanding and Real-time Observation of Resonant Articulations) model is a multivariate linear regression model that performs articulatory inversion.

Input: F1, F2 (in Hz), and an interaction term ( $F1 \times F2$ ).
Output: Six articulatory parameters:
1. X and Y coordinates of the epiglottic vallecula (Knot 1).
2. X and Y coordinates of the tongue tip (Knot 11).
3. PC1 and PC2 (representing tongue shape).
Reconstruction Pipeline:
1. Regression: Predicts the six parameters from input formants.
2. Shape Reconstruction: Inverts the PCA to reconstruct the tongue shape coordinates from PC1 and PC2.
3. Normalization Reversal: Uses Procrustes analysis to apply the predicted position, rotation, and scaling (derived from Knot 1 and 11) back onto the normalized shape.
4. Smoothing: Interpolates the 11 discrete points into a smooth 100-point contour using piecewise cubic polynomials.

Tools Developed

Shiny App: An interactive web application allowing users to adjust F1/F2 sliders to visualize predicted tongue shapes and vowel spectra.
Real-time Biofeedback Prototype: A Python/PyQtGraph application that estimates formants from live audio via Linear Predictive Coding (LPC) and maps them to a real-time tongue contour display using a pre-compiled lookup table for low latency.

3. Key Contributions

Inversion Model: A computationally efficient, linear model that successfully predicts 2D tongue displacement and shape from only two acoustic inputs (F1, F2).
Didactic Tool: A visual interface that translates abstract acoustic values into concrete, interpretable articulatory movements, making phonetics accessible to non-experts.
Clinical Application: A prototype biofeedback system that bridges the gap between acoustic goals (e.g., "raise F2") and articulatory instructions (e.g., "move tongue forward"), specifically useful for gender-affirming voice therapy.
Open Source: The authors provide a Shiny app and a Python-based biofeedback tool to facilitate wider adoption in education and therapy.

4. Results and Evaluation

The evaluation was qualitative rather than purely statistical, focusing on whether the model captures the conceptual relationships between formants and articulation.

Visual Inspection: The model successfully differentiated tongue shapes across a continuum of F1 and F2 values.
- High F2: Predicted elongated, straight tongue roots.
- Low F2: Predicted convexity around the tongue dorsum and root.
- Interactions: The model captured complex interactions, such as how high F1 combined with low F2 creates concavity, while low F1 with high F2 creates convexity.
Comparison to Mean Data: When compared to the mean tongue contours of the training data (derived from actual ultrasound), the predictions were a "very close match."
- Discrepancy: The model underestimated the retraction and lowering for the vowel booed. The authors attribute this to the model's lack of lip rounding input, which lowers formants and confuses the inversion logic.
Accuracy vs. Simplicity: The model achieves sufficient accuracy for didactic and clinical purposes while remaining highly interpretable due to its reliance on standard statistical tools (Linear Regression, PCA, Procrustes).

5. Significance and Limitations

Significance

Pedagogical Impact: Provides a tangible way to teach the relationship between acoustics and articulation, moving beyond abstract metaphors.
Clinical Utility: Offers a pathway for real-time biofeedback in speech therapy, potentially improving outcomes for conditions like voice feminization or motor speech disorders by giving clients visual feedback on tongue movement.
Accessibility: Lowers the barrier to entry for articulatory phonetics, allowing linguists, therapists, and students to explore articulatory inversion without complex hardware.

Limitations

Single Articulator: The model only predicts the tongue. It does not explicitly model the jaw, lips, or velum, though it may indirectly capture some jaw movement via tongue displacement.
Lip Rounding: The model fails to account for lip rounding (affecting vowels like booed), leading to prediction errors for rounded vowels.
Data Bias:
- Trained on Northern English speakers, potentially including dialect-specific articulatory settings.
- Trained on non-normalized Hertz values, resulting in a "mean" model that may be more representative of female speakers (due to the dataset composition) and less accurate for specific individual speakers without calibration.
Generalizability: While based on universal physical principles, the model has not been tested on languages other than English.

Conclusion

The AURORA model represents a successful trade-off between accuracy and interpretability. By using a simple linear framework informed by empirical ultrasound data, it provides a robust tool for visualizing the hidden mechanics of speech production. It serves as a foundational step for developing open-source, real-time biofeedback tools that can transform speech therapy and phonetics education.