💬 NLP

Using embeddings to predict spoken word duration and pitch in Mandarin monosyllabic words

This study demonstrates that contextualized embeddings effectively predict both the duration and pitch contours of Mandarin monosyllabic words in spontaneous speech, enabling the accurate reconstruction of empirical f0 contours on a millisecond time scale.

Original authors: Xiaoyun Jin, Mirjam Ernestus, R. Harald Baayen

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoyun Jin, Mirjam Ernestus, R. Harald Baayen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how long a person will hold a note when singing, or how high or low their voice will go, just by knowing the meaning of the word they are about to say. It sounds like magic, but a new study suggests that for Mandarin Chinese, it's actually possible using a specific kind of "digital brain" technology.

Here is a simple breakdown of what the researchers, Xiaoyun Jin, Mirjam Ernestus, and R. Harald Baayen, discovered.

The Big Idea: Words Have "Secret Identities"

Think of every word in a language not just as a sound, but as a character with a personality. In the past, computers treated words like homonyms (words that sound the same but mean different things, like "bank" of a river vs. a "bank" for money) as identical twins. But this study treats them like distinct individuals.

The researchers used a powerful AI tool called GPT-2 (a large language model) to create "contextualized embeddings."

The Analogy: Imagine every word is a person at a crowded party. A standard dictionary definition is like a name tag that just says "Bank." But the AI embedding is like a detailed biography written while the person is actually talking to you. It captures who they are right now, based on who they are talking to and what they are saying.

The Experiment: Guessing the Music

The team took thousands of recordings of people speaking Mandarin naturally. They focused on short, one-syllable words (like "you," "he," or "big"). They wanted to see if the AI's "biography" of a word could predict two things:

Duration: How long the speaker will hold the word.
Pitch: The melody or tune (high or low) of the word.

They treated the AI's "biography" (the embedding) as a map and tried to draw a line from that map to the actual sound recording.

The Results: It Worked (Better Than Random)

The researchers compared their AI predictions against two "control groups" (baselines):

The "Shuffled" Group: They scrambled the word meanings so the AI had no idea what the words meant.
The "Same Word, Different Token" Group: They kept the word meaning but scrambled the specific instances of that word.

What they found:

For Word Length: The AI was surprisingly good at guessing how long a word would be spoken, even for individual instances of a word. It wasn't just guessing the average; it could tell the difference between a quick "hello" and a drawn-out "hello" based on the context.
For Pitch (Tone): The AI could also predict the general shape of the melody (the pitch contour) for a word.
The "Real-Time" Test: The most impressive part was combining these two. The AI predicted the shape of the melody and the length of the word separately. When they combined them to create a full melody in real time (milliseconds), the result was much closer to the actual human speech than random guessing.

The Catch: It's Not Perfect

The paper admits that the AI isn't a crystal ball.

The "Missing Ingredients": The AI is trained on text, not on the physical reality of speaking. It doesn't know if the speaker is tired, angry, or speaking very fast because of a tight schedule. It also doesn't know exactly who the speaker is (a man vs. a woman) or if there is a pause before the word.
The Analogy: Think of the AI as a chef who knows the recipe perfectly but hasn't tasted the specific ingredients in front of them. The dish (the word) will taste mostly right, but it might miss the specific "sizzle" of the moment.

The "Why" Question

The researchers asked: Is the AI actually understanding the meaning, or is it just picking up on speech patterns?

They tested this by asking the AI to guess other things, like:

Who is speaking? (It was okay at this, but not great).
How fast is the person talking? (It was worse at this than at guessing word length).
Is there a pause before the word? (It failed at this).

The Conclusion: The AI is mostly picking up on meaning. The fact that it predicts word length better than it predicts speech speed or pauses suggests that the "personality" of the word itself (its meaning) is deeply tied to how long it takes to say.

The Takeaway

This study shows that in Mandarin, the meaning of a word and its sound are tangled together like two vines growing on the same trellis. You can't fully separate them. By understanding the "contextual personality" of a word, a computer can predict how long a human will hold that note and what the melody will look like, even without hearing the human speak it first.

It proves that the way we speak is not just a mechanical process of moving our mouths; it is deeply influenced by the ideas we are trying to convey.

Technical Summary: Using Embeddings to Predict Spoken Word Duration and Pitch in Mandarin Monosyllabic Words

Problem Statement
Prosodic properties, such as spoken word duration and fundamental frequency ( $f_0$ ) contours, are influenced by a complex array of factors including neighboring words, speech rate, and speaker emotion. Previous research established that word meanings, operationalized via distributional semantics embeddings, can predict the duration of English homophones and the shape of $f_0$ contours in Mandarin. However, it remained unclear whether contextualized embeddings (CEs)—which approximate the meaning of specific word tokens within an utterance context—could predict spoken word duration in Mandarin, and whether these predictions are precise enough to reconstruct $f_0$ contours in real time (milliseconds) rather than just normalized time.

Methodology
The study utilized the Taiwan Mandarin Spontaneous Speech Corpus, focusing on 7,470 tokens of 102 monosyllabic CV words (vowels /a, i, u, @/ without codas or glides) to control for segmental effects.

Data Preparation:
- Contextualized embeddings were generated using GPT-2 (specifically the ckip-transformers Chinese base model) applied to the corpus utterances, resulting in a $7,476 \times 768$ matrix ( $S$ ).
- Target variables were extracted: vowel duration ( $C_v$ ), word duration ( $C_w$ ), and time-normalized $f_0$ contours ( $C_p$ ).
- A subset of 6,118 tokens (93 word types) was used for pitch contour analysis after removing tokens with extremely short durations or octave jumps.
Modeling Approach:
- The study employed the Discriminative Lexicon Model (DLM) framework. Linear mappings ( $W$ ) were estimated by solving the equations $SW_v = C_v$ , $SW_w = C_w$ , and $SW_p = C_p$ to predict durations and contours from embeddings.
- Validation: A 10-fold cross-validation was used.
- Baselines: To ensure predictions were not due to chance, two permutation baselines were implemented:
  - Global Baseline: Randomly permuted word identifiers across all tokens, breaking the link between form and meaning entirely.
  - Type-wise Baseline: Randomly permuted embeddings within each word type. This tests if token-specific embeddings predict token-specific durations better than other tokens of the same type.
Real-Time Reconstruction:
- Predicted $f_0$ contours in normalized time were back-transformed to real time (ms) by multiplying the normalized time points by the predicted word duration.
- Accuracy was evaluated using Dynamic Time Warping (DTW) to measure the distance between observed and predicted contours.

Key Results

Duration Prediction: Contextualized embeddings significantly predicted both vowel and word durations above chance levels.
- Training Data: Empirical correlations were 0.535 (vowel) and 0.557 (word), significantly outperforming both global and type-wise permutation baselines ( $p < 0.0001$ ).
- Held-out Data: Empirical correlations remained significant (0.366 for vowel, 0.399 for word) and superior to baselines.
- Token-Level Significance: Crucially, the empirical results outperformed the type-wise permutation baseline. This indicates that meaning and phonetic realization are entangled even at the level of individual tokens, not just word types.
Pitch Contour Prediction:
- Embeddings predicted the shape of $f_0$ contours in normalized time significantly better than the global baseline.
- However, the type-wise baseline performed slightly better than or equal to the empirical results for pitch shape, suggesting that token-specific meaning does not add significant predictive power for pitch shape beyond the word type level.
Real-Time Contours: When combining predicted shape and duration to generate real-time $f_0$ contours, the empirical predictions were significantly more accurate (shorter DTW distance) than permutation-based predictions. However, the authors note these predictions remain imprecise due to unmodeled prosodic factors (e.g., speaker identity, local speech rate).

Significance and Claims
The paper claims to extend previous findings on English homophones and Mandarin pitch by demonstrating that contextualized embeddings predict spoken word duration in Mandarin at the token level.

Novelty: The primary contribution is the evidence that for duration, semantic meaning and phonetic realization are entangled at the level of individual tokens. This contrasts with pitch contours, where the signal appears to be stronger at the word type level.
Feasibility: The study demonstrates that embedding-derived duration predictions are sufficiently precise to back-transform normalized pitch contours into real-time contours that approximate empirical data better than random baselines.
Limitations and Scope: The authors modestly acknowledge that embeddings capture an "average, speaker-independent" estimate. Factors like speaker identity, emotional state, and local speech rate are not strongly represented in the embeddings, limiting the precision of real-time predictions. The study does not claim to replace models incorporating these prosodic factors but rather establishes the baseline predictive power of semantic context alone.
Theoretical Implication: The results support the view that semantic and phonetic details are more deeply entangled in the mental lexicon than previously assumed, challenging the notion that prosody is governed solely by non-semantic factors.