Technical Summary: Automated Pronunciation Evaluation for Korean Toddler Speech

Problem Statement

Speech sound disorders (SSD) constitute approximately 44% of pediatric communication disorders in South Korea. Despite the prevalence of these disorders, automated assessment tools specifically designed for Korean toddler speech remain underdeveloped. Existing solutions face three primary limitations:

1. Language and Age Specificity: Most automated models are trained on English-speaking adults under controlled conditions and fail to generalize to Korean phonology or the acoustic characteristics of children aged 2–5.
2. Acoustic Confounds: In Korean home recording environments, caregivers (often young women) frequently use aegyo—a high-pitched, exaggerated speaking style to express affection. The acoustic profile of aegyo (elevated fundamental frequency, high variability) closely mimics toddler speech, causing conventional speaker diarization systems to fail in separating the child from the caregiver.
3. Manual Bottlenecks: Standardized assessments like the Urimal Test of Articulation and Phonology (UTAP) and the digital Hi-DongDong tool rely on manual scoring, which is time-intensive and subject to inter-rater variability.

Methodology

The authors propose an end-to-end automated pipeline that integrates neural speaker diarization with self-supervised learning (SSL) for pronunciation scoring.

1. Dataset Construction

The study introduces a novel, IRB-approved corpus consisting of 53 recordings of Korean-speaking children (aged 2–5) and their caregivers.

• Protocol: Recordings follow the Hi-DongDong prompted-repetition protocol, where a caregiver models a target word and the child repeats it.
• Conditions: Data was collected in naturalistic home settings using personal devices (smartphones, tablets), introducing variability in microphone quality and background noise.
• Annotation: A subset of 34 subjects (with multi-speaker recordings) was annotated by three independent reviewers. This yielded 1,190 consonant and 748 vowel word-level binary correctness labels. Labels were determined by quorum voting (majority rule).

2. Speaker Diarization

To isolate child speech segments, the authors evaluated three pre-trained diarization models:

• Pyannote.audio: Neural segmentation with speaker embedding and clustering.
• SpeechBrain: Utilizes x-vector embeddings and spectral clustering.
• NeMo SortFormer: An encoder-based transformer that employs an Arrival-Time Sorting (ATS) mechanism. This architecture solves the permutation problem in multi-speaker settings by sorting speaker tokens based on their first appearance, training the model to learn temporal ordering jointly with speaker labels.

The performance was measured using Diarization Error Rate (DER) and speaker count accuracy.

3. Pronunciation Scoring (Self-Supervised Learning)

Once child speech segments were isolated, the system evaluated consonant and vowel correctness using a two-stage approach:

• Feature Extraction: A frozen SSL backbone encodes word-level audio segments into frame-level embeddings. Three backbones were tested:
  • wav2vec2-large-xlsr-korean: Korean-specific fine-tuned model.
  • HuBERT-large: Pre-trained on English LibriSpeech with masked discrete unit prediction.
  • WavLM-large: Pre-trained on diverse English audio with a masked speech denoising objective.
• Pooling Strategies: Embeddings were aggregated using Mean, Attention, Statistics, and Multi-layer Fixed-Weight pooling.
• Classification: Pooled vectors were fed into a logistic regression classifier with L2 regularization to predict binary correctness.
• Ensemble Strategy: The authors employed a cross-model ensemble, routing consonant prediction to the best-performing model for that task and vowel prediction to the best for that task, rather than averaging all models.

Key Results

Diarization Performance

• NeMo SortFormer significantly outperformed the other models, achieving 88.69% speaker count accuracy and a 33.04% DER.
• Pyannote and SpeechBrain failed to adequately separate child and caregiver segments (DER > 136%), likely due to the acoustic similarity between aegyo and toddler speech.
• The remaining error in Neomo SortFormer is attributed to overlapping high-pitched sounds, which introduces noise into the downstream classification.

Pronunciation Scoring Performance

• Model Comparison:
  • The Korean-specific wav2vec2 model underperformed (Mean BA: 0.531), suggesting its ASR fine-tuning did not capture the fine-grained acoustic nuances of toddler speech as effectively as general SSL models.
  • WavLM-large excelled at vowel recognition (Balanced Accuracy [BA] = 0.831), likely due to its denoising pre-training objective preserving spectral envelope features.
  • HuBERT-large excelled at consonant recognition (BA = 0.720), likely due to its discrete unit prediction capturing articulatory patterns.
• Pooling: Statistics pooling and Multi-layer Fixed-Weight pooling generally yielded the best results.
• Ensemble Performance: The cross-model ensemble (HuBERT for consonants, WavLM for vowels) achieved a mean balanced accuracy of 0.782 (Consonant BA: 0.720; Vowel BA: 0.845).

Significance and Contributions

The paper claims three primary contributions:

1. Dataset: The creation of a novel, IRB-approved dataset of Korean toddler speech with word-level binary correctness labels, annotated by independent reviewers.
2. Benchmarking: A systematic evaluation of diarization systems on Korean toddler-caregiver recordings, highlighting the specific challenge of aegyo and demonstrating the superiority of arrival-time-sorted transformer architectures in this context.
3. Pipeline: An automated end-to-end pronunciation scoring pipeline that eliminates the need for manual segmentation. The study demonstrates that SSL models pre-trained on adult English speech can transfer meaningfully to Korean toddler assessment without domain-specific fine-tuning.

The authors emphasize that while the residual diarization error introduces some noise, the pipeline successfully automates the segmentation and scoring process, offering a scalable alternative to manual clinical evaluation. The complementary nature of WavLM (vowels) and HuBERT (consonants) suggests a robust "divide-and-conquer" strategy for low-resource pediatric speech assessment.