⚡ electrical engineering

Phoneme-Level Mispronunciation Screening in Polish-Speaking Children with an Explainable Assistant

This paper presents a lightweight, explainable screening pipeline for Polish-speaking children that combines a wav2vec2-based recognizer with a caregiver assistant to detect sibilant mispronunciations, achieving 88.7% sequence accuracy and 72.9% precision in flagging errors while emphasizing safety boundaries and clinician-in-the-loop validation.

Original authors: Milosz Dudek, Daria Hemmerling, Kamil Kwarciak, Maciej Stroinski, Maria Pensko, Mateusz Kowalewski, Leonid Pavlovskyi, Sebastian Jurczak, Anna-Mariia Vitkovska, Zuzanna Miodonska, Natalia Mocko, Micha

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Milosz Dudek, Daria Hemmerling, Kamil Kwarciak, Maciej Stroinski, Maria Pensko, Mateusz Kowalewski, Leonid Pavlovskyi, Sebastian Jurczak, Anna-Mariia Vitkovska, Zuzanna Miodonska, Natalia Mocko, Michal Krecichwost

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A "Silly Sound" Detector for Kids

Imagine a parent trying to help their child learn to say tricky words like "squirrel" or "scissors." In Polish, there are many "hissy" sounds (like s, sz, cz) that are very similar but distinct. Sometimes, a child might mix them up (e.g., saying "s" when they mean "sz").

This paper describes a digital tool designed to act as a helpful first-aid kit for parents. It listens to a child say specific words, checks if they got the "hissy" sounds right, and gives a simple report. Crucially, the authors emphasize that this tool is not a doctor. It doesn't diagnose diseases; it just flags potential trouble spots so a parent knows when to call a real speech therapist.

How the Tool Works (The Three-Step Process)

Think of the system as a three-person team working together:

1. The "Super-Ear" (The Acoustic Model)

First, the system listens to the child. It uses a powerful AI brain (based on a model called wav2vec2) that has been trained specifically on Polish sounds.

The Twist: Standard AI tries to guess the word the child is saying. But this tool is different. It's trained to guess the specific sounds (phonemes) the child made, even if they sounded a little weird.
The "Bracket" Trick: To catch mistakes, the AI has a special list of "suspect sounds." If a child tries to say a "sh" sound but it comes out sounding like an "s," the AI doesn't just say "s." It puts a mental bracket around it: [s]. This tells the system, "Hey, they meant to say 'sh', but it sounded like 's'."

2. The "Matchmaker" (The Alignment)

Next, the system compares what the child actually said against the perfect version of the word.

The Analogy: Imagine lining up two rows of Lego bricks. One row is the "Perfect Word" (the target), and the other is the "Child's Word" (the reality).
The system slides them together to see where they don't match. If the child said [s] where the perfect word needs sh, the system marks that specific spot as a "mismatch." It ignores the rest of the sentence to focus entirely on that one tricky sound.

3. The "Translator" (The Explainable Assistant)

Finally, the system has to talk to the parent. It can't just show a list of technical codes like "Error Type: Place Shift."

The Template: The system uses a pre-written "Mad Libs" style template. It takes the technical data and fills in the blanks to create a friendly, safe message.
The Safety Net: If the system isn't sure (low confidence), it won't guess. Instead, it acts like a cautious librarian and says, "I'm not sure what I heard. Could you please say that word one more time?" It refuses to make medical claims, ensuring parents don't panic over a false alarm.

What Did They Find? (The Results)

The researchers tested this on 10 children they had never seen before (to make sure the tool works on new people, not just the ones it was trained on).

Listening Accuracy: The "Super-Ear" got the sequence of sounds exactly right 88.7% of the time. That's like a student getting an A on a spelling test.
Catching Mistakes: When the system looked specifically for the "hissy" sound errors:
- It caught about 61% of the actual mistakes (Recall).
- When it did flag a mistake, it was right 73% of the time (Precision).
- The Best Part: It rarely cried "wolf." It only flagged a mistake when the child was actually correct 2.7% of the time. This low "false alarm" rate is crucial so parents don't get frustrated by the tool nagging them about perfect words.

Why This Matters (The "Why")

Polish is Hard: Polish has a lot of complex sound clusters. Standard speech apps often fail here because they try to guess the whole word. This tool focuses on the tiny sound differences that matter most for speech therapy.
No "Black Box": Many AI tools are mysterious. This one is "explainable." If it flags an error, it can show a human expert exactly why (e.g., "The child put their tongue in the wrong spot for this sound").
Screening, Not Diagnosing: The authors are very clear: This is a screening tool. It's like a smoke detector. If the smoke detector goes off, you don't assume your house is on fire; you check it. If the tool flags an error, the parent knows to check with a professional.

Summary

The paper presents a specialized AI assistant that helps Polish-speaking parents spot specific sound errors in their children's speech. It uses a "Super-Ear" to hear the sounds, a "Matchmaker" to find the differences, and a "Translator" to give safe, simple advice. It is designed to be accurate, to avoid false alarms, and to always remind the user that a real doctor is needed for a final diagnosis.

Technical Summary: Phoneme-Level Mispronunciation Screening in Polish-Speaking Children with an Explainable Assistant

Problem Statement

Speech sound disorders (SSDs) are prevalent in early childhood, yet timely access to specialist assessment is often constrained by limited capacity and long waiting times. This gap motivates the development of lightweight screening tools for use outside clinical settings. However, applying standard Automatic Speech Recognition (ASR) to child speech is problematic due to acoustic differences (shorter vocal tracts, higher formants), high within-speaker variability, and the presence of systematic non-canonical realizations that are part of normal development. Furthermore, end-to-end ASR systems often rely on strong language model priors that can "normalize" unusual realizations toward likely words, thereby masking minimal-contrast errors essential for detecting mispronunciations.

Polish presents a specific challenge for automatic screening due to its dense consonantal inventory and frequent clusters, particularly regarding sibilants. In Polish clinical practice, sibilants are categorized into three series (sycz ˛acy, cisz ˛acy, and szumi ˛acy), where substitutions (e.g., /ù/ → [s]) and distortions require distinct clinical handling. The goal of this work is not to build a general transcription system, but to create a sensitive, interpretable mechanism to flag likely errors and localize them at the token level for caregiver-guided practice.

Methodology

The authors propose a screening pipeline consisting of three main components: an acoustic recognizer, an alignment-based screening module, and a template-grounded assistant.

1. Acoustic Model and Token Representation
The system utilizes a wav2vec2-based encoder initialized from a Polish checkpoint (jonatasgrosman/wav2vec2-large-xlsr-53-polish). To improve stability in sibilant-heavy segments and provide task-specific context, a 6-layer Transformer post-encoder is added. The model is trained using Connectionist Temporal Classification (CTC).

Token Inventory: Pronunciations are represented as space-delimited IPA-like tokens. Crucially, the vocabulary is extended with bracketed IPA tokens (e.g., [s], [ù]) to represent expert-judged substitution outcomes. These bracketed tokens serve as explicit evidence of deviation.
Adaptation: The model is adapted to child speech and the extended vocabulary using Parameter-Efficient Fine-Tuning (PEFT). Specifically, LoRA modules (rank $r=32$ , $\alpha=64$ ) are injected into attention projections and feed-forward sublayers, and the last 6 encoder layers are unfrozen. This results in 33.3% trainable parameters.

2. Alignment-Based Screening
The system performs a minimum-edit (Levenshtein) alignment between the prompted canonical sequence ( $y_{ref}$ ) and the recognized production sequence ( $\hat{y}$ ).

Error Typing: Alignment operations (Match, Substitution, Insertion, Deletion) are interpreted. For sibilant/affricate targets, substitutions are mapped to interpretable categories (e.g., place-of-articulation shift, voicing alternation, or manner mismatch).
Confidence: A confidence score ( $c_i$ ) is calculated as the mean posterior probability of the recognized token over the frames assigned to it by greedy CTC collapse. This score is used strictly as a suppression signal (e.g., requesting a repeat recording if confidence is low) rather than a calibrated probability.

3. Explainable Assistant
The screening vector (target, realized, type, position, confidence) is fed into a caregiver-facing assistant.

Template Grounding: Feedback is generated from a fixed, reviewable inventory of clinician-authored templates.
Safety Boundaries: The system explicitly avoids clinical diagnosis. It includes refusal rules for uncertainty and does not make medical claims. If confidence is low or evidence is inconsistent, the system requests re-recording rather than providing detailed advice.

Key Contributions

Substitution-Aware Token Recognizer: A wav2vec2-based model with a compact 6-layer Transformer post-encoder, fine-tuned for Polish child speech with an extended token inventory that includes bracketed substitution markers.
Screening-Oriented Evaluation: An evaluation on a held-out set of 10 unseen children that reports both recognition quality and the performance of a conservative screening proxy for target sibilant mismatches.
Template-Grounded Assistant Design: A system architecture that translates token-level screenings into caregiver-friendly feedback with explicit safety boundaries and a plan for clinician-in-the-loop validation.

Results

The system was evaluated on a held-out test set of 10 unseen children (559 utterances) using a fixed inventory of diagnostic prompts.

Recognition Performance: The recognizer achieved an 88.7% exact sequence match (496/559 utterances) and 95.0% token accuracy. The Word Error Rate (WER) was 5.95%, and the Character Error Rate (CER) was 4.09%.
Ablation Studies: Removing the post-encoder reduced the exact sequence match to 84.5%. Using a WavLM-Base baseline instead of wav2vec2 resulted in a lower exact match of 78.6%.
Screening Proxy Performance: Using the presence of a bracketed token at the target position as a flag for mismatch, the system achieved:
- Precision: 72.9%
- Recall: 61.4%
- F1 Score: 0.67
- False Alarm Rate (FPR): 2.7% on target-correct items.
Error Analysis: Among non-exact utterances, 92.1% contained exactly one unit-level substitution. Of these, 60.3% were canonical-bracket confusions, indicating the system often detects the deviation but struggles with the precise categorization between canonical and bracketed labels.

Significance and Claims

The paper claims that SSL-based token recognition can serve as a practical backbone for caregiver-oriented screening in Polish. By focusing on prompted contrasts and aligning to canonical targets, the system avoids the pitfalls of language model priors that mask minimal-contrast errors. The high exact-match rate supports stable downstream alignment and error typing.

The authors emphasize that the system is a screening tool, not a diagnostic one. The significance lies in the design of a "conservative screening proxy" that prioritizes phonetic sensitivity and explainability over general transcription quality. The system successfully flags likely errors with a low false-alarm rate (2.7%), making it suitable for home use where minimizing false positives is critical to avoid unnecessary anxiety.

The paper concludes that while the current results are promising, broader evaluation across ages, severities, and recording conditions is required before deployment. Future work includes expanding coverage beyond substitutions, validating diagnostic utility against clinician outcomes, and conducting user studies to evaluate the safety and clarity of the assistant's feedback.