Goodness-of-pronunciation without phoneme time alignment

1. Problem Statement

Automatic Speech Evaluation (ASE) systems, used for grading oral proficiency in language learning, traditionally rely on frame-synchronous Automatic Speech Recognition (ASR) models. These models are required to:

• Force-align speech with transcripts to generate precise phoneme time boundaries.
• Compute phoneme posteriors (probabilities) for features like Goodness-of-Pronunciation (GOP), Speaking Rate (SR), and Normalized Duration (ND).

The Challenge:

• Data Scarcity: Training robust, frame-synchronous ASR models requires large amounts of paired speech-text data, which is unavailable for low-resource languages.
• Incompatibility with Foundation Models: While open-source, weakly-supervised foundation models (e.g., Whisper) exist for many languages, they are frame-asynchronous and output text tokens (byte-pair encoding) rather than explicit phonemes.
• Current Limitation: The lack of phoneme-level time alignments and explicit phoneme modeling in these foundation models prevents the extraction of standard ASE features, hindering the expansion of speech evaluation to low-resource languages.

2. Methodology

The authors propose a framework to extract ASE features directly from weakly-supervised, frame-asynchronous models (specifically Whisper) without training a language-specific ASR model. The methodology consists of three core innovations:

A. Phoneme-Level Feature Extraction via Confusion Networks (CN)

Since Whisper does not output phoneme posteriors directly, the authors derive them from an N-best list of word hypotheses:

1. Decoding & Normalization: Generate an N-best list of word sequences using beam search. Apply text normalization (casing, punctuation removal, number-to-speech conversion) and transliteration/romanization for non-target scripts.
2. Lexicon Mapping: Convert word sequences into phoneme sequences using a lexicon and a Grapheme-to-Phoneme (G2P) model for out-of-vocabulary words.
3. Confusion Network Construction: Construct a Phoneme-level Confusion Network (CN) from the N-best phoneme list. This network represents competing hypotheses and assumes conditional independence between phoneme positions.
4. Feature Computation: Extract features from the CN:
   • CN-GOP: Log-posterior of the canonical phoneme.
   • CN-GOPmargin: Difference between the canonical phoneme's log-posterior and the most likely alternative.
   • LPP/LPR: Log-posterior vectors and ratios derived from the CN.
   • Note: Duration normalization is omitted for these features as exact phoneme boundaries are unavailable.

B. Word-Level Speaking Rate and Duration

Instead of phoneme-level SR and ND (which require phoneme alignment), the authors compute word-level SR and ND.

• Alignment: Word boundaries are estimated using Whisper's cross-attention weights (via Dynamic Time Warping).
• Mapping: Each canonical phoneme is assigned the SR and ND of the word it belongs to.

C. Alignment-Free Cross-Attention Architecture

Standard ASE models combine per-phoneme features with per-frame Self-Supervised Learning (SSL) features (e.g., from wav2vec 2.0) by mean-pooling frames aligned to each phoneme. Without phoneme alignment, this is impossible.

• Proposed Solution: A Transformer Decoder Cross-Attention layer.
  • Query: Per-phoneme feature sequence.
  • Key/Value: Per-frame SSL feature sequence.
• Refinement: To prevent the model from attending to irrelevant frames, the attention mechanism is restricted to the word level. Each phoneme position is constrained to attend only to frames falling within the time boundaries of its corresponding word (derived from Whisper's attention weights).

3. Key Contributions

1. Decoupling ASE from Target-Language ASR Training: Demonstrates that speech evaluation can be performed using only open-source, multi-lingual foundation models, eliminating the need for expensive, language-specific paired speech-text data.
2. Confusion Network-Based Feature Extraction: Introduces a method to compute phoneme-level posteriors (GOP, LPP, LPR) from word-level N-best lists and CNs, bypassing the need for explicit phoneme modeling in the ASR model.
3. Alignment-Free Feature Fusion: Proposes a cross-attention mechanism that effectively combines per-phoneme and per-frame features without requiring ground-truth phoneme time alignments.
4. Low-Resource Validation: Successfully extends speech evaluation to Tamil (a low-resource language) where no dedicated ASR training data was available, outperforming traditional baselines.

4. Experimental Results

Experiments were conducted on English (speechocean762) and Tamil datasets.

• Feature Quality:

  • Features derived from Whisper (via CN) performed comparably to those from a hybrid frame-synchronous ASR model on English data.
  • Using multiple pronunciation variants in the lexicon improved feature quality; pronunciation priors did not yield significant gains.
  • Confusion Network Decoding: Applying CN decoding to Whisper's N-best list consistently reduced Word Error Rate (WER) compared to selecting the 1-best hypothesis, suggesting CNs improve ASR accuracy even for large foundation models.

• Model Performance (Sentence-Level Pronunciation Accuracy):

  • English: The proposed cross-attention model using Whisper features achieved a Pearson Correlation Coefficient (PCC) of 0.747 and MSE of 1.027, comparable to the hybrid baseline (PCC 0.753, MSE 1.051).
  • Tamil (Low-Resource):
    • The hybrid baseline (trained on internal data) achieved PCC 0.547.
    • The proposed method (Whisper features + Cross-Attention) achieved PCC 0.629 and MSE 0.569.
    • Significance: The proposed method outperformed the hybrid baseline on the low-resource Tamil dataset, proving that leveraging large, multi-lingual pre-trained models is more effective than training small, data-starved target-language models.

• Architecture Impact:

  • Using cross-attention without constraints degraded performance compared to pooling.
  • Word-level attention constraints were crucial, restoring performance to near-baseline levels by forcing the model to learn associations within valid word boundaries.

5. Significance

This paper addresses a critical bottleneck in the democratization of speech technology: data scarcity.

• Accessibility: It enables the deployment of speech evaluation systems for low-resource languages without the prohibitive cost of collecting and annotating large speech-text corpora.
• Scalability: By leveraging existing open-source foundation models (like Whisper), new languages can be supported "out-of-the-box."
• Performance: The results suggest that for low-resource scenarios, the diversity and scale of pre-training data in foundation models outweigh the benefits of fine-tuning a specific ASR model on limited local data.
• Generalizability: The proposed techniques (CN-based feature extraction and cross-attention fusion) are applicable to any frame-asynchronous ASR model, not just Whisper.