Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities

Technical Summary: Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech

Problem Statement
Accurate assessment of second-language (L2) English pronunciation is essential for providing personalized feedback and ensuring fair evaluation of learner progress. While Computer-Assisted Pronunciation Training (CAPT) systems have become vital, traditional approaches relying on phoneme-level scoring (e.g., Goodness of Pronunciation) often fail to capture sentence-level fluency, prosody, and completeness. Furthermore, human evaluation is resource-intensive and lacks scalability. Although Large Language Models (LLMs) have transformed text-based assessment, their application to spoken language assessment—specifically for multi-aspect, rubric-aligned scoring in a zero-shot setting—remains an emerging field. This paper addresses the challenge of evaluating whether instruction-tuned speech LLMs can perform holistic, multi-dimensional pronunciation assessment without task-specific fine-tuning.

Methodology
The study evaluates the zero-shot performance of Qwen2-Audio-7B-Instruct, an instruction-tuned, multimodal speech LLM. The model architecture integrates a Whisper-large-v3 based audio encoder (converting raw waveforms to mel-spectrograms) with a Qwen-7B text decoder, enabling direct audio-text interaction.

• Dataset: The evaluation utilizes the Speechocean762 corpus, comprising 5,000 English utterances from 250 Mandarin-speaking learners. The dataset includes sentence-level annotations from five expert raters across four dimensions: Accuracy, Fluency, Prosody, and Completeness (scaled 0–10).
• Prompting Strategy: The model operates in a zero-shot setting, meaning no fine-tuning was performed on the assessment task. Each utterance was paired with a multi-modal prompt containing the raw audio and the expected reference text. The prompt explicitly instructed the model to act as a pronunciation expert, evaluating the speaker based on the four specific rubrics with defined scoring criteria (e.g., 9–10 for "Excellent," 0–2 for "Unintelligible").
• Output Processing: The model was instructed to return scores in a structured JSON format. A regular-expression-based parser extracted the dictionary-style text fragments to map model outputs to the four rubric categories, handling minor formatting deviations while discarding invalid fields.
• Evaluation Protocol: Model predictions were compared against the human-annotated ground truth (GT). Metrics included exact match rates, agreement within ±1 and ±2 point margins, Pearson Correlation Coefficients (PCC), and sensitivity analysis regarding low-quality utterances.

Key Results
The study reveals a nuanced performance profile for the Qwen-Audio-Instruct model:

• Agreement with Human Ratings: The model demonstrated strong agreement with human raters within a ±2 point tolerance, achieving match rates of 87.4% (Accuracy), 89.5% (Fluency), and 87.5% (Prosody). However, exact match rates were relatively low (23.3%–26.1% for accuracy, fluency, and prosody), and Completeness showed significantly lower alignment (0.7% exact match, 55.9% within ±2).
• Correlation: Direct score correlations (PCC) were generally weak (e.g., 0.140 for Accuracy, 0.053 for Fluency). However, when using tolerance-based matches (±1 and ±2), correlations improved, particularly for Accuracy (0.671 at ±2) and Prosody (0.422 at ±2).
• Bias and Sensitivity: The model exhibited a systematic overestimation bias, particularly for low-quality speech. For utterances with ground truth accuracy scores ≤6, the model failed to assign any score in that range. Similarly, it overpredicted scores for fluency and prosody in 95.7% and 89.3% of low-GT cases, respectively.
• Rubric-Specific Performance:
  • Prosody: The distribution of prosody scores visually mirrored human annotations more closely than other rubrics, suggesting the model captures suprasegmental features (intonation, rhythm) effectively.
  • Completeness: This rubric showed the poorest performance. The authors attribute this to ambiguity in the dataset's definition of completeness and a lack of detailed rubric guidance in the prompt, leading to significant discrepancies between human and model scores.

Significance and Claims
The paper claims to be the first work evaluating an instruction-tuned, speech LLM in a zero-shot setting for multi-aspect pronunciation assessment using rubric-based scoring.

• Potential: The findings demonstrate that instruction-tuned speech LLMs can perform coarse-grained, multi-faceted evaluations without adaptation. This capability is highlighted as particularly valuable for rapid prototyping, low-resource deployments, and scenarios where labeled data for fine-tuning is scarce.
• Limitations: The authors modestly note that while the model shows promise for general alignment, it lacks the precision required for fine-grained, calibrated scoring, especially for detecting errors in lower-quality speech. The systematic overestimation and the difficulty in evaluating "completeness" indicate that current zero-shot approaches are not yet a complete replacement for human evaluation or specialized systems.
• Future Directions: The paper suggests that advancing Computer-Assisted Pronunciation Training (CAPT) with these models will require domain-specific fine-tuning, improved prompting strategies to better differentiate low-end performance, and the integration of phonetic-level representations or alignment mechanisms to enhance error detection precision.