💬 NLP

A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales

This paper introduces a rubric-guided SpeechLLM trained with a hybrid objective to jointly perform multi-granular L2 speech assessment and generate natural-language rationales, achieving competitive performance while revealing that rationale faithfulness degrades at finer word and phoneme levels.

Original authors: Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini, Helmer Strik

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini, Helmer Strik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a language teacher listening to a student speak. You don't just want to give them a single grade like "B-"; you want to tell them why they got that grade. Did they stumble on a specific word? Was their rhythm off? Did they pronounce a specific sound incorrectly? And you want to explain all of this in a clear, natural conversation, not just a spreadsheet of numbers.

This paper introduces a new type of AI "teacher" designed to do exactly that. Here is a breakdown of how it works, using simple analogies.

The Problem: The "Black Box" Teacher

Older AI systems for grading language were like opaque vending machines. You put a speech recording in, and a number pops out (e.g., "Fluency: 7/10"). You get the score, but you have no idea why the machine gave you that number. It's a "black box."

Newer systems use Large Language Models (LLMs) that can write explanations. But these are often like improvisational actors: they might write a beautiful, convincing story about why a student did well, but that story might not actually match the specific errors the AI detected. They are "plausible" (they sound good) but not necessarily "faithful" (they don't accurately reflect the data).

The Solution: A "Rubric-Guided" AI Coach

The authors built a SpeechLLM (a Large Language Model that understands speech directly) that acts like a strict, rubric-following coach.

The "Rubric" (The Rulebook): Just like a human teacher uses a grading sheet with specific rules for "Accuracy," "Fluency," and "Prosody" (rhythm/intonation), this AI is trained to follow a strict rulebook.
The "Hybrid" Training (The Double-Check): The AI was trained in two ways:
- Supervised Fine-Tuning (SFT): It learned from examples of good answers, like a student studying a textbook.
- Preference Optimization (BDPO): This is the clever part. The AI was shown pairs of answers: one "good" answer (matching the rulebook) and one "bad" answer (ignoring the rulebook). It was punished for picking the bad one and rewarded for picking the good one.
- The Analogy: Imagine training a dog. First, you show it the right trick (SFT). Then, you give it a choice between the right trick and a wrong trick, and you only give it a treat if it picks the right one (BDPO). This helps the AI learn to be consistent, even when the mistakes are subtle (like confusing "Good" with "Excellent").

What the AI Does (The "One-Stop Shop")

Instead of running three different tests (one for sentences, one for words, one for sounds), this model does everything in one single pass.

Sentence Level: It gives a grade for the whole sentence's accuracy, fluency, and rhythm.
Word Level: It zooms in to grade individual words.
Phoneme Level: It zooms in even further to grade individual sounds (like the "th" in "think").
The Rationale: Finally, it writes a paragraph explaining its grades in plain English.

The Results: Strong on the Big Picture, Weak on the Tiny Details

The researchers tested this AI on a dataset of English speakers learning the language (mostly Chinese speakers). Here is what they found:

The "Macro" View is Great: The AI is excellent at judging the "big picture." If a student's sentence was fluent and had good rhythm, the AI got the score right and wrote a convincing explanation. It was very consistent with its own scores.
The "Micro" View is Tricky: When the AI tried to point out specific errors in individual words or sounds, it got a bit shaky.
- The Analogy: Imagine a detective who is great at solving the whole crime but struggles to point out exactly which fingerprint belongs to the suspect. The AI would sometimes say, "This word was wrong," but it wouldn't always explain why in a way that matched the technical data perfectly.
- Sometimes, the AI would make up a reason based on how a word looks (spelling) rather than how it sounds. For example, it might say, "The second letter of this word is wrong," when the actual error was in the pronunciation.

The Verdict

This paper proves that we can build an AI that acts like a human language teacher: it gives you a score and a reason.

It works well for overall feedback (e.g., "Your speech was a bit choppy").
It struggles a bit with pinpointing tiny, specific sound errors (e.g., "You mispronounced the 'r' in 'car'").

The authors conclude that while this AI is a huge step forward for giving helpful, understandable feedback, we still need to teach it to be more precise when looking at the tiny details of speech. It is a reliable "big picture" coach, but it's still learning how to be a "micro-surgeon" for individual sounds.

Technical Summary: A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales

1. Problem Statement

Automated Second Language (L2) speech assessment systems have advanced in detecting mispronunciations and assigning proficiency scores, yet they often lack interpretability. Traditional approaches typically output opaque numeric metrics or isolated token-level labels, failing to provide actionable pedagogical insights. While Large Language Models (LLMs) have introduced the capability to generate natural-language rationales, standard cascaded pipelines (ASR followed by LLM) fail to ground these rationales in the acoustic signal. Furthermore, existing End-to-End (E2E) SpeechLLMs face three critical gaps:

Lack of Joint Assessment: Systems rarely address comprehensive multi-aspect (accuracy, fluency, prosody) and multi-granular (sentence, word, phoneme) assessment within a single E2E framework.
Data Imbalance: Training reliable models for ordinal scoring is hindered by severe label skew in L2 data, where correct productions and high-rated categories vastly outnumber errors.
Rationale Reliability: Generated rationales may appear plausible but often lack faithfulness to the assigned labels, particularly at fine-grained token levels.

2. Methodology

The authors propose a novel E2E SpeechLLM framework that jointly predicts multi-aspect, multi-granular L2 labels and generates a label-consistent natural-language rationale in a single response.

Model Architecture:

Backbone: The state-of-the-art Qwen2-Audio-7B-Instruct model is used as the base SpeechLLM.
Efficiency: To preserve pretrained acoustic representations and minimize memory, base weights are frozen under 4-bit quantization.
Fine-tuning: Low-Rank Adaptation (LoRA) is applied to the query, value, output, up-projection, and down-projection modules of the decoder layers (rank $r=64$ , scaling factor $\alpha=128$ ), resulting in approximately 115M trainable parameters (~1.6% of the total model).

Hybrid Optimization Objective:
To address label imbalance and ensure rubric adherence, the authors employ a hybrid objective combining Supervised Fine-Tuning (SFT) with preference-based alignment:

Preference Construction: For each input, a "chosen" set of ground-truth ordinal labels ( $y_c$ ) and a "rejected" set ( $y_r$ ) are constructed. Rejected labels are synthesized by perturbing ground truth (e.g., downgrading "Excellent" to "Good"), with an asymmetric distribution (~88% downgraded) to counter "niceness bias."
Bounded Direct Preference Optimization (BDPO): Standard Direct Preference Optimization (DPO) can induce overly aggressive updates when rejected labels are near-misses (e.g., adjacent ordinal ratings). The authors utilize BDPO, which limits the influence of rejected responses by bounding the log-likelihood ratio. This prevents the model from arbitrarily down-weighting plausible alternatives under skewed supervision.
Loss Function: The total loss is defined as $L_{total} = L_{BDPO} + \lambda \cdot L_{SFT}$ , where $\lambda=1.0$ . $L_{SFT}$ is computed via teacher forcing on the target assessment response, while $L_{BDPO}$ optimizes the preference alignment.

Dataset and Prompting:

Data: The SpeechOcean762 (SO762) dataset is used, containing 5,000 English read-speech utterances from Mandarin L1 speakers. The task involves five dimensions: sentence-level accuracy, fluency, and prosody, plus word- and phoneme-level accuracy. Labels are discretized into five ordinal categories (Excellent to Worst).
Prompting: A single rubric-driven prompt elicits all five dimensions and a free-text rationale. The output format is fixed to ensure deterministic parsing of labels (e.g., WORD/Label) followed by the rationale.

3. Key Contributions

Unified Multi-Granular Framework: The paper presents the first systematic study of a single E2E SpeechLLM that simultaneously predicts sentence-level proficiency, word-level accuracy, and phoneme-level accuracy alongside a natural-language rationale.
BDPO for Ordinal Assessment: The application of Bounded Direct Preference Optimization to L2 speech assessment to handle severe class imbalance and adjacent ordinal ratings, improving rubric adherence and label-rationale consistency.
Rationale Reliability Analysis: A dual-axis evaluation of generated rationales measuring:
- Plausibility: Self-consistency between the rationale's sentiment and the model's predicted labels.
- Faithfulness: Alignment between the rationale's specific mentions (words/phonemes) and both the model's predictions and ground-truth human annotations.

4. Results

Experiments were conducted on the SO762 test set using Pearson Correlation Coefficient (PCC), Root Mean Squared Error (RMSE), and Matthews Correlation Coefficient (MCC).

Performance: The multi-granular model achieved strong sentence-level performance (Fluency PCC: 0.73; Prosody PCC: 0.71). While token-level correlation (PCC) decreased for Word (0.52) and Phoneme (0.42) accuracy, the model maintained effective categorization capabilities, evidenced by a higher MCC for Word Accuracy (0.39) compared to Sentence Accuracy (0.35).
Comparison: The multi-granular approach outperformed single-granularity models on Sentence Accuracy and matched or exceeded prior SOTA approaches (including GOPT and Azure PA) on most metrics, except for phoneme-level scoring where specialized ASR-based methods (GOPT) still hold an advantage.
Rationale Analysis:
- Plausibility: The model demonstrated high self-consistency; rationales rarely contradicted the predicted labels (e.g., a "Negative" label with a "Positive" rationale). However, a "positivity bias" was observed relative to ground truth, where the model tended to produce neutral rationales for speech that human annotators rated negatively.
- Faithfulness: Sentence-level rationales showed strong agreement with predictions (PCC ~0.87). However, faithfulness degraded significantly at the token level. References to specific words and phonemes were sparse, and when present, showed weak agreement with both model predictions and ground truth (Phoneme PCC: 0.07 vs. 0.20). Qualitative analysis suggests the model often relies on orthographic heuristics rather than acoustic evidence for fine-grained error localization.

5. Significance and Claims

The paper claims that the proposed E2E SpeechLLM successfully balances holistic and fine-grained assessment without substantial degradation, offering a more interpretable alternative to opaque numeric metrics. The authors assert that BDPO effectively improves rubric adherence and label-rationale consistency, particularly in the context of skewed rating distributions.

However, the authors maintain a modest stance regarding the limitations of the system. They conclude that while SpeechLLMs demonstrate greater reliability for holistic assessment and high-level diagnostic support, they currently fall short in precise token-level error localization. The paper emphasizes that the observed faithfulness at the word/phoneme level is approximate and that advancing fine-grained faithfulness estimation remains a critical direction for future pedagogically useful generative speech assessment systems.