An Effective Strategy for Modeling Score Ordinality and Non-uniform Intervals in Automated Speaking Assessment

Technical Summary: An Effective Strategy for Modeling Score Ordinality and Non-uniform Intervals in Automated Speaking Assessment

Problem Statement
Automated Speaking Assessment (ASA) systems face two primary limitations in current research. First, existing Self-Supervised Learning (SSL) approaches suffer from modality-specific constraints: speech-based SSL models (e.g., wav2vec 2.0) capture acoustic traits but often miss semantic content, while text-based SSL models (e.g., BERT) rely on Automatic Speech Recognition (ASR) outputs, making them susceptible to transcription errors and unable to encode prosodic nuances. Second, most prior methods treat proficiency levels (e.g., CEFR A1, A2, B1) as nominal classes. This approach ignores the inherent ordinal structure of language proficiency and the fact that the intervals between levels are non-uniform (i.e., the progression from B1 to B2 is not equivalent to A1 to A2). Treating these levels as equidistant or unordered leads to suboptimal training objectives and reduced calibration performance.

Methodology
The authors propose a novel ASA framework that addresses these issues through a multi-aspect modeling strategy and a specialized loss function.

1. Multi-Aspect Architecture:
   The model integrates handcrafted features with SSL-derived embeddings using separate Transformer encoders for three distinct scoring aspects:

   • Content Module: Combines prompt embeddings (from a pre-trained BERT model) with frame-level speech features (from wav2vec 2.0) to capture semantic alignment between the learner's response and the prompt.
   • Delivery Encoder: Processes word-aligned segments from ASR outputs using handcrafted prosodic features (pitch, energy, duration, confidence) to model temporal delivery characteristics.
   • Language Use Module: Encodes word-level linguistic annotations (Part-of-Speech, dependency relations, morphological features) derived from ASR transcripts to capture lexical and syntactic structures.
     These three aspect-specific representations are concatenated, projected, and fed into a prediction head.

2. Multi-Margin Ordinal (MMO) Loss:
   To explicitly model the ordinal structure and non-uniform intervals of proficiency labels, the authors introduce a logit-based Multi-Margin Ordinal (MMO) loss. Unlike previous ordinal methods that impose constraints at the hidden representation level, this approach applies pairwise constraints directly at the logit level.

   • The loss function enforces greater separation in logit space for label pairs that are further apart on the CEFR scale.
   • It utilizes a cumulative margin ($d_{y,y_k}$) representing the ordinal distance between the ground-truth label and negative labels.
   • The final objective function combines the MMO loss with standard cross-entropy loss, balanced by a hyperparameter $\lambda$.

Key Contributions
The paper claims two primary contributions:

1. A Multi-Aspect Framework: A system that integrates handcrafted features with SSL embeddings to capture modality-specific characteristics while addressing the ordinal and non-uniform properties of the CEFR scale.
2. The MMO Loss Function: A novel, fully data-driven loss function that is the first to jointly model the ordinal structure and non-uniform level intervals of CEFR-aligned scores in the context of ASA.

Experimental Results
Experiments were conducted on the TEEMI corpus, a dataset of spontaneous spoken English responses annotated with holistic and analytic scores (Content, Delivery, Language Use) aligned to CEFR levels (Pre-A1 to B2).

• Performance on Test Set: The proposed Multi-Aspect (MA) framework outperformed strong baselines, including pure speech-based (W2V), text-based (BERT), and fused (W2V-BERT) models. The addition of the MMO loss (MA+MMO) yielded further improvements, achieving the highest holistic F1 score (35.55%) and significant gains in macro-averaged F1 for Content (+3.17) and Language Use (+8.39) compared to the base MA model.
• Generalization to Unseen Prompts: The MA+MMO model demonstrated robust generalization on unseen tasks (A02, B02, C01). Notably, on the C01 task (thematic Q&A), the model showed substantial improvements in Delivery F1 (+21.19%) and overall F1 (+9.87%) compared to the MA baseline, indicating enhanced adaptability to task variations.
• Ordinal Analysis: Confusion matrix analysis revealed that the MMO loss reduced errors between distant levels and clarified diagonal concentration. Furthermore, distance analysis confirmed that semantic gaps between CEFR levels are non-uniform (e.g., the gap between Pre-A1 and A1 is significantly larger than B1 to B1+), validating the necessity of the proposed non-uniform modeling approach.

Significance
The paper asserts that its proposed framework offers a more interpretable and effective approach to ASA by moving beyond the simplification of proficiency levels as nominal categories. By explicitly modeling the ordinal nature and non-uniform intervals of language proficiency, the method improves both prediction accuracy and the calibration of scores. The authors suggest that this strategy is particularly valuable for real-world ASA scenarios where generalization to unseen prompts and speakers is critical. Future work is envisioned to explore joint training strategies across prompts and the integration of multi-modal large language models (MLLMs) to further enhance comprehensiveness.