The NTNU System at the S&I Challenge 2025 SLA Open Track

Technical Summary: The NTNU System at the S&I Challenge 2025

Problem Statement
Spoken Language Assessment (SLA) for second language (L2) learners faces a fundamental trade-off between modeling acoustic delivery and linguistic content. Traditional text-based approaches, such as those utilizing BERT on Automatic Speech Recognition (ASR) transcripts, effectively capture semantic content but fail to model critical prosodic and phonetic cues due to their reliance on error-prone transcriptions. Conversely, acoustic self-supervised models like wav2vec 2.0 (W2V) excel at processing raw audio to capture pronunciation and fluency but often lack the semantic interpretability required for higher-level linguistic assessment. While Multimodal Large Language Models (MLLMs) offer a potential unification of these modalities, their application to holistic SLA scoring remains underexplored. The challenge is to develop a system that robustly integrates acoustic and semantic features under the constraints of limited data and computational resources, as presented in the Speak & Improve (S&I) Challenge 2025.

Methodology
The authors propose a dual-grader system, NTNU SMIL V, which integrates two distinct assessment components through a score fusion strategy:

1. Acoustic Grader (W2V): This component utilizes a pre-trained wav2vec 2.0 encoder to extract frame-level acoustic representations from raw speech waveforms. It employs attention pooling to generate utterance-level vectors, which are then processed through a prototypical embedding module. This module calculates cosine similarity against learnable prototypes representing CEFR levels (A2 to C1+), feeding the result into a Multi-Layer Perceptron (MLP) to predict a continuous proficiency score.
2. Multimodal Grader (MLLM): The second component is built upon the Phi-4-multimodal architecture, a 3.8-billion parameter decoder-only Transformer. This model is fine-tuned using Low-Rank Adaptation (LoRA) with a dedicated audio adapter to process raw audio signals alongside ASR transcripts, task descriptions, and question prompts.
   • The authors investigated two configurations: a Cross-Task-Generalized Multi-modal Grader (Phi4-CTG) trained on combined data from all task parts, and Task-Specific Graders (Phi4-STG) where separate models are fine-tuned for each specific assessment part (Interview, Opinion, Presentation, Communication Activity).
   • The Phi4-STG approach prioritizes direct processing of raw audio waveforms to mitigate ASR error propagation, leveraging the model's ability to jointly reason about speech nuances (pronunciation, fluency, prosody) and linguistic content.
3. Score Fusion: The final system combines the predictions of the W2V grader and the Phi4-STG grader. The MLLM's continuous score is first discretized into eight CEFR-aligned intervals. For each interval, an optimal interpolation weight ($w_k$) is determined via grid search on the development set to minimize the Root Mean Square Error (RMSE) against reference scores. The fused score is calculated as $\hat{y} = (1 - w_k)\hat{y}_{w2v} + w_k\hat{y}_{mllm}$.

Key Contributions

• Integration of Modalities: The paper demonstrates a successful fusion of a self-supervised speech encoder (W2V) with a multimodal large language model (Phi-4), effectively combining fine-grained acoustic modeling with deep contextual reasoning.
• Task-Specific MLLM Adaptation: The study highlights that task-specific fine-tuning (Phi4-STG) outperforms generalized cross-task models (Phi4-CTG) in SLA contexts, suggesting that distinct assessment parts benefit from specialized modeling.
• Robustness to ASR Errors: By incorporating raw audio processing directly into the MLLM pipeline, the system reduces reliance on potentially flawed ASR transcripts, addressing a key limitation of text-only baselines.
• Score-Level Fusion Strategy: The implementation of a score-conditioned fusion mechanism, where weights are dynamically adjusted based on the predicted score interval, allows for optimized integration of the two complementary graders.

Results
The proposed system, NTNU SMIL V, achieved a Root Mean Square Error (RMSE) of 0.375 on the official test set of the Speak & Improve Challenge 2025.

• Ranking: This performance secured second place in the competition.
• Comparison: The system outperformed the official baseline (RMSE 0.444) and the third-ranked entry (RMSE 0.384). It narrowly trailed the first-place system (RMSE 0.364).
• Component Analysis:
  • The standalone W2V model achieved an RMSE of 0.394.
  • The standalone Phi4-STG model achieved an RMSE of 0.389.
  • The fused system (W2V + Phi4-STG) achieved the best result (0.375), demonstrating that the combination of acoustic and semantic modeling yields superior performance compared to either component alone.
  • The fusion of W2V with the generalized Phi4-CTG model yielded an RMSE of 0.377, slightly lower than the task-specific variant, reinforcing the value of task-specific adaptation.

Significance and Claims
The authors claim that their work validates the effectiveness of combining self-supervised speech encoders with MLLMs for spoken language assessment. The results suggest that while acoustic features are foundational for capturing delivery characteristics, the semantic reasoning capabilities of task-specific MLLMs significantly enhance overall scoring accuracy. The paper modestly concludes that while MLLMs offer a promising direction for unifying earlier paradigms, their application in SLA requires careful architectural choices, such as task-specific fine-tuning and direct audio processing, to fully realize their potential. The work emphasizes that future improvements should focus on refining fusion strategies and addressing model efficiency and equity across diverse learner groups.