⚡ electrical engineering

CASA: Content-Acoustic Speaking Assessment with Speech Encoder and Large Language Model

The paper introduces CASA, a lightweight and interpretable multimodal framework combining Whisper-medium and Qwen3.5-2B that achieves state-of-the-art speaking assessment performance on the Speak & Improve Corpus 2025 while effectively separating and analyzing the distinct contributions of acoustic delivery and content quality.

Original authors: Nhan Phan, Ilona Lähteenmäki, Anna von Zansen, Olli-Pekka Pauna, Yaroslav Getman, Tamás Grósz, Mikko Kurimo

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nhan Phan, Ilona Lähteenmäki, Anna von Zansen, Olli-Pekka Pauna, Yaroslav Getman, Tamás Grósz, Mikko Kurimo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where learning a new language feels less like a high-stakes exam and more like a conversation with a helpful friend. For decades, the only way to truly judge how well someone speaks a foreign language was to hire a human expert to listen, take notes, and assign a grade. But humans get tired, they get busy, and they might grade the same answer differently depending on their mood. This is where Automatic Speaking Assessment (ASA) steps in. Think of ASA as a tireless, super-smart robot tutor that listens to your voice and instantly tells you how you're doing. To do this, the robot needs to understand two very different things: content (the actual words you chose and the ideas you expressed) and acoustics (how you said them—your speed, your pauses, your rhythm, and your fluency). The big question in the science world right now is: How do we build a robot that can weigh these two factors perfectly without needing a supercomputer the size of a house to do the math?

Enter CASA, a new approach proposed by researchers from Aalto University and the University of Helsinki. You can think of CASA as a "smart duo" rather than a single giant brain. Instead of using one massive, heavy-duty model to try to do everything at once, CASA splits the job into two specialized teams. One team, built on a tool called Whisper, acts as the "ear," listening strictly to the sound waves, the pauses, and the flow of speech. The other team, powered by a Large Language Model (LLM) called Qwen, acts as the "brain," reading the transcript to understand the meaning and logic of what was said.

The researchers tested this duo on the Speak & Improve Corpus 2025, a massive collection of spoken language tests. They found that CASA achieved a score (Root Mean Square Error, or RMSE) of 0.358. This is a tiny bit better than the previous best score of 0.360, but the real magic isn't just the slight improvement in accuracy. The magic is in the efficiency. While other top-performing systems require a massive amount of computing power (roughly double the parameters), CASA does the same job with about half the resources. It's like winning a race with a sleek, lightweight sports car instead of a heavy truck.

The paper also dug deep to see why this works. They ran many experiments, like swapping out the "ears" for different types of microphones or changing the "brain's" size, and found that simply making the models bigger didn't necessarily make them smarter. In fact, they discovered that the "ear" and the "brain" need to talk to each other in a specific way to get the best results. The "ear" can even give a rough guess at the score based on sound alone, which helps guide the "brain," but the "brain" is still needed to catch the actual content. Interestingly, the system struggled a bit more with very short questions or tasks that required describing a process, suggesting that some types of speaking tasks are just harder for robots to grade than others.

One of the most playful discoveries was that the LLM part of CASA could act as a truth-teller without any extra training. The researchers asked the model to check if a student's answer actually matched the question they were asked. When they swapped the real question for something totally unrelated, like a question about nuclear reactors, the model correctly flagged 99.9% of the answers as "off-topic." This suggests that these AI models can do more than just grade; they can understand the logic of a conversation, something a simple sound analyzer couldn't possibly do.

In short, CASA shows us that we don't need to build bigger, heavier AI monsters to grade speaking tests. By separating the "how" from the "what" and letting two smaller, specialized models work together, we can get results that are just as good, if not slightly better, while using a fraction of the energy. It's a step toward making language learning feedback faster, fairer, and available to everyone, not just those who can afford a human tutor.

Technical Summary: CASA – Content-Acoustic Speaking Assessment

Problem Statement
Automatic Speaking Assessment (ASA) aims to estimate second-language (L2) learners' oral proficiency from spoken responses. While recent advances in speech Large Language Models (LLMs) have improved performance, existing approaches face significant challenges:

Computational Cost: Speech-LLM-based systems often rely on large multimodal backbones, increasing inference costs and limiting practical deployment.
Interpretability: There is limited analysis regarding how acoustic information (delivery) and content information (transcripts) individually and jointly contribute to scoring predictions.
Complexity: Some high-performing non-LLM systems (e.g., Cai et al. [5]) rely on complex pipelines involving multiple graders, handcrafted features, and separate ASR models, hindering portability and reproducibility.
Reproducibility: Many prior implementations are not publicly released, limiting further investigation into the contributions of acoustic versus textual modalities.

Methodology
The authors propose CASA (Content-Acoustic Speaking Assessment), a compact, general-purpose, two-branch architecture designed to explicitly separate speech delivery from speech content.

Architecture Overview:
- Acoustic Branch: Utilizes a frozen Whisper-medium encoder. To capture fine-grained acoustic variations often lost in global average pooling, the model processes audio in 30-second chunks, concatenates encodings, and averages adjacent frame pairs (reducing temporal resolution to 40ms). A two-layer Transformer encoder with Rotary Position Embeddings (RoPE) aggregates these frames, summarized via a [CLS] token.
- Content Branch: Uses a frozen Qwen3.5-2B LLM. The input sequence includes:
  1. Four "acoustic soft tokens" projected from the acoustic branch.
  2. A detached text representation of an auxiliary acoustic CEFR estimate.
  3. The scoring rubric.
  4. Task prompts and ASR transcripts (generated offline).
  5. Three handcrafted fluency statistics: duration, silence ratio, and speech rate.
- Training Mechanism: Both branches utilize LoRA adapters for parameter-efficient fine-tuning. The training objective combines a main Mean Squared Error (MSE) loss on the final fused prediction with an auxiliary loss on the acoustic-only branch. The auxiliary loss employs a tolerance threshold ( $\tau=1$ ), meaning predictions within $\pm1$ score point of the ground truth incur no penalty, preventing the acoustic branch from dominating the learning process despite its inability to fully determine CEFR scores alone.
Dataset: The model was evaluated on the Speak & Improve (S&I) Corpus 2025, which contains approximately 315 hours of speech across four task types (short questions, opinion expression, process description, and opinion-based questions). Scores are based on the CEFR scale (2.0–5.5).

Key Results

Performance: On the S&I test set, CASA achieved a Root Mean Square Error (RMSE) of 0.358 and a Pearson Correlation Coefficient (PCC) of 0.829. This marginally outperforms the previous state-of-the-art (SOTA) RMSE of 0.360 reported by Lin et al. [3].
Efficiency: CASA achieves this performance with approximately 3.13 billion parameters, roughly half the estimated inference parameters of the NTNU speech-LLM system (6.24B) that previously held the SOTA record.
Stability: Repeated runs (10 total) showed a mean RMSE of 0.363 with a 95% confidence interval of [0.359, 0.367], indicating some variability but consistent SOTA-level performance.
Ablation Insights:
- Acoustic vs. Content: The auxiliary acoustic branch provides a small but consistent improvement (reducing mean RMSE by 0.004), demonstrating the complementary value of acoustic features.
- Transcription Quality: Replacing Whisper-medium with CrisperWhisper (a verbatim-transcription model) improved performance on A2-level speakers (RMSE 0.485 vs. 0.553) by capturing disfluencies but degraded performance on B2/C1 speakers, suggesting verbatim transcripts may introduce "inauthentic errors" for fluent speakers.
- Model Capacity: Increasing the LLM size (to Qwen3.5-4B) or using a larger acoustic encoder (Whisper-large-v3) did not improve RMSE, suggesting the current bottleneck is not model capacity.
- Task Specificity: Performance was weaker on Parts 1 and 4 (short questions and process descriptions) compared to Parts 3 and 5 (opinion-based), likely due to fewer discriminative textual cues in constrained tasks.

Significance and Claims
The paper claims that CASA offers a superior accuracy–model-size trade-off, achieving SOTA performance with significantly fewer parameters than existing speech-LLM systems. Its primary contributions are:

Interpretability: By separating acoustic and content branches, the architecture allows for clearer analysis of how delivery and content contribute to scoring.
Simplicity and Portability: The architecture is general-purpose, requiring no dataset-specific structural changes and relying on only three simple fluency features rather than complex handcrafted linguistic features.
Training-Free Content Validation: The authors demonstrate that the LLM branch can perform few-shot content validation at inference time (e.g., flagging answers that do not address the prompt) without additional training, a capability not available to acoustic-only graders.
Reproducibility: The authors have released their code and experimental results to facilitate further investigation into acoustic and linguistic contributions in ASA.

The authors remain modest regarding the magnitude of the accuracy gain, noting that the RMSE improvement over the previous SOTA is small. Instead, they emphasize the value of the efficient, interpretable architecture and the potential for LLMs to support ASA tasks beyond simple scoring, such as providing feedback and validation.

Technical Summary: CASA – Content-Acoustic Speaking Assessment

More like this