When alignment is not agreement: Construct-specific validation of large language models for CAF measurement in L2 writing

In the world of second-language learning, teachers and researchers have long relied on a three-part framework to understand how well a student is writing. They look at complexity, which is how intricate and varied the sentence structures are; accuracy, which measures how free the writing is from grammatical mistakes; and fluency, which counts how many words a student produces in a single unit of thought. For decades, measuring these traits required a human expert to read every sentence, count the clauses, and flag every error, a process that is slow, expensive, and difficult to repeat consistently. Now, artificial intelligence offers a new possibility. Large language models, the same powerful computer systems that can write essays or answer questions, can also be asked to count these linguistic features instantly. But a critical question remains: just because a computer agrees with a human on the general ranking of essays, does it actually produce the same numbers? If a teacher uses these numbers to track a student's progress over time, a small, consistent error in the computer's counting could lead to a completely wrong conclusion about whether the student is improving or struggling.

A team of researchers at the University of Maragheh in Iran set out to answer this question by testing six different public artificial intelligence systems against human experts. They gathered 90 argumentative essays written by 45 university students, with each student writing two essays on the same topic about social media, separated by two weeks. The human experts first read every essay and agreed on a single set of numbers for complexity, accuracy, and fluency based on strict rules. Then, the researchers fed the exact same essays into six different AI models, including well-known systems like ChatGPT, Gemini, and others, asking them to apply the same strict rules and return the counts. The goal was not to see if the AI could write a good essay, but to see if it could act as a precise measuring tool, replicating the exact values a human would calculate.

The results revealed a surprising gap between what the AI and humans agreed on and what they actually measured. Every single AI model showed a strong connection to the human scores; when a human rated an essay as having high complexity, the AI also rated it as high. This means the computers were good at understanding the relative order of the essays. However, when the researchers looked at the actual numbers, the computers were often wrong in a specific way. For the measure of complexity, which involves counting how many clauses are packed into a sentence, five out of the six models consistently underestimated the value. They saw fewer complex structures than the humans did. This was not a random mistake; it was a systematic bias where the models seemed to miss the intricate parts of the student writing. In contrast, the models were much better at measuring accuracy. They could reliably identify whether a sentence contained a grammatical error or not, and their numbers were much closer to the human counts.

The study also examined how well the models could detect changes in a student's writing over time. Since the students wrote two essays, the researchers could see if the AI noticed the same shifts in performance that the humans did. The human experts found that in the second essay, the students wrote with significantly fewer grammatical errors but produced shorter, less complex sentences. The AI models successfully copied this pattern for accuracy and fluency, correctly identifying that the students were making fewer mistakes and writing fewer words per sentence. However, when it came to complexity, the models were inconsistent. Some failed to notice the drop in complexity entirely, while others saw a change that was too small. This suggests that while an AI might be useful for spotting general trends in a large group of students, it cannot yet be trusted to provide the exact numbers needed for detailed analysis of sentence structure without human oversight.

Among the six systems tested, one model called DeepSeek performed the best overall, balancing accuracy across all three categories better than the others. Another model, ChatGPT, was particularly strong at measuring accuracy. Yet, the researchers emphasized that no single model was perfect for every task. The performance of the AI depended heavily on what exactly was being measured. A model that is excellent at counting errors might be poor at counting sentence structures. This finding challenges the idea that a high correlation between human and machine scores means the two are interchangeable. It suggests that using artificial intelligence for language assessment requires a careful, specific approach where the tool is validated for the specific job it is being asked to do, rather than assuming it works equally well for everything.

Ultimately, the study concludes that these powerful computer systems are valuable assistants but not replacements for human judgment in this specific field. They can quickly scan large amounts of writing to find general patterns or flag essays that need closer inspection. However, for precise measurements, especially regarding the complexity of sentence structures, the numbers they produce still need to be checked against human standards. The research highlights that in the pursuit of automated language analysis, the difference between moving in the same direction and landing on the same number is a crucial distinction that researchers and teachers must keep in mind.