Psychometric Calibration of Large Language Models for Multidimensional L2 Academic Writing Assessment: The LICA Framework

In the world of language learning, writing an academic essay is a complex act. It is not merely about spelling words correctly or arranging sentences without grammatical errors. To succeed, a student must also develop strong ideas, connect those ideas smoothly so the text flows logically, and adopt a tone that fits the serious, objective expectations of scholarly conversation. For decades, teachers have struggled to grade these essays fairly and consistently. Human judgment is valuable, but it is also slow, expensive, and prone to fatigue, which can lead to different teachers giving different scores for the same work. This has created a demand for computers that can read and grade writing automatically.

Recently, powerful artificial intelligence systems known as large language models have emerged. These are the same types of technology that can hold conversations, write stories, or answer questions. Because they can understand human language so well, researchers began to wonder if they could also act as fair and accurate teachers, grading student essays just as a human would. The hope was that these machines could provide instant feedback to thousands of students at once. However, a critical question remained unanswered: do these machines actually understand what makes writing good, or are they just guessing numbers that happen to look right? To find out, a researcher named Pegah Merrikhi set out to test three of the most advanced AI systems available today against the gold standard of human judgment.

The study focused on a specific framework called LICA, which breaks down the quality of an essay into four distinct parts. The first part is Language, which looks at grammar and vocabulary. The second is Ideas, measuring how deep and well-supported the arguments are. The third is Cohesion, checking how well the sentences and paragraphs stick together. The final part is Academic Appropriateness, which judges whether the writing sounds like a serious scholar speaking, rather than a casual conversation. The researcher took five essays written by second-language learners and asked three different AI systems—GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—to grade each essay on these four scales. At the same time, a human expert provided the "correct" scores. The goal was to see how closely the machines matched the human expert and how they compared to one another.

The results revealed a story of surprising precision mixed with distinct personalities within this specific, small-scale test. One of the AI systems, GPT-4o, performed with perfect accuracy on the five essays used for this preliminary calibration. It matched the human expert's scores exactly on every single essay and for every single category in this specific sample, effectively replicating the human expert's judgment for this limited set of data. The second system, Claude 3.5 Sonnet, was also very close to the human expert, but it showed a slight tendency to be too generous. When grading the flow of ideas and the academic tone, it gave scores that were a tiny bit higher than the human expert would have given. It was not wrong, but it was slightly lenient.

The third system, Gemini 1.5 Pro, showed a different pattern. While it was very good at judging how well paragraphs connected, it tended to be stricter than the human expert. When it came to evaluating the depth of the student's ideas and the appropriateness of the academic tone, it gave scores that were noticeably lower than the human expert's. This suggests that while the machine could see the structure of the writing, it was harder on the content and the voice of the writer.

Perhaps the most revealing finding was not about how the machines compared to humans, but how they compared to each other. When the three AI systems graded the essays without any human help, they agreed almost perfectly on the technical aspects of language and the flow of the text. However, they struggled to agree with each other when it came to judging the academic tone and the quality of the ideas. On the question of whether the writing sounded appropriate for a university setting, the machines often disagreed, with one being generous and another being strict. This indicates that while these systems are excellent at spotting grammar and structure, they do not yet share a single, unified understanding of what makes an academic argument sound right.

The study concludes that these artificial intelligence tools are powerful enough to be useful, but they are not yet ready to replace human teachers entirely. They can handle the mechanical parts of grading with great speed and accuracy, but they still lack the consistent human intuition needed to judge the subtle social and cultural nuances of academic writing. The researcher suggests that the best approach is to use these machines as assistants that provide a first draft of feedback, while keeping a human teacher in the loop to make the final judgment, especially on the more complex aspects of writing. This ensures that students receive the speed of automation without losing the depth of human understanding.