ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation
This paper introduces ArguLens, an open-source, locally deployable system that addresses the limitations of existing automated essay scoring tools by decomposing the process into a discourse-move classifier, a feature-based scorer, and a label-aware feedback generator, achieving strong performance on the PERSUADE 2.0 dataset while ensuring data privacy and interpretability.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quiet hum of a classroom, a teacher reads a student's argumentative essay, weighing sentences for clarity, logic, and strength. For decades, computers have tried to mimic this human judgment, a field known as automated essay scoring. Early attempts relied on simple counts of words or sentence length, while modern versions use complex neural networks that act like black boxes, offering a grade but rarely explaining why. This creates a dilemma: systems that are accurate often lack transparency, while those that are clear often lack precision. Furthermore, the most powerful tools today usually require sending student writing to distant corporate servers, raising concerns about privacy and cost. The goal for researchers has long been to build a system that is both smart enough to grade accurately and open enough to show its work, all while keeping the data safe within a school's own computers.
A team of researchers at Fudan University has addressed this challenge with ArguLens, a new open-source system designed to be installed locally, keeping student essays off the internet. Instead of trying to guess a grade in one giant leap, the system breaks the task into three distinct, manageable steps. First, it acts as a rhetorical detective, reading the essay sentence by sentence to identify the function of each part. It looks for specific moves in an argument, such as a claim, a piece of evidence, a counter-argument, or a rebuttal. To do this, it uses a specialized version of a large language model that has been trained to recognize these patterns without needing to rewrite the entire essay. Once the system understands the structure of the argument, it moves to the second step: scoring. Here, it does not rely on the heavy language model. Instead, it counts thirty-one specific features, such as the variety of vocabulary used, the complexity of sentence structures, and the frequency of the argumentative moves it just identified. These numbers are fed into a lightweight, highly efficient scoring engine that calculates a final grade. The third step is to generate feedback. Using the scores and the identified argument moves, the system writes a structured report for the student, pointing out strengths and suggesting specific improvements, all while ensuring the advice is constructive and grounded in the actual text.
The researchers tested this system on a large collection of middle-school argumentative essays, a dataset containing thousands of examples written by students across the United States. They found that the system's ability to identify the different parts of an argument was quite strong, correctly labeling the function of sentences in more than eighty-two percent of cases. While it sometimes confused a main claim with the evidence supporting it—a common difficulty even for humans—it successfully distinguished between the different types of arguments most of the time. When it came to assigning a final grade, the system performed with a level of agreement with human teachers that is considered very high in this field. The researchers discovered that including the structural information about the argument moves significantly improved the accuracy of the final score. When they tested the system without this structural knowledge, relying only on vocabulary and sentence patterns, the accuracy dropped noticeably. This suggests that understanding how an argument is built is just as important as knowing what words are used.
One of the most significant aspects of this work is not just the numbers, but the design. The system is built to be transparent and reproducible. Every component, from the code that extracts features to the specific settings used to train the models, is available for anyone to inspect. The researchers were careful to distinguish between what the system can do with perfect information and what it can do in the real world. They showed that if the system knew the exact argument structure beforehand, it could grade with high precision, but they also acknowledged that in a real classroom, the system must predict that structure itself. They measured how long each step took on standard computer hardware, finding that the grading step was nearly instantaneous, while the analysis of argument structure took just over a second for a typical essay. The feedback generation step, which produces the written comments, was designed to be flexible, capable of running on local servers or connecting to external services if needed, though the specific speed of that final step was not measured in this initial release.
The team is clear about the limits of their creation. The system was trained exclusively on essays written by American middle and high school students, and the researchers caution that it should not be used to grade other types of writing, such as creative stories or scientific reports, without further training. They also note that the system has not yet been tested on partial drafts, only on finished essays, which matters if a teacher wants to use it for ongoing guidance during the writing process. Perhaps most importantly, the researchers emphasize that while the system is accurate, it is not a replacement for human judgment in high-stakes situations. It is intended as a tool to assist teachers, offering a second opinion and detailed breakdowns that can help students understand their own writing. By making the entire system open and free to use, the researchers hope to encourage other educators and scientists to build upon their work, testing it in different contexts and languages, and eventually refining the feedback it gives to ensure it helps every learner improve.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.