💬 NLP

ICLE++: Modeling Fine-Grained Traits for Holistic Essay Scoring

The paper introduces ICLE++, a new corpus of persuasive student essays annotated with both holistic and trait-specific scores, designed to address the generalization limitations of existing models trained on the ASAP dataset and to facilitate research in multi-trait and cross-prompt automated essay scoring.

Original authors: Shengjie Li, Vincent Ng

Published 2026-07-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shengjie Li, Vincent Ng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers are learning to be teachers, specifically grading essays written by students. This field, known as Automated Essay Scoring (AES), is like a digital grader that reads a student's work and assigns a number to say how good it is. For years, these computer teachers have been trained almost exclusively on one giant pile of essays called "ASAP." Think of ASAP as a massive library filled with essays written by American middle and high schoolers who speak English as their first language. The computer learns the rules of a good essay by studying this specific library. But here's the catch: just because a computer learns to grade American essays perfectly, doesn't mean it will know how to grade essays written by students from other countries, or essays written for different types of assignments. It's like teaching a chef to make the perfect American-style burger and then expecting them to instantly know how to cook a perfect sushi roll without any new training.

To fix this, researchers needed a new, more diverse library of essays to test their computer teachers. They needed a place where they could see if the computer could handle different writers, different topics, and different ways of judging quality. This is where the story of a new dataset called "ICLE++" comes in. It's not just about giving a single score; it's about understanding the why behind the score. Instead of just saying "this essay is a 3 out of 4," the goal is to say "this essay is a 3 because the ideas were a bit messy, but the vocabulary was great." This paper introduces that new library and tests whether the old computer teachers can handle it.

The researchers, Shengjie Li and Vincent Ng, decided to build a new, specialized collection of essays called ICLE++. They gathered 1,006 persuasive essays written by university students from 16 different countries who are learning English as a foreign language. Unlike the old library (ASAP), which had essays of all different lengths, these essays were all written to be roughly the same size (between 500 and 600 words), which removes a tricky variable that might have confused the computers before.

But the real magic of ICLE++ isn't just the essays themselves; it's how they were graded. The researchers didn't just give each essay one overall score. Instead, they broke the grading down into 10 specific traits, like a mechanic checking a car engine part by part. These traits include things like Prompt Adherence (did the student answer the question?), Thesis Clarity (is the main point clear?), Argument Persuasiveness (does the argument convince you?), Development (are the ideas fully explained?), and Technical Quality (are there grammar and spelling errors?).

The team found that these 10 traits are like the ingredients in a recipe. Some ingredients, like Argument Persuasiveness and Development, turned out to be the most important for the final "taste" (the overall score). In fact, they found that if a student had a great argument and well-developed ideas, the essay usually got a high overall score, even if other parts were just okay. However, they also discovered that some traits, like Thesis Clarity, were surprisingly less connected to the final score than one might expect. This suggests that human graders might be forgiving of a fuzzy main point if the rest of the essay is strong, a nuance that simple computer models might miss.

When the researchers tested the best computer models from the old library (ASAP) on this new ICLE++ library, the results were a bit of a shock. The computers, which were champions on the old library, struggled significantly on the new one. Their scores dropped, suggesting that the models had memorized the specific patterns of the American essays rather than learning the universal rules of good writing. It's like a student who memorized the answers to a specific practice test but fails when the questions are phrased differently.

The paper also explored a "what if" scenario: what if the computer could see the scores for all 10 traits before it tried to guess the overall score? When they gave the computer this "cheat sheet" of perfect trait scores, its performance skyrocketed, proving that these 10 traits are indeed very useful for understanding essay quality. However, when the computer had to guess the trait scores itself first, it didn't always help; sometimes, the extra noise from guessing the traits made the overall score worse. This suggests that while knowing the details is powerful, the computer still needs to learn how to find those details accurately on its own.

In short, the paper argues that the old way of testing essay-grading computers on just one type of essay is no longer enough. The new ICLE++ library shows that while computers are getting better, they still have a lot to learn about the subtle, multi-dimensional ways humans judge writing. The researchers believe this new dataset is a crucial step forward, offering a more realistic and challenging playground for future AI teachers to learn how to be fair, detailed, and truly helpful to students everywhere.

Problem
Automated Essay Scoring (AES) research has heavily relied on the ASAP corpus for evaluating models. However, the authors identify significant limitations in this reliance, specifically regarding generalizability, feedback granularity, and cross-prompt applicability. First, models trained on ASAP (composed of native English speakers' essays) may not generalize well to non-native speakers or different essay types. Second, while holistic scores summarize overall quality, they offer little diagnostic feedback; existing trait-based extensions like ASAP++ use coarse-grained traits (e.g., lumping all content-based aspects into a single "CONTENT" score), which limits the ability to provide specific feedback or mimic the human scoring process where distinct content traits are evaluated separately. Third, cross-prompt scoring (training on one set of prompts to score another) is often evaluated on ASAP by mixing different essay genres (e.g., narrative and persuasive) in training and testing, a setup the authors argue is logically inconsistent for assessing generalizability.

Methodology
To address these gaps, the authors introduce ICLE++, a new corpus of persuasive student essays annotated with both holistic scores and fine-grained, trait-specific scores.

Corpus Construction: The dataset consists of 1,006 persuasive essays selected from the International Corpus of Learner English (ICLE). The essays were written by university undergraduates from 16 countries (non-native English speakers) in response to 10 distinct prompts. The essays average 582 tokens and are constrained to a specific length (mostly 500–600 words) to remove essay length as a confounding variable.
Annotation Scheme: The authors defined 10 specific traits: Prompt Adherence, Thesis Clarity, Argument Persuasiveness, Development, Coherence, Cohesion, Organization, Sentence Structure, Vocabulary, and Technical Quality. Unlike ASAP++, which aggregates content traits, ICLE++ scores these four content-based traits individually. Scores range from 1 to 4 in 0.5 increments.
Annotation Process: Six undergraduate native English speakers were trained and hired to annotate the essays. Each essay was graded by two annotators, with discrepancies resolved through discussion. Inter-annotator agreement was measured using Krippendorff's $\alpha$ and Quadratic Weighted Kappa (QWK).
Experimental Evaluation: The authors evaluated state-of-the-art AES models (Uto et al., 2020; Kumar et al., 2022; Chen and Li, 2023) on ICLE++ under two settings: within-prompt and cross-prompt. They compared performance with and without the use of trait scores as features. They also conducted an "oracle" experiment using gold (human-annotated) trait scores to establish an upper bound for trait-informed holistic scoring.

Key Contributions

ICLE++ Corpus: A new, publicly available corpus of 1,006 persuasive essays annotated with holistic scores and 10 fine-grained trait scores, specifically designed for non-native English learners.
Fine-Grained Trait Framework: A detailed annotation scheme that separates content-based traits (e.g., Thesis Clarity, Argument Persuasiveness) which were previously aggregated in other datasets, allowing for more nuanced analysis of the scoring process.
Empirical Analysis: A comprehensive analysis of the correlations between traits and holistic scores, as well as a comparison of feature importance between ICLE++ and ASAP, highlighting that prompt-independent features useful for ASAP are less effective for ICLE++.
Benchmarking: A new benchmark for evaluating AES models on non-native, persuasive essays, including results for both within-prompt and cross-prompt scoring scenarios.

Results

Corpus Characteristics: Inter-annotator agreement was substantial for all traits (Krippendorff's $\alpha$ > 0.6), with Organization and Cohesion showing the highest agreement and Coherence the lowest.
Trait Correlations: Argument Persuasiveness showed the highest correlation with Overall Quality (Pearson Correlation = 0.718), followed by Development and Coherence. Thesis Clarity had the lowest correlation (0.497).
Model Performance:
- Difficulty: Models achieved significantly lower QWK scores on ICLE++ compared to ASAP, indicating ICLE++ is a more challenging dataset.
- Trait Utility: In within-prompt scoring, incorporating predicted trait scores did not consistently improve holistic scoring on ICLE++ and sometimes degraded performance, likely due to the difficulty of accurately predicting traits on this corpus. However, in cross-prompt scoring, using traits showed a generally positive impact.
- Upper Bound: When gold trait scores were used as input, the models achieved very high QWK scores (0.821 for within-prompt and 0.866 for cross-prompt), suggesting that accurate trait scoring could significantly boost holistic scoring performance.
- Feature Analysis: Features that were highly correlated with scores in ASAP (e.g., word types, sentence counts) showed much weaker correlations in ICLE++, suggesting that models trained on ASAP may not transfer well to ICLE++.

Significance
The paper claims that ICLE++ contributes to the set of "much-needed annotated corpora" for AES research. By providing a corpus where essays are written by non-native speakers and scored on fine-grained, content-specific traits, ICLE++ enables the evaluation of model generalizability beyond the native-speaker, coarse-grained constraints of ASAP. The authors posit that the corpus facilitates research into multi-trait scoring and cross-prompt scoring, offering a more realistic and challenging environment for developing AES systems that can provide meaningful, diagnostic feedback to learners. The work is presented as a culmination of long-term annotation efforts and a resource to help bridge the gap between rapid model development and the need for diverse, high-quality evaluation data.

More like this