Cost-efficient generative AI summarization for scalable automated essay scoring in educational assessment

Technical Summary: Cost-efficient Generative AI Summarization for Scalable Automated Essay Scoring

Problem Statement
Automated Essay Scoring (AES) faces a critical bottleneck in processing long-form student essays due to the strict input-length constraints (typically 512 tokens) of standard transformer-based embedding models (e.g., BERT, RoBERTa, DeBERTa). When essays exceed these limits, standard pipelines resort to truncation, which risks discarding educationally relevant information, particularly in high-scoring essays that tend to be longer and more complex. While long-context encoders (e.g., Longformer) and direct Large Language Model (LLM) scoring exist, they often require significant computational resources, task-specific fine-tuning, or introduce challenges regarding prompt sensitivity and score calibration. This study addresses the need for a cost-effective preprocessing strategy that preserves semantic integrity while enabling the use of fixed-length embedding pipelines.

Methodology
The study proposes a hybrid AES framework that utilizes generative AI for adaptive summarization as a preprocessing step. The methodology is structured as follows:

• Dataset: The research utilizes the Learning Agency Lab – Automated Essay Scoring 2.0 dataset (Kaggle), comprising approximately 24,000 argumentative essays with human ratings on a 1–6 scale. The dataset exhibits significant length variation, with higher-scoring essays frequently exceeding 1,000 tokens.
• Generative Summarization: Three GPT-5 model variants (GPT-5, GPT-5 mini, GPT-5 nano) were employed to generate controlled-length summaries. To ensure compliance with downstream token limits, an adaptive iterative summarization strategy was implemented: if a summary exceeded the 512-token limit, the model was re-prompted with a stricter target (80% of the previous limit) until the constraint was met. Prompts enforced high semantic fidelity, requiring the retention of arguments and domain terminology while reducing redundancy.
• Hybrid Feature Representation: The framework integrates two distinct feature sources:
  1. Semantic Embeddings: Summarized texts are encoded using the Qwen3-Embedding-4B model to generate dense, contextual vector representations.
  2. Handcrafted Features: To mitigate information loss during summarization, 22 manually engineered features were extracted from the original, unsummarized essays. These features cover five dimensions: surface statistics, lexical richness, coherence/discourse, readability, and error rates.
• Scoring System: The concatenated features (embeddings + handcrafted) serve as inputs for gradient-boosted decision tree classifiers (XGBoost and LightGBM).
• Evaluation Metrics:
  • Scoring Performance: Measured using Quadratic Weighted Kappa (QWK) to assess agreement with human raters.
  • Summarization Quality: Assessed via lexical overlap (ROUGE-1/2/L), semantic similarity (Cosine Similarity), information retention (Entity Coverage, Keyphrase Overlap, Compression Ratio), and quality control (FactScore, QuestEval, Redundancy).

Key Results

• Downstream Scoring Performance: Contrary to the assumption that larger models yield better downstream results, GPT-5 mini achieved the highest agreement with human raters (QWK = 0.8435), slightly outperforming the full GPT-5 (QWK = 0.8350) and GPT-5 nano (QWK = 0.8332).
• Summarization Quality: The full GPT-5 model demonstrated superior performance across most summarization quality metrics, including ROUGE scores, semantic similarity, entity coverage, and keyphrase overlap. GPT-5 mini followed closely with marginal performance drops, while GPT-5 nano showed more significant declines in content preservation but achieved the highest compression ratio.
• Score-Level Trends: A consistent pattern emerged where summarization quality (semantic similarity, ROUGE, entity coverage) declined as essay scores increased. This suggests that summarizing high-scoring, complex essays is inherently more difficult and prone to information loss than summarizing lower-scoring, shorter essays.
• Cost Efficiency: The study highlights a significant cost-performance trade-off. GPT-5 mini offers near-equivalent or superior AES performance to the full GPT-5 model at a fraction of the API cost (approx. $0.25 vs. $1.25 per 1M input tokens).

Key Contributions and Claims
The paper positions itself not as a new State-of-the-Art (SOTA) benchmark against all existing AES systems, but as a controlled evaluation of a specific preprocessing pathway. Its primary contributions include:

1. Validation of Hybrid Preprocessing: The study demonstrates that generative summarization can serve as an effective bridge between long-form essays and fixed-length embedding pipelines, provided that critical linguistic signals are preserved via handcrafted features from the original text.
2. Model Selection Insights: It challenges the notion that larger models are always superior for AES tasks. The findings suggest that smaller, optimized variants (specifically GPT-5 mini) can achieve better downstream scoring performance, likely by generating summaries that are more efficiently aligned with the needs of the scoring classifier, avoiding the retention of unnecessary detail that larger models might preserve.
3. Identification of Systematic Bias: The research identifies a systematic limitation where summarization quality degrades for higher-scoring essays. This raises concerns about construct validity, as the compression process may disproportionately remove the complex arguments and nuanced structures that define high-quality writing, potentially biasing the system against advanced students.
4. Scalability and Practicality: The study argues that for large-scale educational assessment, the optimal strategy involves balancing model capacity, summary fidelity, and computational cost. The proposed framework offers a scalable, cost-efficient alternative to retraining long-context models or relying on expensive direct LLM scoring.

Limitations and Future Directions
The authors explicitly acknowledge several limitations: the study did not compare against direct truncation, long-context encoders (e.g., Longformer), or direct zero-shot LLM scoring; only one summarization prompt strategy and one embedding model were used; and the results are based on a single dataset. The paper concludes that while the proposed hybrid framework is promising, stronger generalization claims require future work involving comprehensive ablation studies, diverse datasets, and investigations into score-aware summarization methods to mitigate the performance drop observed in high-scoring essays.