From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring

This paper presents a comprehensive comparative study of four major LLM-based paradigms for Automated Essay Scoring on IELTS Writing Task 2, revealing that a configuration combining k-SFT and RAG achieves the strongest overall performance with a 93% F1-Score while highlighting critical accuracy-cost-robustness trade-offs.

The Gist

Imagine you are a teacher with a mountain of essays to grade. You have to read every single one, check if the student answered the prompt, if their grammar is good, and if their ideas flow well. It's exhausting, takes forever, and sometimes you might be tired and give a slightly different score than you would have on a fresh day.

This is the problem Automated Essay Scoring (AES) tries to solve: building a robot teacher that can grade essays as fairly and quickly as a human.

For a long time, we tried to build these robots using "old-school" math (looking at word counts and sentence length) and then "deep learning" (teaching computers to recognize patterns). But now, we have Large Language Models (LLMs)—the same super-smart AI brains that power chatbots like the one you're talking to right now.

This paper is a taste test. The researchers wanted to find out: Which way of using these AI brains works best for grading English essays? They didn't just pick one method; they cooked up four different "recipes" and compared them on the same batch of IELTS essays (a tough English test for non-native speakers).

Here is a breakdown of their four recipes, using some fun analogies:

The Four Recipes for the Robot Teacher

1. The "Strict Accountant" (Discriminative Fine-Tuning)

• How it works: You take a standard AI model and force it to memorize thousands of essays and their scores until it becomes a master of spotting patterns. It's like training a dog to sit by repeating "sit" over and over.
• The Vibe: It's reliable but rigid. It's good at math but bad at understanding the feeling of the essay. It's like a calculator that can add numbers perfectly but doesn't understand why you're adding them.
• Result: It was okay, but not great. It missed the nuance.

2. The "Lazy Intern" (Prompting / Zero-Shot)

• How it works: You don't train the AI at all. You just ask it nicely: "Hey AI, please grade this essay like an IELTS examiner." You might even show it two examples first (Few-Shot).
• The Vibe: This is like hiring a brilliant intern who has read the whole library but has never graded a paper before. Sometimes they are amazing because they are smart; other times, they get confused by how you asked the question.
• Result: It was hit-or-miss. Sometimes it got it right, sometimes it was wildly off. Also, using the "smartest" (most expensive) AI models was very costly.

3. The "Specialized Tutor with a Textbook" (Instruction Tuning + RAG)

• How it works: This was the champion of the study.
  • Instruction Tuning: They taught the AI specifically how to be an IELTS examiner, breaking the job down into four specific tasks (Task Response, Coherence, Vocabulary, Grammar).
  • RAG (Retrieval-Augmented Generation): They gave the AI a "cheat sheet" (a database of perfect essays and scoring rules) to look at while it grades.
• The Vibe: Imagine a tutor who has studied the official grading rules inside out and has a stack of "model essays" right next to them for reference. They don't guess; they check the rules and compare the student's work to the examples.
• Result: This was the winner. It got the score right 93% of the time (F1-Score). It was accurate, consistent, and didn't hallucinate (make things up).

4. The "Human-Mimic Coach" (Supervised Fine-Tuning + Preference Optimization)

• How it works: This is the "Specialized Tutor" from Recipe #3, but with an extra step. After learning the rules, they showed the AI examples of "good feedback" vs. "bad feedback" and taught it to prefer the human-like style.
• The Vibe: This AI is like a coach who not only knows the rules but knows how to talk to students. If the student made a mistake, this AI doesn't just say "Wrong"; it says, "Your ideas are a bit scattered here, try connecting them more smoothly."
• Result: It was slightly less accurate at getting the exact number right compared to Recipe #3, but the comments it wrote were much more natural, helpful, and human-like.

The Big Takeaways (The "So What?")

The researchers found a classic trade-off, like choosing between a sports car and a family van:

• The "Specialized Tutor" (Recipe #3) is the Sports Car. It is fast, incredibly accurate, and great for high-stakes testing (like official exams where you need the exact score). It's the best for getting the number right.
• The "Human-Mimic Coach" (Recipe #4) is the Family Van. It might be slightly slower or less precise with the exact number, but it's much better at giving a comfortable, helpful ride. It's perfect for students who need to learn and improve, not just get a grade.

The Cost Factor:
They also looked at how much money and time each method cost.

• The "Lazy Intern" (just asking the AI) was cheap to set up but expensive to run (because you have to pay for the big AI every time).
• The "Specialized Tutor" took some time to train but was very efficient and accurate.
• The "Human-Mimic Coach" took the most time and money to train, and the extra accuracy didn't always justify the huge cost.

The Final Verdict

If you are building a system to grade thousands of essays for a big exam, use the "Specialized Tutor" (Recipe #3). It's the most accurate and reliable.

If you are building a tool to help students practice and learn, use the "Human-Mimic Coach" (Recipe #4). It gives feedback that feels like a real teacher talking to you.

The paper proves that we don't have to choose between "smart AI" and "human-like AI." By combining the right training methods with a "cheat sheet" of examples, we can build robot teachers that are both accurate and kind.

Technical Summary

Here is a detailed technical summary of the paper "From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring."

1. Problem Statement

Automated Essay Scoring (AES) aims to assign proficiency scores to written essays consistent with human raters, a critical task for English as a Second Language (L2) assessments like IELTS and TOEFL. While Large Language Models (LLMs) have reshaped AES, existing research is fragmented. Most studies evaluate individual techniques (e.g., prompting, fine-tuning, or retrieval) in isolation, lacking a systematic comparison of their relative merits, trade-offs, and interactions. Furthermore, there is a gap in understanding how to best align LLMs with complex, multi-dimensional scoring rubrics (Task Response, Coherence, Lexical Resource, Grammatical Accuracy) while balancing accuracy, cost, and robustness.

2. Methodology

The authors propose a unified framework to compare four distinct LLM-based AES paradigms on the IELTS Writing Task 2 benchmark. The study utilizes a consolidated dataset of 10,328 essays (9,833 for training, 495 for testing), featuring prompts, essays, overall band scores, and automatically regenerated analytic scores for the four IELTS criteria.

The four evaluated approaches are:

• Approach 1: Discriminative Fine-Tuning (Baseline)

  • Mechanism: Treats AES as a supervised classification problem. Encoder-based models (RoBERTa, GPT-2 encoder) map the essay (and prompt) directly to a discrete band score (0–9).
  • Limitation: Produces scores only; does not generate explanatory feedback.

• Approach 2: In-Context Learning (Prompting)

  • Mechanism: Uses generative LLMs (GPT-4o, Llama-3-70B, Gemini 2.5 Pro) in Zero-shot and Few-shot settings. The model acts as an examiner based on natural language instructions without parameter updates.
  • Goal: Evaluate the capability of pre-trained reasoning without training costs.

• Approach 3: k-Instruction Tuning with Retrieval-Augmented Generation (RAG)

  • Mechanism: Decomposes the task into k=4 specific criteria (TR, CC, LR, GRA).
  • Technique: Uses Low-Rank Adaptation (LoRA) to fine-tune four independent adapters on the Llama-3.1-8B backbone, each specialized for one criterion.
  • RAG Integration: Retrieves relevant rubric descriptions and exemplar essays to ground the model's inference, reducing hallucinations and stabilizing scores for borderline cases.
  • Output: Aggregates criterion-level scores to produce the final band.

• Approach 4: Supervised Fine-Tuning (SFT) with Direct Preference Optimization (DPO) and RAG

  • Mechanism: A hybrid pipeline.
    1. SFT: Fine-tunes the model to generate structured score-feedback pairs.
    2. DPO: Aligns the model with human preferences using a dataset of preferred vs. dispreferred feedback pairs to improve feedback quality and coherence.
    3. RAG: Integrates retrieved exemplars during inference.
  • Goal: Optimize for both scoring accuracy and the pedagogical quality of the feedback.

3. Key Contributions

1. First Unified Benchmark: The study provides the first comprehensive empirical comparison of four major LLM adaptation paradigms (Discriminative, Prompting, Instruction Tuning, Preference Optimization) specifically for English L2 essay scoring.
2. Criterion-Aware Framework: It introduces a novel k-Instruction Tuning approach that explicitly aligns model predictions with the four official IELTS criteria via specialized LoRA adapters and RAG, rather than treating scoring as a monolithic task.
3. Trade-off Analysis: The paper systematically reveals the accuracy–cost–robustness trade-offs across methods, offering practical guidance for deploying AES systems in different contexts (e.g., high-stakes testing vs. formative learning).

4. Key Results

The experiments yielded the following findings:

• Performance Hierarchy: A clear performance gradient was observed:

  • Discriminative Fine-tuning (Lowest performance, F1 < 0.31).
  • Prompting-based Inference (Variable; closed-source models like Gemini 2.5 Pro performed well but with high cost/variance).
  • Instruction Tuning (Significant improvement; Llama-3.1-8B with few-shot tuning reached F1 ~0.57).
  • k-Instruction Tuning + RAG (Best Overall).
  • SFT + DPO + RAG (Best calibration/feedback quality).

• Top Performance: The k-Instruction Tuning with RAG configuration (Llama-3.1-8B with 4 LoRA adapters) achieved the highest accuracy (0.9902) and F1-Score (0.9350). It effectively grounded predictions in rubric-specific examples.

• Feedback Quality vs. Accuracy:

  • Approach 3 (k-Instruction + RAG) excelled at precise band prediction and fine-grained diagnostic feedback (identifying specific errors per criterion).
  • Approach 4 (SFT + DPO) produced more holistic, examiner-like feedback that aligned better with official descriptors, though it occasionally showed slight deviations in exact band prediction compared to Approach 3.

• Cost-Efficiency:

  • Prompting had the lowest training cost but lower reliability.
  • k-Instruction Tuning + RAG offered the best balance, achieving near-optimal accuracy with moderate computational cost (7.2 hours on a single GPU).
  • SFT + DPO incurred the highest cost (9.5 hours) with diminishing returns on raw accuracy but superior feedback calibration.

5. Significance

This study bridges the gap between theoretical LLM capabilities and practical AES deployment for L2 learners.

• Practical Guidance: It demonstrates that for high-stakes automated scoring requiring high precision, criterion-specific instruction tuning with RAG is the superior strategy.
• Pedagogical Value: For formative assessment where the quality of feedback is paramount, preference optimization (DPO) is essential to generate natural, coherent, and actionable advice.
• Scalability: The work proves that open-source models (Llama-3.1-8B), when properly adapted with LoRA and RAG, can outperform or match expensive closed-source models in specific domains, making robust AES more accessible.

The code and dataset are publicly available, facilitating further research into LLM-based educational assessment.