Large language model scoring of medical student reflection essays: Accuracy and reproducibility of prompt-model variations

This study demonstrates that large language models can achieve near-perfect accuracy and reproducibility in scoring medical student reflection essays, with fine-tuned models and prompts containing exemplars offering the highest performance while cost-effectiveness varies based on essay volume.

The Gist

Imagine you are a teacher with a mountain of student essays to grade. These aren't just math problems with right or wrong answers; they are reflection essays where students write about their feelings and lessons learned. Grading these is hard work. It takes time, it's expensive to hire enough teachers, and even human teachers get tired, leading to inconsistent scores.

Enter the AI Grader (specifically, Large Language Models like the ones powering chatbots). The question this paper asks is: "Can a robot grade these essays as well as a human, and what's the best way to tell the robot how to do it?"

The researchers treated this like a cooking competition. They had 51 essays (some real, some made up by another AI to test the system) and tried 29 different "recipes" (prompts and settings) to see which one produced the tastiest (most accurate) results.

Here is the breakdown of their findings in plain English:

1. The "Recipe" Matters (Prompt Engineering)

Think of the AI as a very smart but literal-minded sous-chef. If you just say, "Make a cake," it might burn it. You have to give it a specific recipe. The researchers tested different ways of giving instructions:

• The "Full Manual" vs. "Cheat Sheet" (Rubrics):
  • The Finding: Giving the AI the full, detailed rulebook (the scoring rubric) worked best.
  • The Analogy: It's like giving a sous-chef a 10-page recipe with exact temperatures and times. If you only give them a 1-page summary or no instructions at all, the cake (the score) gets messy. The more specific the rules, the better the grade.
• The "Show, Don't Just Tell" (Exemplars/Few-Shot Learning):
  • The Finding: Showing the AI examples of essays that were already graded correctly helped it get better scores.
  • The Analogy: Instead of just saying, "Write a good essay," you show the AI three sample essays and say, "See how this one got a 6? And how this one got a 2? Now grade this new one like that." This "cheat sheet" made the AI much more accurate.
• The "Step-by-Step" Thinking (Chain of Thought):
  • The Finding: Surprisingly, asking the AI to "think step-by-step" or explain its reasoning before giving a score didn't help and sometimes made it worse.
  • The Analogy: It's like asking a chef to narrate every chop and stir while cooking. It slowed them down and didn't make the food taste better. The AI just needed to know the rules, not talk through its process.

2. The "Training" Matters (Fine-Tuning)

• The Finding: If you take a generic AI and "train" it specifically on your grading style (using 18 practice essays), it becomes a grading machine.
• The Analogy:
  • Non-Fine-Tuned AI: Like hiring a general contractor who knows how to build houses but has never built your specific style of house. They do a decent job, but you have to explain everything.
  • Fine-Tuned AI: Like hiring a contractor who has built 100 houses exactly like yours. They know exactly what you want. They are the most accurate, but the "training" cost money upfront.

3. The "Model" Matters (Which Robot?)

• The Finding: Newer, smarter AI models (like GPT-4.1) were much better than older ones. However, the "mini" versions (smaller, cheaper models) were surprisingly good too.
• The Analogy:
  • GPT-4.1: The Ferrari. Fast, powerful, and expensive.
  • GPT-4.1-mini: The reliable Honda Civic. It gets you to the destination (a good grade) just fine, but it costs a fraction of the price.
  • Old Models (GPT-3.5): The horse-drawn carriage. It works, but it's slow and makes mistakes.

4. The Cost vs. Quality Balance

The researchers looked at the price tag for grading 100 essays:

• The "Mini" Model: Cost about $0.04 for 100 essays. (A penny a grade!) It was very accurate.
• The "Full" Model: Cost about $0.21 for 100 essays.
• The "Trained" Model: Cost about $2.00 upfront to train, but if you graded 10,000 essays, the cost per essay dropped so low it became the cheapest option in the long run.

The Verdict:

• If you have a small pile of essays (e.g., 100): Just use the "Mini" model with a clear set of rules. It's cheap and nearly perfect.
• If you have a massive mountain of essays (e.g., 10,000): Spend the time and money to "train" the AI first. It becomes the most accurate and cheapest option in the long run.

The Big Takeaway

AI is now ready to be your grading assistant. It doesn't need to be a genius prompt engineer to get great results.

1. Give it a clear rulebook.
2. Show it a few examples of good and bad grades.
3. Use a modern, slightly cheaper AI model.

The result? You get grades that are almost identical to what a human teacher would give, but for the price of a cup of coffee per 100 essays. The only catch? The AI might be a little too polite or "perfect" in its grammar, so human teachers should still double-check the final work, just to be safe.

Technical Summary

1. Problem Statement

Assessment drives learning, but qualitative narrative assessments (like reflective essays) are difficult to scale due to the high cost, time, and inconsistency of human raters. While Automated Essay Scoring (AES) exists, traditional methods often rely on superficial metrics (grammar, word count) rather than deep content analysis and require significant technical expertise to develop.

Large Language Models (LLMs) offer a potential solution to score narrative assessments with high fidelity. However, there is a lack of evidence regarding:

• The accuracy and reproducibility of LLMs compared to human raters for medical education contexts.
• Best practices for prompt engineering (e.g., use of exemplars, chain-of-thought, rubric specificity).
• The cost-benefit trade-off between different LLM models and prompting strategies.

2. Methodology

Study Design & Data

• Dataset: 51 medical student reflection essays scored on a 6-point "Reflective Ability Rubric."
  • 15 Real Essays: Sourced from a previously validated open-source resource (18 training essays; 3 used as exemplars, 15 for testing).
  • 36 Fabricated Essays: Generated by a separate LLM (Gemini-2.0-flash) to match the quality and reflection depth of the real essays across all 6 scoring levels.
• Reference Standard: Two human investigators (blinded to original scores) established reference scores. Agreement was high (Kappa 0.93).
• Timeline: April–May 2025.

Experimental Conditions
The study systematically compared 29 distinct prompt-model conditions using OpenAI GPT models. Variables included:

• Models: GPT-4.1 (flagship), GPT-4.1-mini, GPT-o4-mini (reasoning), GPT-4o, GPT-4-Turbo, GPT-3.5-Turbo.
• Prompt Engineering Variations:
  • Persona: "Writing evaluator" vs. "Researcher."
  • Rubric Granularity: Full 6-level rubric vs. partial (3 or 1 levels) vs. no rubric.
  • Few-Shot Learning: 0, 1, or 3 exemplars (with/without rubric; full text vs. compressed summaries).
  • Reasoning: Chain-of-thought (general vs. specific) and explanation ordering (explanation before/after score).
  • Temperature: 0.2 (low) vs. 0.7 (default) vs. 1.0 (high).
  • Fine-Tuning: Models were fine-tuned on 18 training essays (5 epochs) and tested with/without rubrics and reasoning prompts.
• Replication: Each condition was run 6 times per essay to measure reproducibility.

Metrics

• Accuracy: Measured against human scores using:
  • Intraclass Correlation Coefficient (ICC): Single-score ICC for agreement.
  • Mean Absolute Difference (MAD): Average absolute difference between LLM and human scores (0 = perfect).
• Reproducibility: ICC calculated across the 6 replications of the same condition.
• Cost & Time: Calculated in USD based on token usage (OpenAI pricing as of May 2025) and API latency.

3. Key Contributions

1. Comprehensive Prompt Engineering Analysis: This is one of the first studies to systematically vary multiple prompt dimensions (persona, rubric depth, exemplars, reasoning, temperature) simultaneously in a medical education context.
2. Fine-Tuning vs. Prompting: It provides empirical data on whether fine-tuning models yields better results than sophisticated prompting (few-shot learning) for scoring tasks.
3. Cost-Benefit Modeling: The study offers a practical framework for educators to choose between models based on volume (e.g., scoring 100 essays vs. 10,000 essays).
4. Synthetic Data Generation: Demonstrated the use of a secondary LLM to generate high-fidelity synthetic essays for research validation, expanding the dataset without compromising ethical standards.

4. Key Results

Accuracy

• Overall Performance: 28 out of 29 conditions (97%) achieved "almost perfect" accuracy (ICC > 0.80) compared to human raters.
• Fine-Tuning: Fine-tuned models were significantly more accurate than non-fine-tuned models (MAD difference: -0.24). The best condition was Fine-tuned GPT-4.1 with rationale (MAD ~0.23).
• Exemplars (Few-Shot): Conditions with exemplars were significantly more accurate than those without (MAD difference: -0.44). 3 exemplars performed similarly to 1 exemplar.
• Rubric Specificity: Accuracy decreased linearly as rubric detail decreased. Full rubrics (6 levels) were significantly better than partial or no rubrics.
• Surprising Findings:
  • Chain-of-Thought (CoT): Contrary to hypotheses, adding CoT prompts (general or specific) did not improve accuracy and slightly increased error rates compared to the baseline.
  • Temperature: Lower temperatures (0.2) did not significantly improve accuracy over the default (0.7).
  • Model Hierarchy: GPT-4.1-mini and GPT-3.5-Turbo (even when fine-tuned) were significantly less accurate than GPT-4.1.

Reproducibility

• Reproducibility was "almost perfect" (ICC > 0.80) for 28/29 conditions.
• Conditions with exemplars tended to have the highest reproducibility.

Cost and Efficiency

• Speed: Average scoring time was ~3.7 seconds per essay.
• Cost (per 100 essays):
  • Lowest: GPT-4.1-mini ($0.04).
  • Baseline (GPT-4.1): $0.21.
  • Few-Shot (3 exemplars): $0.57.
  • Fine-Tuned (GPT-4.1): $2.00 (includes one-time training cost).
• Volume Scaling: When amortized over 10,000 essays, the cost of fine-tuned GPT-4.1 dropped to $0.20 per 100 essays, making it cheaper than non-fine-tuned GPT-4.1 for large-scale tasks.

5. Significance and Implications

• Feasibility of Automation: LLMs can score medical student reflective essays with near-perfect accuracy and reproducibility, validating their use for large-scale educational assessment.
• Prompt Engineering Guidelines:
  • Do: Use a full scoring rubric and include few-shot exemplars for maximum accuracy.
  • Don't: Over-engineer prompts with Chain-of-Thought or complex reasoning steps, as they did not improve performance in this context and increased cost/latency.
• Strategic Implementation:
  • Small Volumes (<100 essays): Use GPT-4.1-mini (low cost, high accuracy) or GPT-4.1 with a baseline prompt (moderate cost, excellent accuracy).
  • Large Volumes (>1,000 essays): Fine-tuned GPT-4.1 becomes the most cost-effective option while maintaining the highest accuracy.
• Future Research: The study suggests that as LLMs become more capable, the need for complex prompt engineering (like CoT) may diminish, and the focus should shift to model selection and fine-tuning strategies.

Limitations: The study used fabricated essays (which may be grammatically superior to real student work) and only tested OpenAI models. Results may vary with other LLM vendors or authentic, messy student writing.