💬 NLP

LLM Performance on a Real, Double-Marked GCSE Benchmark

This paper introduces a large dataset of double-marked GCSE exam responses and demonstrates that off-the-shelf large language models can match or even exceed the consistency of human examiners in grading diverse subjects, including handwritten mathematics and subjective English essays, offering a cost-effective solution for automated marking.

Original authors: Malachy Fox, Kavi Samra, Paul Jung

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Malachy Fox, Kavi Samra, Paul Jung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive school exam where thousands of students hand in their answers. Traditionally, you need two human teachers to read every single paper and give it a grade. This is slow, expensive, and sometimes the two teachers disagree on what a "B" or a "C" really looks like.

This paper asks a simple question: Can a computer (specifically, a Large Language Model or "AI") act as a second teacher as well as a real human does?

To find out, the researchers created a giant "test" for AI. Here is the breakdown of what they did and what they found, using some everyday analogies.

1. The "Double-Graded" Test Kitchen

The researchers gathered 32,534 real student answers from mock exams (practice tests for the UK's age-16 national exams).

The Setup: Every single answer was already graded by two different human experts. This is like having two judges on a cooking show who both taste the dish and write down a score.
The Variety: The answers weren't just typed essays. They included messy, handwritten math problems with diagrams, science calculations, and creative writing.
The Goal: They wanted to see if an AI, given the same question and the "answer key" (mark scheme), could give a score that matched the human judges as closely as the two human judges matched each other.

2. The "Taste Test" Results

The researchers fed these questions to various AI models (like GPT-5.5, Gemini, and Claude) using a very simple instruction: "Here is the question, the rules for grading, and the student's answer. Give me a score in a number." They didn't ask the AI to "think hard" or use complex reasoning; they just asked it to do the job.

The Big Surprise:
In almost every subject, the AI didn't just do a "good job." In many cases, the AI agreed with the human judges better than the two human judges agreed with each other.

The Analogy: Imagine two human judges taste a cake. Judge A gives it a 7/10, and Judge B gives it a 6/10. They disagree by one point. Now, the AI tastes the same cake. It gives it a 6.5/10. The AI is actually sitting right in the middle, acting as a perfect tie-breaker.
The Numbers:
- English Essays: The AI was a "super-judge." It matched the human consensus better than the humans matched each other.
- Maths: Even with messy handwriting and complex diagrams, the AI was incredibly accurate, matching human consistency almost perfectly.
- Science: Same story. The AI was just as reliable as a second human teacher.

3. Size Doesn't Always Matter

You might think you need the biggest, most expensive, "super-smart" AI to do this. The paper says no.

The Analogy: It's like trying to fix a leaky faucet. You don't need a master plumber with a $5,000 toolkit; a standard wrench often does the job just as well.
The Finding: Small, cheaper AI models performed just as well as the massive, expensive ones. In fact, for some tasks, a "budget" model was just as consistent as the "premium" one.

4. The "Personality" Quirk (Bias)

While the AI was accurate in how much it agreed with humans, some AIs had a "personality" bias.

The Analogy: Imagine two human judges. One is a "strict teacher" who always gives lower scores, and the other is a "nice teacher" who gives higher scores.
The Finding: Some AI models were naturally "strict" (giving lower scores than the average), while others were "lenient" (giving higher scores). However, once you accounted for this personality, their ability to grade correctly was still excellent. The best models were the ones that were "neutral"—they didn't lean too hard toward being strict or nice.

5. The Cost of the "Second Teacher"

Finally, the paper looked at the price tag.

The Analogy: Hiring a second human teacher to grade 1,000 papers is expensive. Using an AI is like buying a ticket to a movie: it costs a few dollars instead of a hundred.
The Finding: Grading 1,000 papers with an AI costs between $1 and $120, depending on the model. Since the cheaper models performed just as well as the expensive ones, schools could save a fortune while still getting a reliable "second opinion" on every student's work.

The Bottom Line

This paper proves that today's AI is ready to be a reliable "second marker" for school exams. It can read messy handwriting, understand complex math, and grade essays with a level of consistency that often beats human-to-human agreement. It works well, it works cheaply, and it doesn't need to be the biggest, most expensive model to do the job.

What the paper doesn't say:

It does not claim AI should replace human teachers entirely.
It does not say AI is perfect for every single type of exam question (though it did very well on the ones tested).
It does not discuss how this will change schools in the future, only that the technology works right now under these specific test conditions.

Technical Summary: LLM Performance on a Real, Double-Marked GCSE Benchmark

Problem Statement

Schools and exam boards face a significant workload in manually assessing student examinations, particularly for high-stakes national exams like the UK's GCSEs (taken at age ~16). While automated essay scoring has evolved from feature-based systems to neural approaches, existing benchmarks often lack the complexity of real-world examinations. Most school exams span multiple subjects, include handwritten equations and diagrams, and require multimodal processing. Previous studies indicate that off-the-shelf Large Language Models (LLMs) often struggle with these multimodal, messy inputs and have historically fallen short of task-specific systems on standard benchmarks. The core question addressed is whether commercially available LLMs, operating under a generic prompt with minimal reasoning effort, can serve as reliable "second markers" across diverse response types (typed text, handwritten work, and diagrams) compared to human examiner consensus.

Methodology

The authors introduce a new benchmark dataset comprising 32,534 double-marked student responses across 328 questions in five subjects: English Language, Maths, Biology, Chemistry, and Physics.

Data Composition: The dataset includes both typed text and freehand handwriting/drawings captured as strokes. Approximately 15.3% of the total responses contain handwritten input, with Maths having the highest proportion (44.1%).
Ground Truth: Each answer was independently marked by two qualified examiners against published mark schemes. The "Higher" tier was used for Maths and Sciences.
Evaluation Protocol:
- Prompting: Models were evaluated using a single, generic multimodal prompt containing the question, mark scheme, and student answer (including canvas images for handwriting). The instruction was strictly constrained to output a JSON object with a single integer mark.
- Reasoning Effort: Models were run at their "minimum reasoning effort" setting to simulate a cost-effective, low-latency scenario.
- Metric: Agreement was measured using Quadratic Weighted Kappa (QWK), computed per question and averaged weighted by the question's maximum mark. This metric ranges from 0 (chance) to 1 (identical).
- Comparison: The study compares the mean agreement between the AI and the two examiners ( $R_A$ ) against the agreement between the two examiners themselves ( $R_H$ ). The primary metric of success is whether $R_A \geq R_H$ .
- Bias Analysis: For English essays, signed marking bias (predicted mark minus consensus mean) was calculated to assess leniency or harshness.

Key Contributions

First Public Multi-Subject Handwriting Benchmark: This is the first publicly described benchmark for double-marked GCSE responses that includes handwritten student work alongside typed text across five distinct subjects.
Real-World Multimodal Evaluation: Unlike previous benchmarks relying on "born-digital" text, this dataset tests models on the full spectrum of exam inputs, including complex handwritten Maths scripts and diagrams.
Cost-Effectiveness Analysis: The study evaluates the trade-off between model cost (list price per 1,000 papers) and marking consistency, challenging the assumption that larger, more expensive models are necessary for high-quality automated marking.

Results

The study finds that off-the-shelf LLMs can achieve agreement levels with examiners that are comparable to, and in some cases exceed, the agreement between two human examiners.

Overall Agreement: Across all subjects, leading models agreed more closely with the examiner consensus than the examiners agreed with each other.
- English Language: The top model (GPT-5.5) achieved a QWK of 0.75, significantly exceeding the examiner-examiner agreement ( $R_H$ ) of 0.65 ( $\Delta = +0.10$ ).
- Maths: Despite a high baseline examiner agreement of 0.84, the top model (Gemini 3.5 Flash) achieved 0.86 ( $\Delta = +0.02$ ).
- Sciences: The pooled science subjects showed a top model (Gemini 3.1 Pro) achieving 0.74 against an examiner baseline of 0.69 ( $\Delta = +0.06$ ).
Model Size and Cost: Agreement was not strongly correlated with model size or cost. Smaller, cheaper models (e.g., Claude Haiku 4.5, Gemini 3.1 Flash Lite) performed similarly to their larger counterparts, often achieving better performance at a fraction of the cost.
Bias in Essay Marking: While agreement was high, models exhibited varying degrees of bias. Some models (e.g., GPT-5.5, Claude Opus 4.8) were relatively neutral, while others (e.g., Gemma 4 26B, Claude Sonnet 4.6) showed a tendency toward harsher grading.
Subject-Specific Performance:
- Maths: Google's Gemini models appeared more aligned with GCSE marking criteria than OpenAI's models in this domain.
- Science: Models generally achieved examiner-level consistency.

Significance and Claims

The paper claims that current LLMs, when run with a generic prompt and minimal reasoning, are already reliable enough to function as a "second marker" in a real-world educational assessment context. The significance lies in the demonstration that:

Reliability: Automated systems can match or exceed human inter-rater reliability on complex, multimodal tasks including handwritten work.
Cost-Efficiency: High-quality automated marking does not require the most expensive or largest models, offering a viable path for reducing teacher workload.
Generalizability: The performance holds across diverse subjects and input modalities, suggesting that the "one-size-fits-all" generic prompt approach is effective for standardized testing.

The authors maintain modesty regarding their claims, noting that the dataset is derived from mock exams rather than official papers and that the evaluation was limited to minimum reasoning settings. They also acknowledge that the "examiner line" ( $R_H$ ) is based on only two raters, meaning the benchmark measures consistency relative to a specific pair rather than the broader population of examiners. However, the results suggest that the technology is sufficiently mature to provide cost-effective automated marking solutions today.