LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline

This paper introduces a curriculum-grounded, configurable LLM-as-Judge pipeline that systematically aligns automated question-level marking with authorized educational standards and syllabus artefacts, achieving consistency comparable to human tutors while providing more traceable justifications for exam preparation.

The Gist

Imagine you are a teacher grading hundreds of exam papers. You have a very specific rulebook (the curriculum) and a set of guidelines on how to award points. But you are tired, you have a busy schedule, and sometimes your mood or a student's messy handwriting might accidentally influence your grading. You want to be fair, consistent, and fast, but doing it all by hand is exhausting.

This paper introduces a new way to use Artificial Intelligence (AI) to help grade these exams, but with a very important twist: the AI isn't just guessing; it's following a strict, pre-approved map.

Here is how the system works, broken down into simple concepts:

1. The Problem: The "Black Box" Grader

Usually, when people use AI to grade, they just ask it, "Here is a student's answer, give it a score." This is like asking a chef to cook a meal without giving them a recipe. The AI might make a delicious dish, or it might make something totally different from what the school intended. It might grade based on its own "opinion" rather than the official rules. This is risky for big exams where every point matters.

2. The Solution: The "Curriculum-Grounded" Pipeline

The authors built a system where the AI doesn't just "think" freely. Instead, it acts like a super-organized librarian who has to check every single step against the official library books (the curriculum).

Think of the grading process as a factory assembly line with three main stations:

• Station 1: The Detective (Syllabus Matching)
  Before grading, the system looks at the exam question and asks: "What exactly is this asking?" It doesn't guess. It searches a database of official school rules to find the specific topics, skills, and vocabulary the question is supposed to test. It's like a detective matching a fingerprint to a specific file in a police database.

• Station 2: The Architect (Building the Rubric)
  Once the system knows what the question is about, it builds a custom "scorecard" (rubric) for that specific question. It uses the official dictionary definitions of words (like "explain" or "analyze") to make sure it knows exactly what the student needs to do. It also checks the official "band descriptors" (which describe what a "good" answer looks like vs. a "great" one).

  • Analogy: Imagine a judge in a cooking competition. Instead of just saying "this tastes good," the judge has a specific checklist: "Did they use salt? Did they cook it for 10 minutes? Is the plating correct?" This system builds that checklist automatically for every single question.

• Station 3: The Auditor (Verification)
  Before the AI gives a final score, it runs a self-check. It asks, "Did I actually check the right rules? Did I miss anything?" It ensures the score it gives is backed up by the official documents, not just the AI's gut feeling.

3. How It Handles Mistakes and Nuance

The paper tested this system against human tutors. Here is what they found:

• The Score: The AI gave scores that were very similar to the human tutors.
• The "Why": This is the big difference. When a human tutor says, "You got 3 out of 5," they might just write a quick note. When this AI system says, "You got 3 out of 5," it can point to the exact sentence in the official rulebook that explains why you lost those two points. It's like a GPS that doesn't just tell you you're lost, but shows you the exact map route you missed.

Two Real-Life Examples from the Paper:

1. The Messy Handwriting Case: A student wrote a great answer but with terrible spelling. A human teacher gave it a 0 because they couldn't read it and didn't want to waste time guessing. The AI, however, tried to "read between the lines," figured out the student understood the concept, and gave them 1 point. The AI was more forgiving of the mess but strict on the actual content.
2. The "Discuss" Case: A question asked students to "discuss" a topic. The official rulebook says "discuss" means you must look at both sides of an argument. A student only wrote about the negative side but did it very well. The human teacher gave them a 4/4 (full marks) because the answer was so good. The AI gave them 1/4 because it strictly followed the rule that said, "You missed the other side." This shows the AI is a strict rule-follower, which is good for consistency but might miss the "big picture" brilliance a human sees.

4. The Real-World Test

The team put this system into a real online study platform used by thousands of high school students.

• Result: It worked smoothly. Out of thousands of answers graded, only about 3% were manually changed by a human. This means the system was trusted enough that humans rarely felt the need to step in and fix it.
• Security: The system also successfully blocked students who tried to trick it with "prompt injection" (trying to hack the AI to give high scores), automatically giving them a zero.

The Bottom Line

This paper doesn't say AI is perfect or that it will replace teachers forever. Instead, it shows that if you build AI like a strict, rule-following assistant that is constantly checked against the official school rulebook, it can grade exams fairly and consistently.

It turns the "black box" of AI into a transparent, auditable pipeline. You can look at the AI's work and see exactly which rule it used to give a score, making it a trustworthy tool for helping students prepare for big exams.

Technical Summary

Technical Summary: A Curriculum-Grounded Marking Pipeline for LLM-as-Judge in Education

Problem Statement

The rapid adoption of Generative AI and Large Language Models (LLMs) in education has outpaced the development of robust, verifiable systems for high-stakes assessment. While LLMs show promise in automated marking, relying solely on prompt engineering risks producing outputs that drift from authorized curriculum standards, lack traceability to official guidelines, and fail to capture the tacit knowledge experienced teachers apply during assessment. Current approaches often treat assessment as a model-centric task, neglecting the need for software pipelines that systematically ground model reasoning in authorized syllabus artifacts, performance band descriptors, and marking principles. This gap is critical in high-stakes contexts (e.g., university entrance exams) where consistency, transparency, and alignment with jurisdiction-specific standards are paramount.

Methodology: The Curriculum-Grounded Marking Pipeline

The authors propose a configurable, multi-stage LLM-as-Judge pipeline co-developed with an industrial partner (Studitory) to support exam preparation. Unlike standard RAG (Retrieval-Augmented Generation) systems, this pipeline treats authorized curriculum artifacts as "first-class" inputs rather than auxiliary context. The architecture is designed to be verifiable, auditable, and grounded in the NSW Department of Education's HSC (Higher School Certificate) framework.

The pipeline operates through the following stages:

1. Syllabus Matching and Verification

• Input Processing: The system ingests a question and retrieves relevant "Learn About" (content knowledge) and "Learn To" (cognitive skills) statements from the HSC syllabus.
• Semantic Retrieval: A vector database stores embeddings of syllabus statements. The system performs semantic similarity matching to link the question to relevant topics and subtopics.
• Skill Selection: The number of skills to be assessed is determined by the question's maximum mark allocation (markmax), using a configurable mapping derived from historical exam analysis.
• Coherence Verification: A dedicated verification step calculates "outcome recall" to ensure the selected skills and concepts sufficiently cover the expected learning outcomes. Incoherent mappings are discarded or re-ranked.

2. Marking Criteria Generation

This stage translates curriculum intent into structured, question-specific rubrics through two sub-steps:

• Criteria Structuring: Matched skills and outcomes are clustered into functional assessment dimensions (e.g., conceptual understanding, application). The HSC Keywords Glossary is injected to stabilize the semantic meaning of directive verbs (e.g., "explain," "evaluate") according to official definitions. The markmax value constrains the granularity and distribution of marks.
• Performance Calibration: Structured criteria are mapped to four quality levels (Comprehensive, Sound, Some Understanding, Limited) using authorized Performance Band Descriptors. The granularity of these levels is dynamically adjusted based on markmax (e.g., subdividing "Some Understanding" for high-mark questions).
• Normative Constraints: Thirteen official marking guideline principles (e.g., fairness, allowance for alternative approaches) are embedded as non-negotiable rules to prevent hallucinated norms and ensure partial credit is awarded appropriately.

3. Automated Marking and Feedback Loop

• Scoring: When a student submits a response, the system evaluates it against the generated criteria on a per-criterion basis, supporting partial credit.
• Feedback Generation: Justifications are generated by referencing the specific syllabus skills and performance bands used in the marking decision, ensuring feedback is traceable to learning objectives rather than generic comments.
• Provenance Verification: All intermediate artifacts (matched skills, generated rubrics, scoring logic) are persistently recorded with timestamps and metadata, enabling post-hoc auditing and reconstruction of the marking decision.

Key Contributions

1. Architectural Operationalization: The design and implementation of a curriculum-grounded, configurable LLM-as-Judge pipeline that embeds authorized assessment artifacts as structured context, moving beyond simple prompt engineering to a verifiable software pipeline.
2. Tacit Knowledge Integration: The identification and operationalization of tacit human judgment (e.g., marker heuristics, coherence checks) through structured artifacts and constraints, addressing a critical gap in trustworthy educational assessment.
3. Empirical Validation: A preliminary evaluation demonstrating that the pipeline achieves marking outcomes comparable to human tutors while providing justifications that are explicitly traceable to curriculum intent and official standards.

Preliminary Evaluation Results

The system was evaluated on a dataset of 47 questions marked by a human tutor, using metrics including Lin's Concordance Correlation Coefficient (CCC) and weighted Cohen's kappa.

• Marking Agreement: The pipeline achieved a CCC of 0.81 and a weighted kappa of 0.64 when compared to human marks. Interestingly, direct LLM prompting (without the pipeline's structured workflow) showed comparable or slightly higher numerical agreement with human marks, suggesting LLMs can mimic scoring patterns via internal knowledge. However, the authors argue this does not guarantee alignment with authorized standards.
• Justification Quality: In pairwise comparisons, the pipeline's justifications significantly outperformed both human-generated justifications and direct LLM prompts.
  • Win Rates: The pipeline won 80.4% of comparisons against human justifications and ~69.6% against direct LLM baselines (gpt-5 and gpt-5-nano).
  • Traceability: BLEU score analysis showed the pipeline's justifications had significantly higher lexical overlap with authorized syllabus text, glossary definitions, and performance band descriptors compared to other methods.
• Case Studies: Analysis of specific disagreements highlighted that the pipeline applies a "criteria-first" interpretation tolerant of partial semantic recovery, whereas human markers may apply pragmatic legibility cutoffs. Conversely, the pipeline can be overly rigid regarding directive verb semantics (e.g., "discuss" requiring both sides of an argument) if not guided by nuanced professional practice.
• Operational Deployment: Integrated into the Studitory platform for 5,700+ students, the system processed over 3,000 answers with a manual override rate of only 2.91%, indicating operational usability. The system also demonstrated robustness against prompt-injection attempts by assigning zero marks.

Significance and Claims

The paper claims that the proposed architecture demonstrates how context engineering can operationalize responsible generative AI for assessment. By treating authorized curriculum artifacts and marking principles as first-class components, the pipeline embeds professional judgment at design time while supporting scalable, consistent marking at runtime.

The authors assert that while unguided LLMs may approximate human scores numerically, they lack the guarantees of alignment with authorized curriculum artifacts and fail to produce justifications traceable to official standards. The study concludes that curriculum grounding and verification are critical architectural mechanisms for building trustworthy LLM-as-Judge systems, offering a path toward consistent, transparent, and auditable automated assessment in high-stakes educational contexts. The work positions the pipeline not as a replacement for human oversight, but as a tool that produces moderation-ready, traceable artifacts to support human review and calibration.