Technical Summary: Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering

Problem Statement

The integration of Large Language Models (LLMs) into K-12 education offers a transformative opportunity to automate grading and provide standards-based feedback. However, existing automated scoring systems often require extensive data collection and model training, making them inaccessible to individual educators. Furthermore, direct application of commercial foundation models without specific tuning suffers from bias, hallucination, and inconsistent performance, particularly when attempting to implement Standards-Based Grading (SBG). There is a critical gap in understanding how to systematically design an AI grading pipeline that leverages context engineering—the systematic optimization of information payloads beyond simple prompt design—to achieve reliable, equitable, and valid scoring across diverse subjects (Mathematics, Science, and English Language Arts).

Methodology

The study evaluated an AI grading pipeline powered by four commercially available foundation models: Claude Sonnet 4, Claude Haiku 4.5, GPT-5, and GPT-5 Mini. The research utilized a dataset of 822 student responses from the Massachusetts Comprehensive Assessment System (MCAS), covering 166 distinct assessment items across Mathematics (68 items), Science (64 items), and English Language Arts (34 items).

Context Engineering Pipeline

The core methodological innovation was a structured grading pipeline designed to ensure consistent evaluation. For each student submission, the LLM was provided with a "grading bundle" containing:

1. Assessment Item: The complete text of the question.
2. Official Rubric: The specific scoring criteria and performance level descriptors.
3. Exemplar Responses: Anchored answer keys representing different score points.
4. Structured Instructions: Detailed constraints guiding the model's evaluative reasoning.

This architecture aimed to prevent criteria drift by holding the context constant while varying only the student submission, allowing for parallel processing and systematic comparison.

Evaluation Metrics

The study employed a multi-metric approach to validate AI performance against human raters:

• Quadratic Weighted Kappa (QWK): To measure agreement while penalizing larger discrepancies more heavily, accounting for the ordinal nature of scoring scales.
• Proportional Reduction in Mean-Squared Error (PRMSE): To evaluate the utility of the AI score in explaining true score variance, treating human scores as observed variables with inherent uncertainty rather than infallible ground truth.
• Exact and Adjacent Agreement: To quantify strict identity and one-point margin agreements, respectively.

Key Results

1. Domain-Specific Performance Hierarchies

The study revealed a distinct performance hierarchy based on the nature of the academic construct:

• Mathematics: The pipeline achieved exceptional reliability. GPT-5 reached a QWK of 0.951 and PRMSE of 0.946, with an exact agreement rate of 83.9%. Even smaller models like Claude Haiku 4.5 maintained strong performance (QWK 0.832). Errors were predominantly confined to adjacent score points, suggesting the models understood the constructs but faced minor boundary uncertainties.
• Science: Performance remained robust, with GPT-5 achieving a QWK of 0.831 and PRMSE of 0.854. Claude Haiku 4.5 achieved the highest exact agreement at 58.4%. The inclusion of answer keys and specific conceptual rubrics effectively anchored the models to domain standards.
• English Language Arts (ELA): Performance was highly variable and dependent on model capacity.
  • Reading Comprehension: Claude Sonnet 4 achieved exceptional results (QWK 0.890, PRMSE 0.941), while GPT-5 Mini failed significantly (QWK 0.564, PRMSE 0.295).
  • Written Expression: All models struggled with this divergent construct. GPT-5 and GPT-5 Mini showed inadequate measurement utility (PRMSE < 0.30), while Claude Sonnet 4 achieved moderate reliability (QWK 0.755). No single architecture achieved uniform excellence across both ELA dimensions.

2. Model Capacity and Architecture

A clear capability gradient emerged. High-capacity models (GPT-5, Claude Sonnet 4) consistently outperformed reduced-capacity models (GPT-5 Mini, Haiku 4.5), particularly in interpretive tasks. Notably, Claude Sonnet 4 outperformed GPT-5 in ELA reading comprehension, suggesting that intrinsic architectural characteristics and training methodologies influence scoring quality beyond raw parameter counts.

3. Data Attrition and Reliability

The study identified a critical limitation regarding data attrition. Models experienced differential failure rates, particularly with complex or long ELA responses, leading to "survivorship bias" where reported metrics might be artificially inflated for models that failed to process the most difficult items. This highlights that system reliability (completion rates) is as crucial as accuracy for real-world deployment.

Key Contributions

1. Operationalization of Context Engineering: The paper demonstrates that a rigorously designed pipeline—integrating rubrics, exemplars, and metadata into a unified "grading bundle"—can standardize LLM behavior across different architectures and subjects.
2. Cross-Domain Validation: Unlike prior studies focusing primarily on essay scoring, this research provides systematic evidence across Mathematics, Science, and ELA, revealing that LLMs are highly effective for convergent, procedural tasks but struggle with divergent, holistic writing evaluation.
3. PRMSE as a Primary Validity Metric: The study advocates for PRMSE over QWK as a more robust metric for AI scoring, as it accounts for human measurement error and provides a direct measure of the AI's utility in explaining true score variance.
4. Stakeholder Perception Analysis: Feedback indicated strong acceptance of AI-generated narrative feedback for formative purposes but significant skepticism regarding numerical scores for summative evaluation, reinforcing the need for human-in-the-loop frameworks.

Significance and Claims

The authors claim that thoughtfully designed hybrid models combining AI efficiency with teacher judgment represent the most pedagogically sound application of this technology. The study concludes that:

• Formative vs. Summative: LLM graders are currently best suited as formative tools to reduce educator workload and enhance feedback quality, rather than as standalone summative evaluators for high-stakes decisions.
• Subject Suitability: Automated scoring is viable for Mathematics and Science where procedural accuracy can be verified against answer keys, but ELA scoring, particularly for written expression, requires high-capacity models and likely human oversight.
• Context Engineering is Essential: The success of the pipeline demonstrates that the "context" provided to the model (rubrics, exemplars, constraints) is as critical as the model architecture itself.

The paper modestly asserts that while AI can significantly augment educational assessment, it does not yet possess the reliability to displace professional expertise. Instead, it offers a path toward equitable assessment practices where AI handles the "first pass" of feedback, allowing educators to focus on final authoritative judgment and complex instructional interventions.