Technical Summary: GradeLegal – Automated Grading for German Legal Cases

Problem Statement

Grading German legal examinations is a high-stakes, resource-intensive process characterized by a shortage of qualified graders and growing volumes of student submissions (approx. 115,000 students annually). The task involves evaluating long-form, handwritten case solutions (10–30 pages) on an 18-point ordinal scale. The complexity arises from the need to assess multiple legally coherent reasoning paths, internal consistency, and argumentative soundness, rather than just surface-level correctness. While Large Language Models (LLMs) have shown promise in legal reasoning, their applicability to automated essay scoring (AES) in the German legal domain remains underexplored. Existing studies report mixed results, often showing near-zero agreement in criminal law, and lack systematic comparisons across model families, prompting strategies, and legal subdomains.

Methodology

The authors conducted a systematic evaluation of 27 state-of-the-art LLMs (both proprietary and open-source, including reasoning and non-reasoning variants) on a non-public dataset comprising 87 student case solutions: 71 from a take-home criminal law exam and 16 from a timed public law mock exam.

Experimental Design

The study framed grading as a zero-shot task where models assign a grade (0–18) based on case facts and student solutions. The core variable was the prompting strategy, which incrementally added task-specific information:

1. Task-Agnostic: Minimal instructions to grade.
2. Instr.+Rubric: Instructions plus a grading rubric (mapping legal elements to points).
3. Instr.+Solution: Instructions plus an instructor-written sample solution.
4. Instr.+Rubric+Solution: Full context including instructions, rubric, and sample solution.

Evaluation Metrics

• Primary Metric: Quadratic Weighted Kappa (QWK) to measure inter-rater agreement with expert human grades, penalizing larger score deviations more heavily.
• Secondary Metric: Pearson correlation to assess linear relationship.
• Ensemble Approach: The authors tested whether combining multiple open-source models (using Min and Median aggregation) could outperform the best individual member or closed-source models.

Key Contributions

1. Comprehensive Benchmark: The first systematic evaluation of 27 LLMs (reasoning and non-reasoning, open and closed-source) on German legal exam grading, covering both criminal and public law.
2. Prompt Strategy Analysis: A controlled benchmark of prompting strategies on a non-public corpus to mitigate data contamination concerns, deriving practical guidance for rubric and sample solution usage.
3. Performance Validation: Demonstration that reasoning-oriented LLMs (specifically GPT-5) can approximate expert grading in public law (QWK 0.911) when provided with a rubric and sample solution, matching human inter-rater reliability. In criminal law, a substantial agreement of 0.599 was achieved, exceeding previous reports.
4. Ensemble Effectiveness: Evidence that ensembles of open-source models can outperform their best individual members and narrow the performance gap with top-tier closed-source models, offering a privacy-preserving alternative.

Results

Model Performance

• Reasoning vs. Non-Reasoning: Reasoning-oriented models consistently outperformed non-reasoning models. GPT-5 achieved the highest agreement in both domains (0.911 in public law, 0.599 in criminal law) under the full-context prompt.
• Domain Disparity: Performance was significantly higher in public law than in criminal law across all models. The authors attribute this to the higher complexity and open-ended nature of the criminal law dataset (take-home, longer reasoning chains) compared to the timed, constrained public law exam.
• Open-Source Viability: While open-source models generally lagged behind the best closed-source models, specific models like Mistral-Large-3-675B and Qwen3-Next-80B-Th showed competitive performance. Ensembles of open-source models (e.g., Open-L and Open-S) successfully outperformed their best single members.

Impact of Prompting

• Task-Agnostic Failure: Without specific task information, models performed at or near chance levels (QWK ~0 or negative), indicating "out-of-the-box" grading is insufficient.
• Rubric Importance: The grading rubric proved more critical than the sample solution alone. It provided an explicit mapping from legal quality to the 18-point scale, which models struggled to infer from narrative solutions alone.
• Synergy: The highest performance was achieved when both the rubric and sample solution were provided, suggesting they offer complementary signals (explicit point allocation vs. holistic structural cues).

Ensemble Results

• Ensembles using Median aggregation (for stronger models) and Min aggregation (for moderate models) consistently improved agreement over the best single member.
• Ensembles narrowed the gap to the strongest closed-source models, particularly in the full-context setting, offering a viable alternative for privacy-sensitive deployments.

Significance and Claims

The paper claims that LLM-based grading can serve as a scalable, low-stakes formative tool for student self-assessment and practice feedback in German legal education, provided that:

1. Reasoning-oriented models are selected.
2. Structured task information (specifically a rubric and sample solution) is provided.
3. Domain-specific validation is performed, as performance varies significantly between legal subdomains (criminal vs. public law).

The authors explicitly state that these results should not be used for summative decisions (final exam grading) without further validation. The study highlights that while LLMs can approximate expert judgment, the complexity of criminal law grading remains a significant challenge compared to public law. The work establishes that effective prompt design and model selection are prerequisites for reliable automated grading in this domain.