💬 NLP

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

The paper introduces BenGER, a comprehensive benchmark dataset for evaluating LLMs on subsumption-based legal reasoning in German law, demonstrating that closed-flagship models lead performance while human-AI co-creation significantly outperforms unaided human work, all validated through a robust LLM-as-a-Judge framework.

Original authors: Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach, Aleyna Koçak, Anne Zettelmeier, Elly Breu, Angelina Greiner, Sofija Milijas, Matthias Grabmair

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach, Aleyna Koçak, Anne Zettelmeier, Elly Breu, Angelina Greiner, Sofija Milijas, Matthias Grabmair

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a lawyer. But not just any lawyer—one who specializes in the specific, rigid, and highly structured legal system of Germany. In this system, solving a legal problem isn't about guessing the right answer; it's about following a strict recipe called "subsumption."

Think of it like baking a cake where the recipe demands you list every single ingredient, explain exactly why it fits the recipe, and then prove how it mixes with the other ingredients before you can even say, "The cake is done." If you skip a step or just say "It's a cake," you fail, even if the cake tastes good.

This paper, BenGER, is a massive new test designed to see if modern AI (Large Language Models) can actually follow this strict German legal recipe.

Here is the breakdown of what they did, using simple analogies:

1. The Test Kitchen (The Dataset)

The researchers built a giant "test kitchen" called BenGER. It contains three types of challenges:

The Big Exams: 596 long, complex legal cases (like final exams for law students) where the AI has to write a full, structured solution.
The Quick Quizzes: 531 short questions about legal principles.
The "Benchathon" (The Human vs. AI Lab): A special set of 15 new problems where they didn't just ask the AI to solve them. They also asked real humans to solve them in two ways:
1. Solo: Humans working alone with books and the internet.
2. Co-Creation: Humans working with an AI assistant to write the solution.

2. The Judges (How they graded the answers)

Grading legal essays is notoriously tricky. Even human experts often disagree. One professor might give a paper an "A," while another gives it a "C" for the same work.

To handle this, the researchers didn't just ask one person to grade the AI. They built a "Robot Judge" (an LLM-as-a-Judge) that was trained on a very specific rubric (a grading checklist).

The Rubric: Imagine a scorecard with 10 categories, like "Did you find the right law?", "Did you connect the facts to the law?", and "Is your writing organized?"
The Validation: To make sure their Robot Judge wasn't biased or crazy, they compared its scores against a panel of three real human experts who graded the same answers blindly.
The Result: The Robot Judge was surprisingly fair. When they swapped a human grader out and put the Robot Judge in its place, the final group score didn't change much. The Robot Judge was as reliable as a human expert.

3. The Results (Who won?)

They tested 12 different AI systems, ranging from the most powerful "Flagship" models (the super-computers of the AI world) to smaller, faster "Efficiency" models.

The Champions: The big, closed-source "Flagship" models (like those from OpenAI, Anthropic, and Google) came out on top. They scored the highest, often getting "passing grades" that rival top law students.
The Underdogs: The open-source models (free to download) did okay, but they generally trailed behind the big commercial models.
The Magic Combo: The most surprising finding was about Human–AI Co-creation. When humans were allowed to use AI to help them write their answers, their scores skyrocketed. They didn't just do as well as the AI; they did better than humans working alone, and they even beat the AI working alone. It was like a human chef using a high-tech mixer to bake a cake that was better than the chef could make alone or the machine could make alone.

4. The "Gotchas" (What the paper warns us about)

The authors are very honest about the limitations:

Black Box: They tested the AI as it is sold to the public (via an API). They couldn't see the "engine" under the hood, so they don't know exactly why one model is better than another, just that it is better.
The Recipe is Specific: This test is strictly for German Law. The way German lawyers think (the "subsumption" recipe) is different from how lawyers in the US or UK think (who rely more on past cases). You can't take these results and say, "This AI will be a great lawyer in New York."
Memorization Risk: Some of the test questions came from public journals. It's possible the AI just memorized the answers from the internet rather than actually "thinking" through the problem. However, the new "Benchathon" questions (which the AI hadn't seen before) showed similar results, suggesting the AI is actually learning the logic, not just cheating.

The Bottom Line

This paper says: "We built a strict, fair test for German legal reasoning. The best AI models are now very good at following the rules, but they aren't perfect yet. However, when humans team up with AI, they become super-lawyers, outperforming both humans and AI working alone."

They also proved that a smart Robot Judge can grade these essays almost as reliably as a room full of human professors, which could help scale up legal education and testing in the future.

Technical Summary: BenGER – Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

Problem Statement

Evaluating Large Language Models (LLMs) on legal reasoning is inherently difficult due to the "noisy" nature of human expert judgment. In German legal education and practice, grading exhibits substantial variability even among experts; identical answers can receive markedly different scores depending on the grader. This challenges the creation of reliable benchmarks, as a single "ground truth" label is often an oversimplification of a distribution of plausible expert assessments.

Furthermore, existing legal NLP benchmarks predominantly focus on classification tasks, short-form question answering, or common-law jurisdictions (e.g., US, UK) that rely on precedent-based reasoning. They largely fail to address subsumption-based reasoning, the core paradigm of German civil-law doctrine. This paradigm requires a structured, four-step generation process (Obersatz–Definition–Subsumtion–Ergebnis) where the system must anchor specific facts under statutory norms. Current benchmarks do not adequately test this scaffolded generation capability or evaluate systems against the specific, rubric-based grading conventions used in German state examinations.

Methodology

The BenGER Dataset

The authors introduce BenGER (Benchmark for German Law), a dataset comprising three distinct components totaling over 1,100 tasks:

ZJS Corpus: 581 publicly available, exam-style free-text legal case tasks from the Zeitschrift für das Juristische Studium, covering civil, criminal, and public law across various educational levels.
Benchathon Corpus: 15 newly collected exam-style tasks of intermediate difficulty. Crucially, this subset includes 220 human solutions (65 traditional, unaided; 155 human–AI co-created) generated in a controlled environment to serve as human baselines.
Doctrinal Principles Corpus: 531 short-form reasoning items (question–answer pairs) probing core legal principles and recent case law.

Evaluation Framework

The study evaluates 12 contemporary LLM systems, categorized as closed-flagship, efficiency-oriented, and open-weight reference models. The evaluation employs a hybrid methodology to address the variability of legal grading:

LLM-as-a-Judge: A specialized LLM judge (primarily GPT-5-mini) is aligned with German legal grading rubrics. It evaluates solutions across 10 dimensions: result correctness, issue identification, legal grounding, doctrinal knowledge, subsumption quality, problem depth, methodological structure, organization, terminology, and formal correctness. Scores are aggregated to a 0–100 raw score and mapped to the German 0–18 grade-point scale.
Human Validation: The LLM judge is cross-validated against a multi-rater human protocol on the Benchathon subset. Each selected solution receives three independent blind reviews and one author-informed creator review.
Statistical Analysis: The study treats evaluation as a distribution over plausible expert assessments rather than a deterministic label. It utilizes metrics such as Pearson/Spearman correlation, Cohen's $\kappa$ , Intraclass Correlation Coefficients (ICC), and the Calderon alternative-annotator test to determine if substituting an LLM judge for a human reviewer degrades agreement with the full human pool.

Experimental Setup

Generation: All systems were prompted with identical German-language instructions mirroring the Gutachtenstil (examination style) scaffold.
Metrics: Evaluation includes automatic surface metrics (BLEU, ROUGE, METEOR, BERTScore, etc.) and the rubric-aligned LLM judge scores.
Conditions: Human baselines were tested under "traditional" (unaided) and "co-creation" (AI-assisted) conditions to measure the impact of AI tools on human performance.

Key Results

1. System Performance (RQ1 & RQ2)

Closed Flagship Dominance: Closed-flagship systems (OpenAI, Anthropic, Google) lead the leaderboard across all three corpora. On the Benchathon subset, top-tier models (e.g., Sonnet-4.6, Opus-4.7, GPT-5.4) achieve raw scores in the low-to-mid 80s with pass rates near 100%.
Open-Weight Gap: Strong open-weight systems (DeepSeek-V4-Pro, Qwen3-235B) trail the top closed models by approximately 9–20 raw points on Benchathon, though this gap narrows on the ZJS corpus.
System Tiers: The performance gap between flagship and open-weight systems is significant but narrows at the mid-tier. Efficiency-tier closed systems and open-weight models form an overlapping mid-band.

2. Evaluation Validity (RQ3)

Lexical vs. Semantic Metrics: Contrary to expectations for long-form text, lexical surface metrics (METEOR, ROUGE, BLEU) correlate more strongly with the rubric-aligned LLM judge than semantic embedding metrics (BERTScore, Sentence-BERT). This is attributed to the highly templated nature of German legal writing, where high-quality answers share substantial n-gram material regarding statutory citations and structural scaffolding.
Metric Limitations: Relying solely on lexical metrics would misrank systems at the flagship–mid-tier boundary compared to the rubric-aligned judge.

3. Human–AI Co-Creation (RQ4)

Performance Boost: Human–AI co-creation substantially outperforms unaided human work. The mean judge score for co-created solutions is 82.2 (raw), compared to 62.3 for traditional unaided solutions—a difference of +19.9 raw points.
Benchmarking: The mean performance of co-created human solutions lands within the 95% confidence interval of the closed-flagship LLM tier, significantly outperforming unaided humans and weaker open-weight models.

4. Grading Reliability (RQ5)

LLM Judge Validity: The LLM judge's deviation from the human mean is comparable to the inter-rater variability among human graders.
- Correlation: The judge agrees with the mean of blind human reviewers at Pearson $r=0.78$ .
- Calderon Test: Substituting the LLM judge for one human reviewer in a pool degrades agreement with the full human reference no more than removing that human reviewer altogether ( $r=0.96$ for both judge-substituted and human leave-one-out pools).
Stability: The LLM judge is more self-consistent across re-runs (stdev 2.51) than human raters are with each other (spread 23.9 raw points).
Dimensional Agreement: Agreement is highest on structured dimensions (Result correctness, Issue identification) and lowest on presentation-based dimensions (Organization, Terminology).

Significance and Claims

The paper claims three primary contributions:

A New Benchmark: BenGER provides the first comprehensive benchmark for subsumption-based legal reasoning in German law, addressing a gap in existing legal NLP resources which favor common-law or classification tasks.
Empirical Evaluation: It offers a rigorous evaluation of 12 LLM systems against human baselines, revealing that while closed-flagship models lead, human–AI co-creation yields the highest performance, surpassing unaided human experts.
Methodological Innovation: The study validates a rubric-aligned LLM-as-a-Judge framework that accounts for the inherent variability of legal grading. By demonstrating that an LLM judge can substitute for a human reviewer without degrading the reliability of the evaluation pool (per the Calderon test), the paper supports the use of LLMs as scalable raters alongside humans, rather than as standalone ground truth.

The authors emphasize that the benchmark is intended for research-scale evaluation and is not yet validated for use as a graded assessment instrument in legal education or for deployment in legal practice. The work highlights the structural demands of German legal reasoning and the potential of AI to augment, rather than replace, human legal analysis when used in a co-creative capacity.