💬 NLP

LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

This paper introduces LP-Eval, a rubric and dataset co-designed with legal experts to evaluate the quality of LLM-generated legal propositions from EU court decisions, revealing that while models produce generally high-quality outputs, their performance varies by case age and that rubric-guided LLM evaluators, though better than direct scoring, still lack the fine-grained sensitivity of human experts.

Original authors: Shanshan Xu, Johan Lindholm, Amogh Raina, Henrik Palmer Olsen, Daniel Hershcovich

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shanshan Xu, Johan Lindholm, Amogh Raina, Henrik Palmer Olsen, Daniel Hershcovich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a lawyer trying to explain a complex court ruling to a client. You don't want to read them the entire 50-page document; you want to boil it down to one clear, powerful sentence that captures the "rule of the game." In the legal world, this summary sentence is called a Legal Proposition.

This paper is about teaching computers (specifically Large Language Models, or "LLMs") to write these summary sentences automatically and then figuring out how to grade them fairly.

Here is the breakdown of the research, explained simply:

1. The Problem: The "Gold Standard" is Hard to Find

In many computer tasks, you can easily check if an answer is right or wrong (like a math problem). But in law, there is rarely just one "correct" way to summarize a rule. You could say, "The law says X," or "Under condition Y, X applies." Both might be right, but they sound different.

Because of this, old computer grading tools (which just count how many words match) don't work. They can't tell if a sentence sounds like the law but actually means something totally different. Also, computers sometimes "hallucinate"—they make up rules that don't exist.

2. The Solution: A "Legal Report Card" (The Rubric)

To fix this, the researchers worked with real law professors to create a Rubric (a grading checklist) called LP-Eval. Think of this like a report card for a student's essay, but for legal rules.

Instead of giving a single grade like "B+", the rubric checks two main things:

The Skeleton (Formal Validity): Does the sentence have the three necessary bones?
- Stance: Is it taking a position?
- Object: Is it talking about a legal rule (not just a fact)?
- Specification: Is it specific enough?
- If it's missing a bone, it's "Invalid."
The Flesh (Quality Dimensions): If the skeleton is good, how well-written is it?
- Conciseness: Is it short and sweet, or rambling?
- Fidelity: Does it stick to the original text, or did it make things up?
- Generality: Is it a broad rule, or does it only apply to one tiny detail?

3. The Experiment: Teaching the Computer

The researchers took 100 paragraphs from real European Union court decisions and asked three different AI models to write legal propositions for them. Then, they had two human legal experts grade these AI outputs using the new "Report Card."

What they found:

The AI is pretty good at the basics: Most of the AI-generated sentences were "valid" (they had the right bones).
Old cases vs. New cases: The AI did a significantly better job summarizing old, famous cases (the ones everyone knows and has talked about for years) compared to brand new, recent cases. It's like how a student might ace a test on a classic story they've read a hundred times but struggle with a story published yesterday.
The AI as a Grader: The researchers also asked the AI to grade the other AI's work.
- Good news: When the AI used the detailed "Report Card" (the rubric), it agreed with the human experts much more often than if it just tried to give a single overall score.
- Bad news: Even with the rubric, the AI was "tone-deaf." It couldn't spot the subtle differences between a "good" summary of an old case and a "good" summary of a new case. The human experts could feel the difference; the AI just saw them as roughly the same.

4. The Dataset

The team didn't just do the experiment; they released the "homework" to the public. They made a dataset containing:

The original court paragraphs.
The AI's attempts at summarizing them.
The human experts' detailed grades and notes.

The Bottom Line

This paper shows that computers are getting good at writing legal summaries, especially for well-known laws. However, they still struggle with the nuance of new, complex cases. Furthermore, if you want a computer to grade legal work, you can't just ask it "Is this good?" You have to give it a detailed checklist (a rubric) to follow, or it will miss the important details that human experts catch.

In short: Computers can write the legal rules, and they can grade them if given a strict checklist, but they still lack the "legal intuition" to tell the difference between a classic case and a fresh one.

Technical Summary: LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

Problem Statement

Legal proposition generation—the task of extracting normative statements (rules, principles, conditions, and consequences) from legal texts—is a foundational element of legal reasoning and doctrinal scholarship. Despite its centrality to legal practice and the organization of legal knowledge, this task remains under-examined in Legal Natural Language Processing (Legal NLP). Existing research has largely focused on precedent retrieval, judgment prediction, or case summarization, often treating legal norms implicitly rather than explicitly.

A significant barrier to progress in this domain is the lack of robust evaluation methodologies. Legal proposition generation is an open-ended task where multiple valid outputs may exist, and minor phrasing differences can alter legal meaning. Consequently, standard n-gram metrics like BLEU and ROUGE are ill-suited as they measure surface overlap rather than legal fidelity. Furthermore, while Large Language Models (LLMs) show promise as evaluators ("LLM-as-a-judge"), they are vulnerable to hallucinations and often lack the nuance required to distinguish between high-quality and low-quality legal abstractions without structured guidance.

Methodology

The LP-Eval Rubric

To address the evaluation gap, the authors introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts. The rubric decomposes the quality of a generated legal proposition (LP) into two primary categories:

Formal Validity (Required Components): An LP is deemed valid only if it contains three specific structural elements:
- Stance: The normative position (e.g., "imposes a duty," "prohibits").
- Object: The entity or right being regulated.
- Specification: The conditions, scope, or consequences of the rule.
- Note: If any component is missing, the proposition is labeled INVALID.
Quality Dimensions (Scoring): Valid propositions are scored on a 1–3 Likert scale across five dimensions:
- Source Independence: The degree to which the proposition is abstracted from specific case facts.
- Fact Independence: Similar to source independence, focusing on the removal of factual specifics.
- Conciseness: Brevity and lack of redundancy.
- Generality: The applicability of the rule beyond the immediate case.
- Fidelity: Faithfulness to the authoritative source text.

Dataset Construction

The authors released a dataset comprising 100 LLM-generated legal propositions derived from 50 paragraphs sampled from 10 Court of Justice of the European Union (CJEU) decisions.

Sampling Strategy: To test for training data memorization, the dataset includes a mix of well-established (highly cited, older) cases and recent cases.
Generation: Three models were used to generate propositions: GPT-OSS (120B parameters), OLMo-3-7B-Instruct, and Saul-7B-Instruct (legal-specialized).
Annotation: Two legal experts (a PhD researcher and a final-year law student) annotated the GPT-OSS and OLMo outputs using the LP-Eval rubric. Inter-annotator agreement was high (Gwet's AC1 = 0.96), validating the rubric's clarity.

Experimental Design

The study investigates two tasks:

LLM as Generator: Assessing the quality of propositions generated by LLMs against expert annotations.
LLM as Evaluator: Assessing whether LLMs can replicate expert judgments. This was tested in two modes:
- Rubric-Guided: The LLM follows the step-by-step LP-Eval protocol (checking components, then scoring dimensions).
- Direct Scoring: The LLM provides only an overall score without intermediate steps.

Key Results

1. Generation Quality

Formal Validity: LLMs demonstrated strong capability in generating structurally valid propositions. 95 out of 100 generated LPs were rated as VALID. The primary failure mode was the omission of the "Object" component (a normative statement), which also proved to be the most difficult component for human annotators to agree upon.
Quality Scores: Experts assigned high average scores for Fidelity (2.95/3) and Fact Independence (2.96/3), indicating that models successfully abstracted from specific facts while remaining faithful to the text. Source Independence (2.31/3) and Conciseness (2.48/3) received slightly lower scores, suggesting occasional redundancy or explicit source referencing.
Case Prominence Effect: A statistically significant difference was found between propositions derived from well-established cases versus recent ones. Propositions from well-established cases were rated significantly higher in Conciseness ( $p=0.02$ ), Source Independence ( $p<10^{-5}$ ), and Overall Quality ( $p<0.001$ ). This suggests that LLMs perform better on cases with more established, perhaps more memorized, legal discourse.

2. Evaluation Performance

Rubric Alignment: When using the LP-Eval rubric, GPT-OSS achieved high agreement with human experts (AC1 = 0.91–0.93), comparable to inter-expert agreement (0.94).
Importance of Structure: Removing the rubric steps and asking for a direct overall score reduced agreement to 0.85, demonstrating that structured guidance is critical for LLM evaluators.
Sensitivity Limits: While rubric-guided LLMs aligned well with experts on overall scores, they failed to detect the statistically significant quality differences between well-established and recent cases that human experts identified. This indicates that while LLMs can mimic expert scoring, they lack the sensitivity to capture finer-grained qualitative distinctions related to case prominence.

Significance and Contributions

The paper makes three primary contributions to the field of Legal NLP:

LP-Eval Rubric: It introduces the first structured, expert-designed evaluation framework specifically for legal proposition generation, moving beyond surface-level metrics to assess formal validity and substantive quality dimensions.
LP-Eval Dataset: It releases a curated dataset of 100 expert-annotated legal propositions from CJEU decisions, providing a benchmark for future research in legal reasoning and generation.
Empirical Insights: The study provides the first systematic evidence that:
- LLMs can generate high-quality, formally valid legal propositions.
- The quality of generation is correlated with the prominence and age of the source case, likely due to training data effects.
- Rubric-guided LLM evaluation is a viable alternative to human scoring for general alignment but currently lacks the sensitivity to detect nuanced variations in legal abstraction that human experts perceive.

The authors conclude that while LLMs offer promise for automating legal proposition generation and evaluation, current models remain insufficiently sensitive to the subtle, high-stakes distinctions inherent in legal reasoning, necessitating continued reliance on human expertise for fine-grained assessment.