💬 NLP

Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics

This contribution introduces LEGIT, a large-scale dataset comprising 24,000 expert-level legal reasoning paths structured as hierarchical problem trees and serving as a robust evaluation framework to assess and demonstrate how retrieval-augmented generation and reinforcement learning can complementarily enhance the coverage and correctness of legal reasoning in LLMs.

Original authors: Jinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a junior lawyer to help you solve a complex legal puzzle. You do not merely want them to guess the final outcome (such as "The defendant pays $50,000"); you must review their workbook. Did they consider all the right clues? Did they connect the dots correctly? Or did they overlook a crucial piece of evidence and still arrive at the correct result by luck?

This work, titled "Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics," introduces a new method for evaluating AI lawyers (Large Language Models) not only by their final verdict but by the quality of their reasoning process.

Here is a breakdown of their work using simple analogies:

1. The Problem: The "Black Box" of AI Reasoning

Current AI models excel at solving math problems or writing code because answers are usually clearly right or wrong. In law, however, it is more chaotic. An AI might predict the correct court ruling (e.g., "The case is dismissed") but arrive there by ignoring important facts or applying flawed logic.

If you ask an AI, "Why did the court decide this way?" and it provides a wrong explanation, that is dangerous in high-stakes fields like law. Existing methods for evaluating these AI "reasoning traces" (the step-by-step thought processes) are too vague, like a teacher giving a student a "B" on an essay without explaining why.

2. The Solution: LEGIT (The "Legal Issue Tree")

The authors created a massive new dataset called LEGIT (LEGal Issue Trees). Imagine a court case not as a single question, but as a family tree of arguments.

The Root: The main claim (e.g., "Pay me my insurance money").
The Branches: To prove the root, you must prove smaller things (e.g., "Was the event sudden?", "Was it external?", "Was there a pre-existing condition?").
The Leaves: The specific facts (e.g., "The victim choked on a rice cake").

In LEGIT, they converted real court rulings into these structured trees. This tree acts as a rubric (an evaluation checklist).

3. How They Evaluate the AI: The "Rubric" Analogy

Instead of asking an AI, "Is this reasoning good?" (which is vague), they ask it to check specific boxes against the "Issue Tree":

Coverage: Did the AI mention this specific branch of the tree? (Did it notice the "pre-existing condition" argument?)
Accuracy: Did the AI draw the correct conclusion for this branch? (Did it correctly decide that the pre-existing condition likely caused the death?)

They presented this dataset to human lawyers for evaluation. The lawyers agreed almost perfectly, proving that this "tree" method is a fair and objective way to assess legal reasoning.

4. What They Found: The Weaknesses of AI

When they tested the best AI models on LEGIT, they found that even the smartest AIs struggle. They identified two main types of "errors":

The "Decomposition Error" (Missing Branches): The AI forgets to consider an entire branch of the tree. It completely ignores a critical legal question.
The "Deduction Error" (Flawed Logic): The AI considers the branch but draws the wrong conclusion from the facts.

The Big Discovery: If an AI misses a branch or misjudges a small branch, it almost always gets the final result wrong too. You cannot build a stable house if you skip the foundation or use rotten wood for the beams.

5. Two Ways to Improve the AI: RAG vs. RL

The authors tested two common methods for improving AI and found they have opposite effects:

RAG (Retrieval-Augmented Generation): The "Open-Book Test"
- How it works: Before the AI answers, the system searches a library of laws and past cases and feeds the relevant pages to the AI.
- The Result: The AI becomes a better "researcher." It finds more branches of the tree (better coverage) and argues better because it has the rules in front of it. It improves everything.
RL (Reinforcement Learning): The "Drill Sergeant"
- How it works: The AI is trained by a computer judge that only gives points if it gets the final result right.
- The Result: The AI becomes a "laser-focused" guesser. It gets the final verdict right more often (better accuracy) but starts skipping the difficult, complex branches of the tree to avoid mistakes. It sacrifices coverage for precision.

The Bottom Line: RAG helps the AI understand the whole picture, while RL helps it get the final answer right but makes it lazy about checking every detail. The authors suggest using both together to achieve the best results.

Summary

This work created a massive, structured "grading key" based on real court cases to teach us how to evaluate AI lawyers. They found that current AIs often miss important legal details or reason poorly, and that while giving them access to laws (RAG) helps them think broadly, training them only to "give the right answer" (RL) causes them to skip steps. To create truly reliable AI for law, we must review their workbooks, not just their final grades.

1. Problem Statement

Evaluating Large Language Models (LLMs) in expert domains such as law is crucial for ensuring credibility and explainability, yet it remains a significant challenge.

Limitations of Current Benchmarks: Existing datasets for Legal Judgment Prediction (LJP) typically focus exclusively on predicting the final court verdict (e.g., "win" or "lose"). They ignore the quality of the intermediate Chain-of-Thought, which is critical in high-risk domains.
Evaluation Gaps: Standard evaluation metrics fail to capture two essential dimensions of legal argumentation:
1. Issue Coverage: Did the model identify all relevant legal issues and sub-issues?
2. Issue Correctness: Did the model correctly derive the facts and laws regarding these specific issues?
Reliability of "LLM-as-a-Judge": Generic "LLM as Judge" approaches often lack the domain-specific nuance required to evaluate complex legal arguments, leading to inconsistent or superficial assessments.

2. Methodology: The LEGIT Framework

The authors introduce LEGIT (LEGal Issue Trees), a novel framework consisting of a large-scale dataset and a structured evaluation methodology.

A. The LEGIT Dataset

Scope and Source: A dataset of 24,406 Korean civil and administrative court judgments.
Selection Criteria: Only deterministic judgments were selected (where the outcome is prescribed by law; cases requiring judicial discretion, such as sentencing ranges, were excluded).
Structure:
- Fact Extraction: Instead of using raw "facts" sections, an LLM extracts "atomic facts" (uniform factual events) and generates a coherent case narrative.
- Legal Issue Trees: The core innovation. Court judgments are converted into hierarchical trees, where:
  - Root: The purpose of the claim and the final court verdict.
  - Internal Nodes: Legal issues decomposed into sub-issues (e.g., "Was the accident external?" $\to$ "Did the victim choke?").
  - Leaves: Atomic facts and common-sense inferences.
- Rubrics: Each node in the tree serves as a specific rubric for evaluation and contains the party's argument as well as the court's conclusion.

B. Evaluation Metric: LEGIT Score

The framework evaluates LLM responses using a 10-point scale composed of three weighted components:

Final Verdict Correctness (5 points): Binary score (5 if the final verdict matches the Ground Truth, otherwise 0).
Issue Coverage (2 points): Measures whether the model identified the legal issues contained in the Ground Truth tree. Score = $2/N$ per covered issue ( $N$ = total number of non-root issues).
Issue Correctness (3 points): Measures whether the model predicted the correct conclusion for the covered issues. Score = $3/N$ per correctly concluded issue.

C. Verification of Reliability

Alignment with Human Experts: The authors validated the rubrics by having licensed Korean lawyers annotate the data. The lawyers achieved a Krippendorff's $\alpha$ of 0.87, indicating high inter-rater agreement and the objectivity of the rubrics.
LLM as Judge: Strong LLMs (e.g., GPT-4.1, Gemini-2.5) achieved significant agreement ( $\alpha \approx 0.62-0.74$ ) with human experts when using LEGIT rubrics, outperforming coarse Likert-scale evaluations.

3. Main Contributions

LEGIT Dataset: The first large-scale LJP benchmark containing structured, hierarchical reasoning traces (Legal Issue Trees) rather than just final answers.
Structured Rubrics: A method for automatically extracting fine-grained, instance-specific rubrics from court judgments, enabling scalable and expert-aligned evaluation.
Error Taxonomy: Identification of two distinct error modes in legal reasoning:
- Decomposition Errors: Failure to identify relevant sub-issues (missing coverage).
- Deduction Errors: Failure to correctly derive facts/laws for identified issues (missing correctness).
Complementary Optimization Strategies: Empirical evidence on how Retrieval-Augmented Generation (RAG) and Reinforcement Learning (RL) differently influence reasoning.

4. Main Results

A. Performance of State-of-the-Art LLMs

Even the most advanced models (e.g., GPT-4.1, Gemini-2.5) struggle with LEGIT.

The highest achieved score was 5.71/10 (GPT-4.1).
Error Analysis: Models frequently commit Decomposition Errors (missing sub-issues) or Deduction Errors (incorrect logic regarding facts). These errors propagate through the tree, causing the final verdict prediction to fail.

B. Impact of RAG vs. RL

The study tested two methods for improving legal argumentation:

Retrieval-Augmented Generation (RAG):
- Effect: Improves all components (coverage, correctness, and final verdict).
- Mechanism: Providing relevant laws and case citations helps the model identify more issues and argue more accurately.
Reinforcement Learning (RL) with LEGIT Rubrics:
- Effect: Significantly improves the final verdict and issue correctness, but reduces issue coverage.
- Mechanism: The model learns to "play it safe" by focusing only on issues where it is highly confident to maximize reward, thereby omitting uncertain (but potentially relevant) sub-issues.
Conclusion: RAG and RL offer complementary benefits. RAG expands the model's knowledge base and issue identification, while RL sharpens the precision of its reasoning on identified issues.

C. Consistency of Evaluation

LEGIT Rubrics vs. Likert Scale: Modular LEGIT rubrics showed significantly higher agreement among LLM evaluators than coarse Likert-scale prompts, proving that structured, granular evaluation is more robust.

5. Significance and Future Work

Advancing Expert AI: LEGIT goes beyond simple answer prediction to evaluate the process of thinking, which is essential for trust in high-risk domains like law and medicine.
Scalable Expert Evaluation: By automating the extraction of rubrics from judgments, the paper demonstrates a path to creating high-quality evaluation datasets without prohibitively high costs for human annotation.
Paradigm Shift in Training: The results suggest that optimizing the quality of the reasoning trace (via RL on rubrics) leads to better final outcomes than optimizing solely for the final answer. This challenges the "answer-only" reward paradigm common in mathematics and programming.
Limitations: The current dataset is limited to the Korean legal system and the Korean language. The authors propose extending this framework to other legal jurisdictions and languages as future work.

In summary, LEGIT offers a rigorous, structured, and scalable framework for evaluating and improving the legal reasoning capabilities of LLMs, underscoring that true legal intelligence requires both comprehensive issue coverage and precise logical deduction.